From nch@roe.ac.uk Fri May 23 17:42:03 2008 Date: Fri, 23 May 2008 17:24:44 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 23/05/08 Minutes of WFAU VDFS Science Archive meeting: 23rd May 2008 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RSC, MAR, LGR, MSH, ETWS, NJC, RPB Apologies: JPE, JMS, AL, BCW, RGM, PMW DONM: 10am, Friday 30th May 2008 in the Vista Hut NB: PMW in the chair Actions discharged this week: ----------------------------- ACTION: RSC to create open time/proprietary metadata DBs on public server "WFCAMOPENTIME - created last Friday, just needs to be visible in interface and announced. WFCAMPROPRIETY - interface will switch over to this when new software is released allowing a daily cron job to run and schema changes begin." Discharged; see below for final action ACTION: MAR to get ITSG to mount a RAID filesystem on thoth to cache SDSS pixels. Discharged; request in to mount khufu:/disk07 for this purpose. Actions partly discharged but continuing: ----------------------------------------- None this week. Actions carried forward from 09/05/08 meeting: ---------------------------------------------- ACTION: RPB to convene a meeting of interested parties to discuss locations and importance of all data help on WFAU servers to organise a consistent backup and DR strategy. Continues; RSC noted that new data products (e.g. as for DR4) must be considered, and some kind of automated copying into backup staging space might be a good idea. Specific points and new actions: -------------------------------- Project management: NCH thanked RSC for representing WFAU at the UKIRT Board meeting last Thursday. RSC has put notes up ont he internal TWiki: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ConferenceMeetings and the main item of note was public release timescale for non-survey proprietary flat files (see Non-survey below). WFCAM & VISTA updates: Nothing of note this week. Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: ETWS noted that transfers are up and running as normal after a few hiccups due to UKlight connectivity, and ingest DB downtime owing to attempts to correct a load-server glitch (see below). WSA Operations: NCH noted that a disk IO glitch on the main DB load server ahmose has interupted preparations for DR4 and routine operations. As usual, it's the big GPS detection table that has been hit, and efforts are underway to correct this with minimal impact on routine ops. RPB is working on this; the team discussed and agreed a plan to copy out the affected data into a flat file, then drop the corrupted table, backup both ingest DB and flat file, then reload. This will take all of the coming weekend and then some, but hopefully the ingest DB can stay online throughout to minimise disruption to routine ops at least. DR4 preparations are discussed further below. RSC noted: "The WSA has been completely updated to the new schema, following the steps listed in the RecalibrationStrategy TWiki page. The splitting of the detection tables were heavily delayed by ahmose locking up, but when it was running it was taking a swift 3 hours to copy 100 million sources. This included revising the default detection row to have default cuEventID values, and the insertion of default rows for every catalogue. A few non-survey detection tables had to be patched up following foreign key constraint failures." Hardware NCH and MAR noted problems in compiling and running C/Fortran codes and Starlink software on 64bit web server horus. The suspicion is something to do with system-level libraries for 64bit Debian4.0; MSH will investigate, and liaise with ETWS/MAR as necessary. NCH noted this may help ITSG in their efforts to get 64bit curation server khafre up and running with Debian4.0 MSH noted that he is looking into procurement of a PanSTARRS database mirror server - this is likely to be a big rackmount cluster running SQL2005 on a central server with up to 16nodes, quad dual-core, 16GB memory per node, 50TB-ish total RAID1+0 storage etc. etc. Software: NCH noted that he and ETWS have been making exhaustive checks on the photometric recalibration implementation to ensure we are completely in line with CASU on the low-level photometric corrections for field distortion and illumination. After some interaction with MJI and using a non-survey dataset as a test case, it looks like everything is fine with CASU/WFAU agreement at the +/-0.5 millimag level. Given that RSC and ETWS are confident that all software mods are propagated forward into the merged trunk, NCH noted that we should be fine to go ahead with the photometric recalibration for DR4 (see below). RSC noted: "Should be all go for DR4. I've merged the development and release branches back into SVN trunk to keep it simple for the release runs whilst we iron out any remaining bugs due to the schema changes. The latest features are documented in the BranchNews Wiki page in trac. I've put a generic UpdateDetectionsTable helper script into SVN that can be used to make common changes to all of the new detections tables, which is particularly useful for keeping non-survey data and schemas up-to-date. This was used to patch up missing default rows from the empty DetectionPhotometry tables. Also I've updated the UpdateSchema helper script to populate the existing totalExpTime entries in the database. CU21 has been updated to produce the new schemas for non-surveys correctly." RSC noted that a daily cron job needs to be set up to invoke the mirroring of flat file metadata to the public server so that users are not affected by loadserver downtime when accessing flat files ACTION: ETWS to set up a daily cron job to synchronise amenhotep..WFCAMPROPRIETY (sic) with ahmose..WSA Survey Data Release: RSC noted for DR4: "The detection quality bit flags have been set, from scratch for the boundary and cross-talk flags due to the new dither offset calculation, for all surveys save the GPS, which experienced a database torn page error. The script runtimes have been entered on the TWiki CuPerformance page. In the detection table splitting I kept the UDS detections that were deprecated due to reprocessing, as these may be required for future mosaic cross-talk flagging. However, the cross-talk flags for the detections from the DR3 UDS mosaics have now been set back to the DR3 values." MAR and NCH will meet this afternoon to quickly run through the QC2 procedure for DR4 which MAR will oversee (NCH at UKIRT next week and then on hols for 2 weeks after that). Otherwise, the immediate priorities for DR4 are: ACTION: RPB (with help from NCH) to secure the GPS detection table data and ingest DB backup ACTION: RSC to update total exposure times for all multiframes ACTION: NJC to test CU13/14 from newly merged SVN trunk in advance of running by ops for DXS in DR4 ACTION: ETWS to run CU4 for individual UKIDSS surveys after successful backup of the ingest DB ACTION: MAR (with advise from NCH) to progress as much as possible of QC2 ACTION: ETWS to continue with routine CUs1-3 to ensure timely ingest & release of 08A processed flat files for Mar/Apr etc. For the GPS, quality error bit flagging (interupted by load server glitch) and CU5 difference images will be interleaved as time allows. Non-survey Data Release: RSC noted that the most important matter arising from the UKIRT Board meeting was that of the proprietary period for releasing non-surveys - the Board were quite adamant that we adopt a policy of releasing the non-survey data exactly 12 months from the date of the last night of the semester in which the data were taken. This would implies that all 06A, 06B & 07A data were released now ACTION: RSC to update WFCAMOPENTIME in line with the Board's wishes. The WSA team wish it to be noted that this may produce grumbles from PIs whose data are obtained towards the end of the Semester, particularly if there are any hold-ups in the dataflow system in getting processed flat-files released to them. Both RSC and MJI pointed this out at the Board meeting; anyway so be it - any complaints will be forwarded to the appropriate folks. NCH noted that the Board and JAC were understandably concerned about the delays in releasing survey-like DBs to non-survey users. MAR noted that the necessary automated QC is one of the main issues, and this will get sorted as time allows but the bottom line is that staff resource limitations are the main hold-up. NCH noted that he will continue to communicate with JAC folks about this. Astrogrid deployment & Data Analysis services MSH noted that the AG workshop took place last week, and potential users requiring UKIDSS access are directed to email him to set up the appropriate access certificates. LGR noted that his SDSS pixel analysis service needs a little adjustment due to pixel server-side limitations in the US. Miscellaneous: Nothing else this week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================