From nch@roe.ac.uk Mon Nov 26 15:18:09 2007 Date: Mon, 26 Nov 2007 14:31:09 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 23/11/07 Minutes of WFAU VDFS Science Archive meeting: 23rd November 2007 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RSC, MAR, PMW, JB, ETWS, LGR Apologies: BCW, JPE, AL, RGM, JMS, MSH, NJC DONM: 10am, Friday 30th November 2007 in the Plate Library Actions discharged this week: ----------------------------- No specific actions dished out last time. Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; will get sorted (honestly!) as part of the general ingest DB schema revamp to facilitate rapid generalised photometric/astrometric recalibration after DR3. ACTION: RSC to progress implementation of automated replication of the ingest DB file metadata on a public server. Continues; RSC has located and fettled the appropriate scripts, and will now test against the restored ingest DB. A similar solution is proposed for proprietary-lapsed non-survey data access (see Non-survey Data Release below). Some discussion took place as to the names of these databases: FlatFiles? WFCAMfiles? ...? Actions carried forward from 08/11/07 meeting: ---------------------------------------------- ACTION: JB & ETWS to restore SegueDR6 on thutmose at their convenience (i.e. no hurry given the present circumstances) Continues; team agreed to postpone until after DR3 now. ACTION: JB to set dfg_1 shrinking in the WSA this weekend. Continues; will try this weekend end as DXS DR3 preparations are on hold (see below) Specific points and new actions: -------------------------------- Project management: NCH and PMW reported back from the VDMT; no substantive issues. PMW noted that the Research Council have now slipped the review date of the WFAU Rolling Grant (which includes of course a bid for operational and residual development resources) to the New Year. JB announced his resignation as of 1st Jan 2008 to take up an IT position in Geophysics; the team thanked him for all his hard work and wished him well in his new position. PMW noted that in drawing up the job spec for a replacement we now have the opportunity to review the mix of IT systems and WSA operations required based on JB's experience. NCH asked JB to complete a "brain dump" onto the TWiki of all relevant ops information to aid his successor. ACTION: JB to complete operations "brain dump" onto the TWiki. Finally, PMW noted that NCH, MAR and RSC will be moving offices during the week commencing 10th Dec in order to vacate the rapidly disintegrating plate library in advance of redevelopment works. WFCAM & VISTA updates: No WFCAM news this week; an update on the Vista schedule was given by JPE and JMS in the VDMT. Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: JB noted that August 07B has all been successfully transfered; ingests are on hold until after DR3 phase-1 release to avoid any danger of overloading the poor old cpus. JB also noted that UKERNA have informed us of work on UKLight next Thurs (29th) - it will be down for an hour at 6pm this Thursday for maintenance. WSA Operations: Operations have consisted almost entirely of the DR3 preparations fire-fight; JB, ETWS, NJC and RSC have all contributed to the running/checking/debugging of CUs. JB noted: "All work has been directed towards releasing DR3 phase 1; GCS reached the end of CU16 and is now ready for release, LAS is currently running CU16 - should finish sometime very early next week [now done], UDS is awaiting a fix before CU7 and CU16 (quick) runs make it ready, DXS is having problems but these are under investigation - it was looking very good until the bugs showed up." Hardware JB reported that sneferu's /disk05 is all shifted to osorkon's NAS now and extraneous crossmounts removed. MAR noted that interactive logins on web server thoth seem to be somewhat sluggish. ACTION: JB to have a poke around to see if all is well on thoth and it's network filesystems. Software: RSC reported: "Created a release_4 branch containing late fixes to CU13/14 and the ability to apply detection quality bit flags to DXS/UDS intermediate stacks, as well as the cross-talk flag for the UDS mosaics. This followed some tests of the cross-talk calibration and required some tidying up of the branch commits. Since then my main focus has been to prepare a release_5 branch of the software that will solve the potential problem of an index conflict that will occur when reseaming the GPS for DR3. The best solution is to remove indices from the WSA and ensure ingests don't reattach them (see trac ticket 100). However, as the WSA is still in use as a pre-release metadata server they are still required. So my first task has been to prepare the script to create the metadata mirror of the WSA, WFCAMPROPRIETY (see trac ticket 97). With the help of MAR I've got the script, just about, working now. In between, I've helped with some bug fixes: updated exmeta to treat as default the 'UNDEF' values in the new keys contained in older FITS files (trac ticket 103). Updated the exmeta test to sample a fuller range of FITS files. Helped NJC release CU13/14 bug fixes to the release branch." ETWS reported: "Redesigned the code to create GPS difference images since 06B/07A had quite a lot of fits files with less than 4 extensions, and created all difference images. Enhanced transfer checking code as well as logging for CU1 to 4." RSC has pointed out several areas where schemas/software/procedures can be enhanced to create a better DR preparation experience (e.g. robust auto-deprecation of mosaicdeepleavstack products; reordering of create statements in metadata schema files etc.) NCH has been doing various DR3 sanity checks, and found only one issue (incorrect ZPs applied in Dec 28/29th data). Turns out to be a bug in CU4 which is being fixed. Survey Data Release: Updated schedule following on from recent work over the last week: Weeks What: 1.0 Final CUs. Noting that the GPS will be excluded from the initial DR3 release in order to expedite, the team have made great progress over the last two weeks and GCS/LAS/UDS are ready to go as at Mon 26th. Unfortunately some hiccups with the DXS following changes to data and/or code; these are being investigated at the moment. The team discussed who will do what to sort out the remaining issues to make sure it is all done with great rapidity. ETWS noted after the meeting that he has checked the production of the DR3 browser pages, and all is well. Non-survey Data Release: No news this week. Astrogrid deployment: No news this week. Miscellaneous: NCH noted that the WSA paper has been accepted for publication in MNRAS, and was put on astro-ph yesterday (will appear Monday 26th); also the UKIDSS Workshop programme is now out so NCH, NJC, ETWS and LGR should now start to think about their presentations. JB reported that the twiki and software documentation has been updated to provide more details for Operations and with new information from the DR3 phase 1 build experiences.