From nch@roe.ac.uk Mon Sep 17 15:13:09 2007 Date: Fri, 14 Sep 2007 17:29:31 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , John Taylor , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 14/9/07 Minutes of WFAU VDFS Science Archive meeting: 14th September 2007 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RSC, PMW, NJC, JB, MSH, LGR, ETWS Apologies: BCW, JPE, AL, JMS, MAR, JDT, RGM NB: DONM: 10am, Friday 21st September 2007 in the Plate Library Actions discharged this week: ----------------------------- ACTION: NCH to enquire as to the status of the photometric zeropoint recalibration activities at CASU. Discharged - see Survey release below. ACTION: PMW to assemble the usual reporting materials for the next VDMT during the coming week. Discharged ACTION: NCH to send MAR the GCS eyeball check results, and forward the DXS results (received yesterday). Discharged ACTION: MAR to put a note that some archive services will be offline for up to a day on Mon 1st Oct on the WSA website downtime page. Discharged Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; will get sorted (honestly!) as part of the general ingest DB schema revamp to facilitate rapid generalised photometric/astrometric recalibration. ACTION: RGM to look through requirements docs to see if any users have mentioned GALEX. Continues; RGM supplied GALEX details; will now check WSA SRAD and VISTA URD for any mention of this. Actions carried forward from 31/08/07 meeting: ---------------------------------------------- ACTION: JB to restore BestDR1 to thutmose from LTO tape at a convenient time (but no rush!). Specific points and new actions: -------------------------------- Project management: PMW, JMS and NCH attended yesterday's VDMT telecon. JPE gave a VISTA schedule update; the usual progress charts were reviewed. JPE noted that the Survey Management Team will be reviewing VISTA Survey Management Plans on 22 Oct, after which survey designs will be finalised. NCH noted that Luca Rizzi (JAC UKIDSS survey support chap) will be visiting WFAU on the 24/25 October. SJW has emailed all interested parties to note that registration for the UKIDSS ESO workshop opens on 15th Sept. WFCAM & VISTA updates: Good news on the JAC WFCAM homepage: "WFCAM resumed normal operations on Sept 11th following successful work on the cold head". Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: Nothing new to report this week. WSA Operations: JB reported that transfer and ingest of data for DR3 is now complete barring the fixes for the 1/1/07 image astrometry problem and the reprocessed GPS selection from 06A. Hardware: JB noted that curation server sneferu has been rebooted, and has subsequently shown a dramatic improvement in performance. NCH asked that the archive curation Linux servers be checked for free memory to ensure optimal performance. ACTION: JB and RSC to check the archive linux curation servers for leaked memory. Software: RSC reported: "Tested CU7 with the new slimmed-down GPS source table schema (see trac ticket 61). This revealed a few small problems with the code that have now been fixed, so CU7 should now run without any problems for DR3. I've updated the CurationOverview TWiki article with a note on how to apply the GPS schema change to the WSA for DR3. Enhanced the curation scripts Logger module to continuously write the log to file when the messages aren't verbose (see trac ticket 18). This will make it easier to keep track of long runs of CU7, CU8 and CU19 for DR3 if we lose a terminal, e.g. due to a power cut. Also enabled simultaneous running of the same CU for different programmes, just in case this proves to be more efficient for DR3, e.g. for CU7. Finally managed to create a working Python wrapper to both WCSLib and AST to perform WCS co-ordinate transformations (see trac ticket 90). I've chosen the latter library for PyWCS since we already have a system installation and have created a WCSforWFCAM TWiki article for reference. I'll now be able to make a serious start on solving the problem of cross-talk flagging the UDS mosaic for DR3 (see trac ticket 33)." ACTION: NCH to check the final DR3 tests of CU7 done by RSC. NJC reported: "This week I have done further updates to CU13 and 14, notably running the shell script to set the environment variables needed for the CASU extractor in the python script. I have also been working on the synoptic pipeline - a variability classifier and on the galaxy photometry paper." ETWS reported: "-Wrote a script to check if detections are completely ingested or if there is a mismatch between database entries and catalogue file lengths due to possible disk failures. - Installed the whole software package new on sneferu due to the change in its setup. - Wrote a script that allows deprecation, upload, and ingest of the list of files that have been regenerated to fix the GPS problems reported in the 06A data. - Included more flexibility in code to cope with the afore mentioned changes. - Updated code so that ProgrammeCurationHistory entries can also be updated. - Worked on the ADASS poster." Survey Data Release: Revised schedule (weeks remaining) in the light of the past 2 week's progress: Weeks What: 1.0 CU3/4 (ingest and provenance updating) 1.0 CU8 (photometric zeropoint recalibration) 2.0 QC1 (much work to be done in parallel with previous tho') 0.0 CU5 (diff images for the GPS - after QC, but in parallel with the following and it's very quick anyway) 1.0 Quality bit flagging 3.0 CU7 (source merging from scratch again, unfortunately) 0.0 CU13/14 for DXS/UDS (in parallel with shallow survey CU7s) 2.0 Final CUs. Note that the degree to which some of QC1 can run in parallel is a little uncertain and the above estimates are generally pessimistic; the worst-case scenario of is DR3 release by the end of November. Everything will be done to expedite and arrive sooner. Note that transfer and ingest having finished, but there is still some early CU fettling required to a) correct the astrometry problem on the night of 1st Jan'07, and b) retransfer/reingest of the reprocessed GPS cross-talk fixed images. This will be done to fill in time as we await the final detector-frame level photometric zeropoint recalibration updates from CASU. ACTION: ETWS to liaise with CASU over the timing of the GPS 06A retransfer to avoid any collisions with header updates NCH noted that the JPEGs are all in so a gentle reminder will be sent to the UKIDSS Survey Heads to wrap up their eyeball checks asap. ACTION: NCH to email a gentle reminder to the UKIDSS Survey Heads about jpeg eyeballing. Non-survey Data Release: JB noted that 3 newly registered non-survey PI and one service programmes have been set up for flat file access. Astrogrid deployment: MSH noted that the "personal" PSSA database (1% full SSA) has been added to the WFAU DSA service to enable small scale experiments through VO query builders etc. Miscellaneous: So Pavarotti arrives at the Pearly Gates. St Peter opens them and says "Oh, it's you Luciano, come on in - squeeze through". Pavarotti says "hold on, I've got an envelope for you, from the Pope." St Peter opens it up and reads it. "Here's that tenor I owe you."