From nch@roe.ac.uk Fri Jun 30 12:20:44 2006 Date: Fri, 30 Jun 2006 11:49:24 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU WSA weekly meeting minutes, 30th June 2006 Minutes of WFCAM Science Archive meeting: 30th June 2006 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, NJC, PMW, RSC, ETWS, MSH, JB Apologies: AL, JPE, MAR, JMS, RGM NB: DONM: 10am, Friday 7th July 2006 Actions discharged this week: ----------------------------- ACTION: PMW to get a quote from Eclipse for another 4TB JBOD for batch catalogue server thutmose (that'll annoy premises...) Discharged; cost per TB is roughly in line with predictions used in the 2004 RG renewal for HW provision for archive operations. ACTION: NJC & RSC to tweak source merging code to cope with the differences between CASU- and S-extractor (mainly in handling of merged classifications) Discharged; needs final testing with final products. ACTION: NJC & ETWS to finalise modifications to CU3 & 4 to facilitate ingestion of data, esp. S-extracted catalogues, into the archive Discharged ACTION: NCH to fettle the CU19 release script to expurgate all CASU-extracted intermediate stack catalogues from udsDetection in release DBs to avoid utter confusion on the part of WSA users. Discharged; NCH has also prepared CU19 for the DR1PLUS run. ACTION: MSH to co-ordinate abstract submission to IAU Prague on WFAU data handling activities, and poster design. Discharged; MSH will be attending. Actions partly discharged but continuing: ----------------------------------------- ACTION: ETWS & NJC to unlink all UDS documentation and glossary entries pointing to CASU-extractor info Discharged; NCH to review and tweak as necessary Actions carried forward from 23/06/06 meeting: ---------------------------------------------- ACTION: JB to review Hardware/OS/DBMS design doc to note areas that need updating and itemise new sections required - CONTINUES: Note that this is on hold until after DR1. ACTION: RGM to investigate further any funding opportunities from elsewhere (i.e. AG and IfA) - CONTINUES Specific points and new actions: -------------------------------- Project management: Progress charts for June and forward plan for Q3 will be reviewed next week. JPE/JMS are arranging for a VDMT meeting in July. WFCAM update: Nothing new to report this week Comments and issues arising from CASU fortnightly minutes: No new minutes as of this morning. Networking: Nothing new this week. WSA Operations: LAS is now ready for release; system backups have taken place as normal; GPS source merging is proving to be extremely time consuming and has been chopped into two halves to get around the power interruption scheduled for tomorrow (see below). NCH noted that final quality control (QC2) for the UKIDSS LAS and GCS indicated that the quality and uniformity of the DR1 releases of these surveys looks superb. Hardware: A scheduled power-down in the server room C1 will result in all WSA servers being offline during the day, Saturday 1st July. This creates a rather unwelcome interuption in curation activities immediately prior to DR1 release, but consists of essential maintenance to install a single UPS for the entire room. JB noted that he had new power consupmtion stats; PMW suggested that these should go on the relevant TWiki topic pages. Software: ETWS reported integrating the old SSA stored procedures and functions into the WSA schema and browser, along with miscellaneous installation work. NJC reported much testing of DXS/UDS related software and procedures. RSC reported: "I've developed a new command-line interface tool, which makes the process of adding/processing command-line arguments for your Python scripts even simpler than it was before. It also gives every script a consistent, robust command-line interface, including a fully detailed "--help" option, with things like argument ranging checking automatically performed behind the scenes. The full description of what it does, and how to use it (both for operators and developers) is on the TWiki here: http://apache.roe.ac.uk/twiki/bin/view/WFAU/CliTool Also, I've finally got around to putting my helper scripts for schema updates / database patching into CVS, having made them user-friendly enough and robust enough for anyone to use. I've got document a on them on the TWiki here: http://apache.roe.ac.uk/twiki/bin/view/WFAU/HelperScripts There's UpdateCatData.py for updating the catalogue data in the database for a given date if an earlier ingest has screwed up (which has happened at least twice before requiring this script to fix it all up). Then there's UpdateSchema.py which will automatically update your database's schema to reflect the current schema as described by the .sql files, and will update every entry in the database for these new attributes from the FITS files header values. Also there is an option to specify a specific attribute to update, in which case all of its entries will be updated from the FITS file values. There's also the UpdateMoon.py script to fill the moon position/phase attribute entries, which I can't imagine we will need to use again, as Cu3 handles this now, but it's there for posterity anyhow. Developers, if you would like to see more detailed documentation on the design of the command-line tool, please read the epydoc documentation. The purpose of this documentation is to obviate the need to read the code directly to figure out how the code works. In principle, if the documentation is good enough you should never need to read a module's code directly. For this reason I'm maintaining a reasonably up-to-date version of the code documentation on-line here: http://www.roe.ac.uk/~rsc/wsa and have provided a multitude of links to it on the TWiki at convenient locations to encourage us to make better use of the epydoc documentation." Survey Data Release: Reiterating the DR1 preparation schedule as it currently stands: DR1 release at: 14/7 (NB: databases to be DR1 & DR1PLUS !!) Copying/transfering/ 7/7 backups etc. start UKIDSS pre-release 1/7 checks start Final CUs: 16 (incl. 23/6 SDSS DR3 hopefully), 18,19 start CUs 2,3,4,7 for DXS 20/6 and UDS start Archive QC2 starts 14/6 CU7 for wide/shallow 7/6 surveys starts We're ahead on some stuff, but behind on some as well. Status as at today (30th June): LAS, GCS & PTS: done and ready for final indexation then release. GPS: Source merging taking rather a long time, and being interupted by the scheduled power-down in C1 this weekend. DXS: Source extraction finalised as previous CASU-extractor version (the newest one needs further testing as it seems to be producing erroneous skyLevel and skyNoise estimates under certain circumstances) along with scaled pixel values to shift sky noise well above 1 ADU. UDS: Final enhanced depth/quality mosaics anticipated to be delivered on Monday by the Nottingham group. A rough schedule over the coming week was sketched out with regard to the deep stack surveys: UDS DXS source extraction Tues end of today ingest Wed Mon CUs 7, 16 & 18 end of week Tues documentation review Today --- Time is running short for the GPS source merging and indexing, which will happen in parallel with the above. Non-survey Data Release: No time even to finish typ Astrogrid deployment: Ditt Miscellaneous: Nothing else this week.