From nch@roe.ac.uk Tue May 26 13:21:45 2009 Date: Fri, 22 May 2009 17:16:38 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 22/5/09 Minutes of WFAU VDFS Science Archive meeting: 22nd May 2009 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, NJC, RGM, RPB, RSC Apologies: JPE, JMS, AL, PMW, MAR, MSH DoNM: 10am, Friday May 29th 2009 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: NCH to make polite enquiries concerning grant renewal status to STFC attendees at the UKIRT Board meeting next week Discharged; STFC representatives could give no indication of when the situation might be resolved; the UKIRT Board will be noting this unsatisfactory state of affairs in their minutes. Actions partly discharged but continuing: ----------------------------------------- ACTION: ETWS to parse that latest keyword info into the VSA metadata schema files. Continues; the team noted in the latest CASU minutes that draft headers for tile products are being considered now. Actions carried forward from 15/05/09 meeting: ---------------------------------------------- None this week Specific points and new actions: -------------------------------- Project management: NCH attended the UKIRT Board meeting in Nottingham yesterday; as noted above, no update on RG renewal from the research council. WFCAM & VISTA updates: NCH noted that JPE is in Chile at the moment, and anticiptes that "surveys will begin all too soon" so it's time to reinstate VDMT telecons and take a close look at readiness of the system - fine by us (unless we've all been made redundant in the interim...) Comments and issues arising from CASU fortnightly minutes: The team noted the minutes of the meeting of 18th May; thanks for the reprocessing updates: GPS: NCH noted that the original "250 or so" catalogues for reprocessing has proliferated somewhat to around 600 (!), and that the reprocessed data actually consists of image/catalogue pairs where the image is a filtered version of the usual stack. Given an agreed suffix (NCH couldn't remember what MJI suggested when chatting at the Board meeting yesterday), the team agreed that WFAU can define new "frametype"s in the archive (say filtstack and filtleavstack), ingest as normal, with mods to: CU3 (auto-deprecate corresponding unfiltered image/cat pair); possibly CU7 (but we think it picks up *stack for source merging so no probs); and CU19 (to specifically include the unfiltered image in the release because sooner or later some users are bound to want to look at and/or compare with the filtered version). UDS: looks like Nottingham will deliver the new UDS K stack in a couple of weeks or so; NCH suggested to the Notts folks that WFAU ideally would like to have the corresponding intermediate stacks so that the new synoptic products are available in the DR5, and furthermore that the provenance is correctly tracked etc. UDS Survey Head Omar doesn't want any more delays than are absolutely necessary for the release, but ETWS suggested that we should be able to copy up UDS (and GPS) reprocessed products over the next few weeks once released to us, in the small gaps between routine 09A transfers. Regarding transfer of these reprocessed products, ETWS noted that he's off on hols now until June 4th, so no problems on the final checking of both GPS/UDS data next week before invoking any transfers. Concerning the mechanics of transfering, the GPS data presumably can be inserted in the usual nightly directories as there are no filename collisions, and provided we have a list we can grab them easily enough; the UDS data are a little more complicated since no file-by-file versioning has been implemented at either end... we're open to suggestions from CASU on the best way to handle this. Networking: Nothing new this week. WSA Operations: RPB noted: "DR7 split continues. The problems inserting data into some tables turned out to be "Identity" columns. Basically this is SQL Server's way of implementing auto-incremental columns. Since we're not interested in auto-incrementing columns as we're never going to add data to these tables, I simply dropped the half-dozen or so affected tables and recreated them without Identity columns. Data is now transferring without problems. GPS preparation for DR5. The Quality Bit flags had to be rerun prior to reseaming and source merging. This finished last night, so we're now clear to kick off the reseaming and source merging to run over the weekend. IT Support have finally fulfilled the support ticket to change the root passwords on all WFAU linux servers. We now have control!" ETWS noted: - Transferred and ingested the available 09A data from Jan to April. - Set up the monitor webpages for 09A. - Send a list of broken files to MJI and got back the fixed ones immediately. Thanks, Mike! - Fixed some installation problem with the Cirdr code, MakeMaker seems not to accept self-defined environment variables. The code is available for testing on djoser and khafre. Hardware: All systems functioning normally (for a change). Software: RSC reported: "Put the new skyAlgorithm column into WSA.Multiframe ready for 09A ingests. Helped ETWS by checking the release_10 branch to spot any more bugs introduced in that release. Updated the code to handle changes to the SVN certificate and also included a new feature to warn the user if their working copy is outdated. Completed tests on the changes to CU7's frame set association algorithms for the multi-epoch GPS requirements. Finally solved the issue over whether CU7 can be safely interrupted in append mode - turns out we can, but must ensure the begin date option is not supplied. Also, took the opportunity to investigate the scaling issues with the frame set association algorithm and discovered that the simple algorithm isn't the most the efficient one. However, it would require a complete re-write, so I'm leaving it alone whilst it's not too slow." NJC reported: "This week, I submitted a paper on "Archiving multi-epoch data", which describes all the new variability tables and how to use them. http://xxx.lanl.gov/abs/0905.3073 I have also worked on ProgrammeBuilder and cu13_stack so that they use the same matching criteria. This will avoid a few situations where non-surveys have failed. As part of this, the stacking software now uses data where some extensions are deprecated. This part has been tested. The rest will be tested in the next day or two. Finally I have built a test script for the CASU extractor and have started testing the latest version." Survey Data Nothing else to report concerning DR5 preparations. Non-survey Data Release: Nothing new this week. Astrogrid deployment & Data Analysis services: Thanks to Richard McMahon who discovered a bug in the beta test version of the intra-WFCAM list-driven photometry tool (the sign of the extinction correction was wrong - ETWS is fixing as part of his review of the code). Miscellaneous: RGM asked that folks steer clear of remote logins to public server thutmose next week, as he's got a bit of research time and wants to hammer that particular machine. INACTION: All to steer clear of SQL Server thutmose next week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================