From nch@roe.ac.uk Mon Dec 8 15:30:35 2008 Date: Fri, 5 Dec 2008 16:09:42 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 5/12/08 Minutes of WFAU VDFS Science Archive meeting: 5th December 2008 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, MAR, ETWS, RPB, NJC, RSC, PMW, MSH, LGR Apologies: JPE, JMS, AL, BCW, RGM DoNM: 10am, Friday 12th December 2008 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: PMW to check that we have a rolling weekly booking for the Vista Hut meeting room. Duscharged; meetings to be held in the Vista Hut as normal for the foreseeable ACTION: RPB to run FlatFileFinalise as appropriate for recent ingests Discharged (a lock-out problem solved immediately following the meeting) ACTION: RPB: Make sure the cron job behaves as required, then request ITSG amend the cron job to email results to ETWS and MSH, as well as rolling it out across the rest of the linux servers. Discharged: turns out there are some teething troubles so RPB is liaising with ITSG over these. ACTION: RPB to check out available space per LV on SDSS catalogue host thutmose Discharged by RPB: "Checked free disk space on thutmose. There is a grand total of 3.4Tb free with the emptiest four drives being N (1.45Tb), G (880Gb), L (415Gb) and J (220Gb)." Actions partly discharged but continuing: ----------------------------------------- ACTION: PMW to get a quote for a new NAS box a la takelot. Continues; 3 quotes now in with one further to come. ACTION: NJC to check out the incremental changes in DR7 over DR5 ACTION: ETWS to check out availability of SDSS-DR7 split over several files/groups/volumes All continue; NCH suggested it would be good to have all the facts in time for the UKIDSS meeting as users will want to know the upgrade plans. Actions carried forward from 28/11/08 meeting: ---------------------------------------------- ACTION: NCH to break everything by deprecating the loadServer DbConstant so that curations codes can be generalised to work for both VSA and WSA... but not until the WSA non-survey SW has been rolled out. - continues ACTION: All to comment on the development of the VSA metadata schema as described on the relevant TWiki page - continues ACTION: ETWS to add a delta-mag attribute to the VSA template detection table schema. - continues Specific points and new actions: -------------------------------- Project management: PMW noted a little progress on the grant renewal from STFC in that we've been invited to put in a new JeS; he asked for a quick get-together with NCH and RGM at 12pm today. WFCAM & VISTA updates: Nothing to report this week. Comments and issues arising from CASU fortnightly minutes: The team noted the minutes of the meeting of 27th November. With reference to the Section 7 (VISTA update) we note the issues regarding keywords and WCS etc. In order to check on any compatibility quirks with software at this end, we would like to see some pipeline processed data products asap - just let ETWS know when some new stuff is available for transfer and we'll haul it up to have a play. Networking: ETWS noted - Transferred 08B October data successfully and ingested metadata (CU3). WSA Operations: ETWS noted: - CU4 for 08A GPS has finished. - Helped LGR debugging magnitude calculation code. - Catalogue data for all non-surveys are ingested and checks are continuing. RPB reported: "GCS processing pretty much finished. Quality bit flags have been updated, reseaming and source merging completed, and neighbour tables created. DXS continues. There were a few teething troubles with the Autocurate script which Nick managed to track down. So far it has updated the errbit flags, created the deep stacks, and finished the provenance filler." NJC has been debugging the DXS synoptic curation scripts; NCH has checked out the GCS for release in DR5. Hardware: MSH reported that there is no news on the migration of core servers to new server room C2. MSH also noted that he has been investigating time synchronisation of the NAS boxes. Software: RSC noted: - Solved the problem of single passband source "merging" and reseaming, which is the last software hurdle for automated non-survey releases. - Enhanced the SyncDb script to exclude non-surveys from world release that have extensions on their proprietary period; and updated NonSurveyCuration documentation. - Minor improvements to auto-curation codes following DXS DR5 experience and created a new AutoCurate TWiki page to document how the script should work. - Started testing our software on Khafre following the recent mxODBC and Python upgrade. The changes required to support Python 2.6 will be quite extensive since all external process calls will need to use the new subprocess module. As we will also need to make extensive changes to support the VSA, I suggest we also work on the VSA support now, and do all the testing in one go. RSC asked for the development and ops team to get together for a VSA software preparation meeting this afternoon at 2pm in the Vista Hut. MAR reported: "Altered the multiGetImage interface to produce the necessary output and call LGR's list driven code. Also looked at altering getImage for single coordinate list driven input but this is not so easy to re-work." Survey Data Release: The GCS is now ready for DR5; the DXS is undergoing it's final synoptic table generation; we are awaiting eyeball QC results for the LAS. NCH suggested that QC1 for the GPS should be progressed as far as possible in the mean time, noting that there is a small amount of H2/K data in programme 07B/D3 that should be auto-QC'd as well (they are allocated to the GPS in ProgrammeFrame but won't be picked up in the standard QC SQL scripts). Non-survey Data Release: As noted above, RSC has closed out the final remaining issue on non-survey support (single passband source "merging") so end-to-end tests of a few example programmes can now take place. Astrogrid deployment & Data Analysis services: LGR noted: "Comparing executions on horus vs sneferu spotted a float rounding issue (with python 2.6 -- thanks ETWS), so indeed CASU's code returns coords within a fraction of a pixel. However, the 1%-order discrepancy still remains. MAR arranged multigetimage to provide filenames for the photometry tool, so the interface is being completed soon and posted within the beta version. Unfortunately the SDSS photometry tool does not work anymore following recent SDSS webservice upgrades." Miscellaneous: Nothing else this week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================