From nch@roe.ac.uk Fri Nov 21 15:35:28 2008 Date: Fri, 21 Nov 2008 14:41:03 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 21/11/08 Minutes of WFAU VDFS Science Archive meeting: 21st November 2008 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, MAR, ETWS, RPB, NJC, RSC, PMW Apologies: JPE, JMS, AL, BCW, RGM, MSH, LGR DoNM: 10am, Friday 28th November 2008 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: MSH to check with ITSG that WFAU server moves can take place early in the new year. Discharged; this is fine as far as ITSG are concerned Actions partly discharged but continuing: ----------------------------------------- None this week Actions carried forward from 14/11/08 meeting: ---------------------------------------------- ACTION: NCH to break everything by deprecating the loadServer DbConstant so that curations codes can be generalised to work for both VSA and WSA... but not until the WSA non-survey SW has been rolled out. - continues ACTION: All to comment on the development of the VSA metadata schema as described on the relevant TWiki page - continues ACTION: ETWS to add a delta-mag attribute to the VSA template detection table schema. - continues Specific points and new actions: -------------------------------- Project management: Nothing to report this week. WFCAM & VISTA updates: Nothing to report this week. Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: Nothing new this week. WSA Operations: RPB noted that he has tidied up some extra jpegs in the 08A data, having written a script to automatically delete any duplicated (in perl, much to Eckhard's disgust). RPB will attempt to be assimilated into the python world by recoding it in a clearly superior language. MAR has worked through a substantial chunk of QC1 for DR5 for the DXS, GCS and LAS. MAR noted that some retrospective QC is now necessary (particularly for important for the GPS) since odds and sods of reprocessed data have appeared in previous semesters. The automated, scripted part of QC is being checked retrospectively as 08A is being worked through; MAR plans to apply also the depth cuts to weed out a few inevitable grossly discrepant frames that no amount of reprocessing can ever fix. NCH noted that he has briefly looked at checks on manual (mainly eyeball) deprecations for reprocessed data by comparing lists of newly ingested versus their old, deprecated counterparts. Checks on the GCS have shown no offending frames, but another check for the GPS clearly shows up a handful of examples that must be redeprecated manually. Simple script fragments will be added to the cross-semester QC script for DR5 for this purpose. ETWS reported: - 08B Septemper data has been transferred and ingested successfully. - 08A catalogue data has been ingested but for 4 days of GPS; the ingest will finish later today. - I've written a thumbnail creator that will create web pages to check CU2 more easily and might be helpful for quality control as well. - Lists for all non-survey catalogues (for all semesters up to 08A) that haven't been ingested yet have been created and will be processed as soon as the 08A catalogues are ingested. RSC noted that we need to refettle any new 08A non-survey programmes that are present in the new batch (September). ACTION: NCH to rerun the non-survey assignment script for 08B. Hardware: RPB reported: "Endless problems with ahmose continue. It's like deja-vue [sic] all over again. One of the jbods on ahmose was reporting a fan error. Ian from Eclipse came out with some replacements to fix it, as well as the fan error that had been showing on hatshepsut for quite a while. Hatshepsut was fixed quickly and easily. Ahmose, however, upon reboot reported a failed disk. The disk was rather tricky to pin down as it seemed to be interfering with communication to other disks, but eventually it was replaced and a RAID rebuild started. At least this time operations were not unduly affected (apart from having ahmose running slow while the array was rebuilt). Raised a support ticket with IT support to deal with a degraded array on djedefre. IT support reports that everything is fine, so need to check out and badger them if necessary." ACTION: RPB to badger ITSG to sort out the RAID maintenance SW on the Linux file servers. ETWS noted that we start to think about getting the next file storage NAS box in to ensure sufficient space over the next few months of ops. ACTION: PMW to get a quote for a new NAS box a la takelot. Software: RSC noted: "Merged commits from release_7 into trunk, which took longer than usual due to a complicated mix of partial merges over the past 4 months that the branch has existed. I recommend that in future we try to keep trunk up-to-date with the latest changes to the release branch, even if it means the software in trunk is broken more often. It's quicker and easier to fix these problems at the time, and will allow creation of new release branches to be as easy as just pressing a button (once testing is complete). I've completed testing of the integration of these release_7 changes into the automated non-survey release procedure and took the opportunity to do further non-survey release tests. I'll create the new release_8 branch today, once the last few changes to release_7 from this week have been merged into trunk." ACTION: RSC to create the new release_8 SVN branch NJC noted that he has sped up deprecation of old deep stacks during testing. Survey Data Release: NCH noted that following various discussions with MJI, SJW and Seb Foucaud (UDS), the agreed shedule for DR5 is LAS/GCS/DXS before the end of the year and UDS/GPS to be included early in the New Year. MJI has agreed to reprocess a bunch of 05B data for the UDS by the end of the year, and this will need to be retransfered/reingested asap after Xmas to make sure the provenance and intermediate stack synoptic products are correct for the UDS in DR5. Non-survey Data Release: ETWS noted that final checks of CU4 ingests up to and including data semester 08A will take place next week, to be followed by an end-to-end test of the new support regime for survey-like DB products for a few example programmes prior to a full production run. Astrogrid deployment & Data Analysis services: No news this week. Miscellaneous: Draft programme for Science from UKIDSS II is out - see http://www.ukidss.org/rasworkshop08/talks ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================