From nch@roe.ac.uk Sat Oct 11 16:53:18 2008 Date: Fri, 10 Oct 2008 17:20:07 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 10/10/08 Minutes of WFAU VDFS Science Archive meeting: 10th October 2008 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RPB, RSC, NJC, ETWS, MAR, PMW Apologies: JPE, JMS, AL, BCW, RGM, MSH, LGR DoNM: 10am, Friday 17th October 2008 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: NCH to make ProgrammeFrame assignments for all non-survey projects Discharged with help from RSC ACTION: MAR to link the existing DR3 mosaic in place of the old one on the archive pages, and to look into a DR4 version if necessary. Discharged; DR3 mosaic is linked and DR4 mosaic is nearly ready to replace it...! Actions partly discharged but continuing: ----------------------------------------- None this week. Actions carried forward from 03/10/08 meeting: ---------------------------------------------- ACTION: NCH to break everything by deprecating the loadServer DbConstant so that curations codes can be generalised to work for both VSA and WSA... but not until the WSA non-survey SW has been rolled out. - continues Specific points and new actions: -------------------------------- Project management: PMW distributed the updated Q4 plan, and noted that a couple of items have carried forward into the draft Q1'09 plan. WFCAM & VISTA updates: Nothing new to report this week. Comments and issues arising from CASU fortnightly minutes: With reference to last week's minutes, MJI had clarified that the new MSB reject status keyword is along the following lines: MSBFUD = 'Observation ****ed up in some way' / CASU added-value keyword so the team will implement for 08B onwards as follows: ACTION: NCH to assign a new deprecation code for rejected MSBs ACTION: RSC to update the WSA Multiframe schema with a new varchar(128) attribute which defaults to NONE ACTION: NCH to update the generic non-survey QC script to set deprecation on the basis of the above ACTION: MAR to include reject deprecation in the survey QC script(s) On the subject of 08B enhancements, NCH noted that a new filter is present in WFCAM which needs to be poked into the database ACTION: NJC to update the WSA schema with the new filter details Networking: ETWS noted: - The transfer of VSA data with CU1 worked well. Transfer rate was 16.8MB/s. - I've also extracted the FITS keywords from the different file types and put them on a TWiki page. ETWS will be making a first-pass parse of the FITS keys into a baseline metadata schema for the VSA, and all are encouraged to follow this and make comments/suggestions: ACTION: All to comment on the development of the VSA metadata schema as described on the relevant TWiki page WSA Operations: RPB noted that the DR4 phase 2 outgest continues to grind through the GPS. RSC and NCH noted that if it fails again, then code enhancements to robustify are nearly ready. PMW enquired about backups, and RPB noted the TWiki backup policy doc, and the fact that routine weekly diffs and monthly full backups are done for the catalogue servers. MAR reported that a note has been placed on the archive websites concerning service interuptions from further power outages on the 18/19 Oct as the site power infrastructure is fettled for the new server room. Hardware RPB noted that the Sun ZFS system appears to be too difficult to optimise for immediate use, so we will continue with our original plan on fibre-linked catalogue storage employing home-brew SAN built upon hardware RAID. PMW noted that he and MSH have been looking at upgrades of the oldest catalogue server JBODs from 150GB to 300GB disks to cover expansion and modernisation of the older archive equipment now that it is out of warranty. Software: RSC noted: Prepared WSA for the transition between the old and new regimes for non-survey curation: - removed duplicate entries in ProgrammeFrame with default programme ID values - Revised Programme.catalogueSchema entries and updated code for changes - All new programmes now have detection tables and schema in SVN following successful first test of the new CU21's automatic commit to SVN feature. Some further deep non-survey release testing and documentation updates, but had little time to work on this following CU19's problems with the gpsDetection outgest. After many diagnostics and experimental fixes it was decided to be a spurious error that could not be mitigated. So the plan is to now split up large detection table outgests in CU19 to make the process more fault tolerant. The software is almost ready to do this and development is discussed in a new trac ticket #130. NJC noted further testing and development of synoptic features, while MAR noted some work on the UI for non-surveys. Survey Data Release: As noted above, the GPS DR4 release continues to be a top priority but is yet to be released owing to load server struggling with the sheer volume of data. Software mods are in hand to make things easier; keep your fingers crossed... NCH noted that the DR4 UDS is still a few weeks off being delivered, but no biggy seeing as it's a relatively tiny amount of data that can be appended to the DR4 release DB. NJC noted that all will be ready to include new synoptic features for the UDS and DXS in DR5. NCH has been in communication with SJW concerning a possible release schedule for DR5, which will incorporate survey data to 08A. Obviously it would be ideal if this could be released in advance of the December science meeting; the team agreed that this is no problem at all for the LAS/GCS/DXS (especially as eyeball QC is already complete for the latter 2). Potential GPS volume issues would be OK if the source merging could be done in append mode, but it turns out that the survey head wants recent contemporaneous JHK to be merged into JHK1 while any older K-only observations in those fields currently assigned to K_1 should appear as K_2. While not unreasonable, this requires those frame sets to be identified, expunged and remerged/ reseamed, so it's not so straightforward and NCH expressed the opinion that we shouldn't commit to having this redone by December. Anyway, we will do what we can... Non-survey Data Release: Final testing of new regime on hold until GPS DR4 is out of the door... Astrogrid deployment & Data Analysis services Nothing to report this week. Miscellaneous: Nothing else this week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================