From nch@roe.ac.uk Fri Aug 8 14:17:15 2008 Date: Fri, 8 Aug 2008 14:03:28 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: VDFS Science Archives weekly project meeting minutes 8/8/8 Minutes of WFAU VDFS Science Archive meeting: 8th August 2008 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RPB, ETWS, RSC, NJC, MAR, LGR, PMW, MSH Apologies: JPE, JMS, AL, BCW, RGM DoNM: 10am, Friday 15th August 2008 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: MAR and LGR to get together Tues (am) next week to sort out a web interface for the new pixel analysis tool. Actions partly discharged but continuing: ----------------------------------------- ACTION: RPB to test out attaching a normal database as an MS SQL "reporting database" in readonly mode on thutmose/hatshepsut RPB noted that some tests with attaching databases as readonly on read-write OS filesystems were inconclusive; in any case, a spare volume on public server thutmose needs to be used to set up a readonly volume for further tests. Unfortunately thutmose is ill (see below) so this has stalled. Actions carried forward from 01/08/08 meeting: ---------------------------------------------- ACTION: RPB to formulate backup plan arising from meeting Continues; currently wholesale new backups running; formal policy to be documented. NCH noted that he has made a TWiki page with information on the legacy SuperCOS dataset backups, and this could be used as a model. Specific points and new actions: -------------------------------- Project management: Nothing of note this week. WFCAM & VISTA updates: NCH reported back on several communiques from JPE on updates to the VISTA schedule, and also the fact that a Public Surveys workshop has been convened for the 15-17 September (2pm Mon to 5pm Wed). The latter is to educate the PS proposers in the ways of preparing survey OBs, and also time has been reserved for VDFS presentations on processing/archiving. ETWS has volunteered to represent WFAU. NCH noted that it would be good to have the first-pass survey schemas set up by the time of the meeting to illustrate to the users how data will be presented in the archive. To that end, the schema requirements need to be summarised for the various surveys. ACTION: NCH to assemble all feedback from VISTA PSP PIs concerning survey design in a TWiki page The main item of note from the VISTA schedule is that Science Verification will take place in January, and that VDFS will undertake to process/archive those "mini-surveys" as part of the shake-down of the system, and as a (hopefully) useful supporting resource for the SV programmes. Comments and issues arising from CASU fortnightly minutes: No new minutes this week Networking: Nothing of note this week. WSA Operations: ETWS noted - Transfer of the reprocessed wfcam_fud and UDS data has now finished. The deprecated files are stored on khufu's disks. - All databases have been updated with the new paths for the deprecated FITS files. RPB noted removing test DB TWONAS and tidied up the spare files. Hardware RPB reported: "Mounted spare volume on thutmose for reporting database testing. Ian from Eclipse replaced the motherboard on Takelot on Monday. Replaced a disk which had failed. Tuesday, replaced a disk which had failed on ahmose. Rebuilt RAID array. Wednesday, another disk failed on ahmose, as well as one on amenhotep. Replaced both, rebuilt RAID arrays. Thursday Thutmose died. Major hardware error. Refused to boot. Nothing much to be done but call out Eclipse again. Ian arrived Friday morning with spare parts. Left him in Mark's capable hands. Watch this space." RPB, MSH and NCH all noted that the flurry of disk failures are likely due to the recent overheating episodes in server room C1. Regarding batch catalogue server thutmose's problems, NCH noted that this means the AG DSA access to all DBs will be down currently, and apologised to MSH in advance of any complaints from users accessing via that interface. MAR has put a note on the archive downtime to inform users of the temporary loss of SDSS and crossmatch functionality. RPB noted that JNTD has mentioned that all SAN quotes are now in, but the latter has been too busy to forward them on. ACTION: MSH to chase JNTD for final SAN specs & quotes Software: RSC reported: "I've been refactoring CU19 to aid automation of non-surveys by creating the database directly on the public catalogue server (also took the opportunity to support automatic database reattachments, which wasn't possible before). Modified SyncDb to deny public user access to the metadata mirror during an update. Various other improvements and bug fixes to SyncDb for use with test databases, together with an improved regression test procedure to cover 10 different use cases of SyncDb. Also created a DatabaseOptions TWiki page to document the best SQL server options for various types of database." ETWS reported: - Upgraded the compressed image creation code to handle single file deprecation. MAR reported working on rollout of software one the new web server horus. Survey Data Release: NJC discovered a dodgy consituent intermediate stack frame in the DXS EDR, DR2 and DR3 in the week (only significantly affected the EDR). A note has been put on the website under known issues, and NCH will take care of deprecating the offending data after DR4 phase 2 release. Regarding the latter, ETWS noted that all constituent reprocessed data has now been retransfered and that ingestion should be completed quickly to allow forward curation of the GPS from the middle of next week. Non-survey Data Release: NCH enquired as to the current status of the automated non-survey support enhancements. The team agreed to revisit next week, once the recently transfered odds-and-sods have been ingested. Astrogrid deployment & Data Analysis services LGR noted that he is now developing a jsp front-end for his new pixel analysis tool. NCH noted that the failure of batch server thutmose has unfortunately hit the DSA services for WSA and SDSS etc. Miscellaneous: ADASS XVIII reminder (see http://adass.org/) for info. Doesn't look like anyone from WFAU will be attending this time. As far as the VOTC:UK workshop at IoA at the end of September is concerned, attendance will be MSH, AL, RGM (?) and possibly PMW wearing his finance hat. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================