From nch@roe.ac.uk Sat Sep 13 10:26:30 2008 Date: Fri, 12 Sep 2008 15:42:19 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 12/09/08 Minutes of WFAU VDFS Science Archive meeting: 12th September 2008 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RPB, RSC, NJC, MAR, PMW, MSH, LGR Apologies: JPE, JMS, AL, BCW, RGM, ETWS DoNM: 10am, Friday 26th September 2008 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: RPB to formulate backup plan arising from meeting Discharged; TWiki pages provide details. ACTION: NCH, NJC and ETWS to fold SV survey info into the VSA baseline schema. Discharged; first-pass baseline schema now accessible at http://horus.roe.ac.uk/vsa/www/vsa_browser.html (yes we know there are a lot of WSA details still in linked pages...) Actions partly discharged but continuing: ----------------------------------------- None this week. Actions carried forward from 29/08/08 meeting: ---------------------------------------------- None this week. Specific points and new actions: -------------------------------- Project management: The team filled in the VDFS progress charts for August. WFCAM & VISTA updates: NCH noted a communique received from Andy A at JAC: "Note on timetable. Unless anything goes further amiss, we will be starting operations with WFCAM next Wednesday [17th]. Two nights of engineering and alignment, a third possibly UKIDSS, and then we're into operations again full time." Comments and issues arising from CASU fortnightly minutes: The team noted the minutes of the meeting of 3rd Sept. Regarding the question of GPS release time-lags and world release, see below under Survey Data Release. Networking: NCH noted that, after some negotiations with interested parties at St Andrews, we have agreed to bulk copy several 100 GBs of LAS images in the MGC area onto a USB2.0 portable hard disk (to be supplied by them) so that they can do their own extended source photometry consistent with their optical MGC catalogue. For some reason they seemed reluctant to use a wget script for bulk copying as several others are doing. WSA Operations: RPB noted: "Finished the backup documentation, though at NCH's suggestion, will add a table listing all surveys along with when they were last backed up. CU7 continues. Reseaming started yesterday morning. Contacted eclipse about possibility of removing one LTO tape drive from Overland to use as an external. Apparently the lack of a front panel may make that impossible. May have to pinch existing LTO drive from djoser, or order new hardware." RSC noted that for DR4 he tracked down and fixed a rare bug in CU7 that only occasionally occurred when frame sets were updated, which was discovered by RPB when it interrupted GPS source merging. RSC has improved CU7 tests to hopefully spot this kind of bug in future. Hardware MSH noted that he has taken delivery of a 48TB storage box from Sun to try out some proto-SAN tests. MSH and PMW have ordered a fibre interface card for hatshepsut to put the unit through it's paces, although MSH cautioned that the SW set-up of the unit is less than straightforward. NCH suggested PSB may be able to advise as we understand IoA may have some similar Sun hardware. In order to avoid china-syndrome in C1, the team decided to shut down the rather poorly khafre as it currently is not being used owing to crapulent Debian4.0 64-bit Python behaviour. Scottish Power are still giving ITSG the run-around regarding the site power infrastructure, so it is still a little uncertain if/when there may be significant downtime in archive services. Knowing our luck, everything will be switched off the minute the GPS DR4 goes live; our path is strewn with cow-pats from the Devil's own herd... Software: RSC noted for: Non-survey automation: * Implemented in CU7 frameset association between deep stacks and intermediate stacks from filters that do not contain any deep stacks. * I've redesigned the interface of the automated curation part of CU6 to allow it to be integrated into the non-survey release script. Tested successfully with a shallow non-survey, but still need to improve some ragged edges like the logging as well as completing a deep non-survey test. * Simplified non-survey template now that table creation is automated, and am making sure that new schemas are consistent with the old schemas. Also now continuing to make automatic programme setup stage robust and integrated into the non-survey release script. Misc: * Robustified metadata mirror update against schema evolution discrepancies. * Updated CU16 to allow test databases to use the WSA's skinny SDSS flag tables. MAR noted new work on colour image production, with the following examples: LAS near NGC objects YHK http://surveys.roe.ac.uk/data/mar/ngcNew/ GCS "some interesting" ZJK http://surveys.roe.ac.uk/data/mar/neb/ GCS random framesets ZJK http://surveys.roe.ac.uk/data/mar/gcs/ The team congratulated Mike on these striking images. The plan is now to automate production of colour jpegs for all defined framesets in the LAS, GPS and GCS, and then to provide new interface facilities to produce on-demand cut-outs and mosaics. NJC noted doing some support work for the GAMA collaboration, and also further testing of non-survey automation script CU6. NCH also noted that he has tested out the automated set-up of non-survey curation along with CU6 for a few specific projects, and it all appears to be working very well. This should make release of non-survey DB products much easier and quicker for the operations side. NCH thanked RSC and NJC for all their hard work on these developments. Survey Data Release: MAR noted that UKIDSS DR2 has gone world public. With regard to the question raised by both Mikes R and I, the default position is to world-release 18 months after the first phase proprietary release, i.e. not to worry about the time lag in getting the GPS out; the UKIDSS Survey Heads, including GPS SH Phil Lucas, seem to be in full agreement on this. Non-survey Data Release: RSC reminded everyone that we need to keep an eye on when to release semester's worth of non-survey data into world flat-file access. Nothing else to note this week other than SW modifications and testing continue in advance of roll-out of the new non-survey support regime. Astrogrid deployment & Data Analysis services MSH noted that UKIDSS DR2 is now world VO-accessible through the DSA for catalogues and SIAP for images. MSH also suggested a get-together next Tuesday (am) to discuss priorities/position for the up-coming Cambridge VOTCUK meeting. LGR noted that he, PMW and RGM are getting together next week to discuss roll-out of the new archive pixel data analysis facility. Miscellaneous: SCIENCE FROM UKIDSS II The second 'Science from UKIDSS' workshop will be held at the Royal Astronomical Society, Piccadilly, London, 15-17 December 2008. The workshop aims to bring together, in an informal atmosphere, astronomers who are working with UKIDSS data, to hear about science in progress, to share knowledge about exploiting the UKIDSS archive, and to foster collaborations. Registration is now open at http://www.ukidss.org/rasworkshop08/ The deadline for offering to give a talk is 1st November. We recommend booking accommodation early (see the link on the web page). Any queries should be directed to Steve Warren s.j.warren@imperial.ac.uk ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================