From nch@roe.ac.uk Fri Sep 26 12:48:22 2008 Date: Fri, 26 Sep 2008 12:28:26 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 26/09/08 Minutes of WFAU VDFS Science Archive meeting: 26th September 2008 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RPB, RSC, NJC, LGR, ETWS Apologies: JPE, JMS, AL, BCW, MAR, PMW, MSH, RGM DoNM: 10am, Friday 3rd October 2008 in the Vista Hut Actions discharged this week: ----------------------------- None this week. Actions partly discharged but continuing: ----------------------------------------- None this week. Actions carried forward from 12/09/08 meeting: ---------------------------------------------- None this week. Specific points and new actions: -------------------------------- Project management: Nothing of note this week. WFCAM & VISTA updates: ETWS gave a debrief from the ESO phase-2 Workshop. Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: ETWS noted that he and PSB now have a new connection set up for VIRCam products, and furthermore PSB has made available some processed data for us to copy up and play with. See Software below for notes on the resulting discussions. NCH and NJC noted that we're still waiting to hear from St. Andrews about if/when they plan to bring a portable disk for some bulk copying of image data. WSA Operations: RPB reported: "Started testing the new Sun SAN. Mark has set up sit different RAID arrays with different settings, so we want to find out which has the best performance. Put a copy of the TWOMASS database on each and started running a trawl through them. Results have been hampered by the fact that Hatshepsut is a public facing server and the SSA has seen a lot of activity this week. GPS nearly ready for release! Neighbour tables finished last night, so just CU19 to run. Best to wait 'till Monday morning given the power outage in C1 over the weekend. Fixed BIOS problem with Hatshepsut after Mark installed the Fibre Channel card. Updated to the latest BIOS and everything seems fine now." As noted above, there will be a power outage in the server room this Sunday, so a differential backup will take place as usual this weekend and then ops are on pause until Monday morning. NCH has put a note on the archive downtime pages. Hardware RPB noted initial experiments with the Sun fibre-linked disk array are proving difficult to benchmark given incoming user queries on the SSA, but anyway something's fishy as the trawl rate for a 200GB test DB is appallingly slow. NCH suggested a quick check on the local copy trawl rate for 2MASS on the server to provide a zeropoint benchmark. Software: RSC noted: "Revised automatic non-survey programme setup and schema creation to allow it to be smoothly integrated into the post-CU3 and non-survey release scripts. Ensured that all the steps from the old CU21 scripts are still followed, and removed these old scripts as they are no longer compatible with the new automatic regime. Restored the capability to regenerate all non-survey schemas according to the current Programme table state following a template change (e.g. documentation) in the updated nonsurveyparser script. Also reverted it to creating a single index schema file for all non-surveys, so that the rest of the software doesn't need to be modified. Prepared non-survey release script to automatically select all newly registered non-surveys for release by default with options to refine the programme list in various ways including a re-release of all registered programmes. Performed initial tests of deep non-survey releases, which requires further development of the automatic curation method and CU13/14, including the ability to prepare a single file list for the deep products from all non-surveys. Also helped MAR to get started with the colour thumbnail curation script." NJC reported fixing the automation in CU6 with some fixes for mixtures of deep and shallow stacks in CU7, and also working on catalogues for GAMA. The team had a long discussion about enhancements to make all the SW work for both WSA and VSA. The only sticking point is some old static parsed script constants, and it was agreed that these should be deprecated in the SVN trunk and then we simply work through the inevitable fall-out on a case by case basis. ETWS noted that he has already fettled CU1 so that it can copy up the VIRCam test products. ACTION: NCH to break everything by deprecating the loadServer DbConstant so that curations codes can be generalised to work for both VSA and WSA... but not until the WSA non-survey SW has been rolled out. Survey Data Release: As noted above, the GPS DR4 is ready to be copied into a new phase2 release DB. But as this will take several days, and because of the power outage on Sunday, this will have to wait until Monday to be started as it can't be interupted. Non-survey Data Release: RSC noted that he hopes to get on to the final major piece of SW enhancements (CU7 single-passband source "merging") next week. Astrogrid deployment & Data Analysis services LGR has been communicating closely with users at Imperial concerning their requirements for list-driven functionality, and is looking into implementing an intra-UKIDSS facility employing the list-driven photometry tool provided by CASU. Miscellaneous: Nothing else this week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================