From nch@roe.ac.uk Fri May 28 16:55:55 2010 Date: Fri, 28 May 2010 16:26:38 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren , Tom Kerr Subject: WFAU VDFS Science Archive weekly project meeting minutes, 28/5/10 Minutes of WFAU VDFS Science Archive meeting: May 28th 2010 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RSC, MSH, ETWS, RPB, MAR Apologies: JPE, AL, NJC, KTN, RGM DoNM: 10am, Friday June 4th 2010 in the VISTA Hut Actions discharged this week: ----------------------------- ACTION: ALL to consider who goes to Garching in September for the VST PS readiness meeting. Discharged; ETWS was kindly volunteered (needs the dates from RGM though) ACTION: RSC & NJC to update CU codes accordingly for mandatory date range and shallow VSA curation Discharged Actions partly discharged but continuing: ----------------------------------------- ACTION: ALL to review new hardware requirements/procurement over the next couple of weeks Proposal is for one new curation server and one new server dedicated to web application running (though not necessarily a web server). This will be discussed with KTN next week. ACTION: RSC & NJC to document new operational procedures on the TWiki Rough notes done but needs review by NJC and polishing for ops use. Actions carried forward from 21/05/10 meeting: ---------------------------------------------- ACTION: NCH to fill out a Q2 2010 plan and continue with the progress monitoring for the time being. Continues ACTION: RPB/KTN to order several crates of LTO-4 tapes. Continues Specific points and new actions: -------------------------------- Project management: NCH noted that two important meetings occur next week: RGM, KTN and MJI are visiting ESO to continue negotiations on in-kind contributions for VISTA; while NJC and MAR will have a teleconference with the PSPIs to discuss archive-end things viz. QC, releases, archive curated products and user-generated products. NCH noted that an issue related to both meetings is the question of delivery of products to the ESO-SAF, where there is a potential snafu in that ESO are insisting that the PIs load stuff piecemeal through a manual interface, whereas the PIs may be assuming that WFAU will take care of this as originally proposed via an automated dataflow system solution. The team discussed the pros and cons of taking a pro-active and pre-emptive approach in sorting this out, and it was agreed it would be to everybody's advantage if WFAU escalated the issue, starting with a discussion with the PIs at next week's teleconf. The idea is to hold the PIs hands in the writing of an MOU that could be negotiated between them and ESO-SAF ... ACTION: NCH to brief NJC and to get included an agenda item on ESO-SAF data delivery WFCAM & VISTA updates: Nothing of note this week. Comments and issues arising from CASU minutes: No new minutes this week. Networking: April (P85) VIRCam data (process version v0.8) is being transfered from CASU to WFAU; some fixed files (v0.9 mixed in with v0.8 directories?) have also been transfered up. ETWS noted that for maximum flexibility he intends to tweak things slightly at this end to cope with nightly directories having a mixture of non-integer process versions, just to avoid any confusion. ACTION: ETWS to enhance the early CUs to cope better with CASU's process versioning CASU have released some prototype tiles to WFAU for ingest tests. ACTION: ETWS to try some test ingests of VIRCam tiles. WSA/VSA Operations: RPB reported: "Vista curation continues. A new VIKING database was created in the absence of the ops team by NCH, and released this week. VMC is proceeding with synoptic processing. VVV caused a few problems by filling up the disk used for ops tasks when trying to create a neighbour table with UKIDSSDR4 GPS data. Much moving around of data freed up plenty of space on this disk and VVV has now also moved on to synoptic fun. The latest WSA data (April 2010) was transferred up during the ops holiday by the hard working NCH, and this is now being processed." MAR asked if the daily cron-job mirror update for the VSA was now working - RPB noted it's still switched off ACTION: RPB to reinvoke the VSA metadata mirror cronjob MAR noted that GPS DR7 QC is as up to date as possible; now awaiting availability of reprocessed nebulous regions before continuing with GPS DR7 preparations. Hardware and Systems: MSH noted: "The new curation server shepseskaf is pretty much setup and ready to go. I'll be installing the infinband card and software, and other than that all it needs is Eckhard's software install. I've already setup the user accounts and ssh host based access (from menkaure). This is the beefiest machine we have for curation with 16 Intel Xeon processors and 12GB of RAM." MSH noted also that the old web server thoth is now being rebuilt as a systems machine, and that infiniband firmware updates are now going ahead using a linux rebuild on hatshepsut. Thereafter, driver updates on the Windows machines may fix an old OS-level file copy bug. Software: RSC noted: "The creation of the release_16 branch was delayed somewhat as NJC and I worked through some regression bugs affecting non-survey releases due to the re-write of the auto-curation process for the VSA. Some of the reliability that had been built into the system over the past couple of years may still have been lost, but, having fixed the more obvious bugs, the release was made on Thursday. Included in release_16 is the automatic generation of table statistics by CU19 for release databases and pause-and-retry support for the handling of database deadlocks by auto-committing scripts. Seven curation scripts have been altered to make their command-line options for date ranges become mandatory arguments (notably SetupProgramme, AutoCurate, CU5, CU7, CU13, CU19) and semester dates are now only defined in one place, ObsCalendar, to make maintenance easier. These changes will be released and documented when operations confirm that they are satisfactory." The team discussed possible optimisations for bulk loads to try to speed up ingests further; bulk insert hints on ordering and exclusive table-level locks will be tested. Survey Data Release: A new release of VIKING was put online earlier in the week; this includes paw-print level deep stacks, source merging and epoch photometry etc. Non-survey Data Release: Non-survey ops trumped by more important things over the last week or so. Astrogrid deployment & Data Analysis services: MSH noted that he's building a test crossmatch table between 2MASS and the SDSS at the request of some users out in the VO-sphere. A 10" match criterion was suggested as being appropriate. Miscellaneous: Nothing else this week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================