From nch@roe.ac.uk Sat Feb 26 11:21:40 2011 Date: Fri, 25 Feb 2011 17:29:53 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Jim Emerson , "Noddle, Keith -- Keith Noddle" , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Bob Mann , Stephen Warren , Tom Kerr Subject: WFAU VDFS Science Archive project meeting mins, 25/02/11 Minutes of WFAU VDFS Science Archive meeting: February 25th 2011 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, MAR, NJC, RPB, MSH, RGM, RSC Apologies: JPE, AL, KTN DoNM: Friday March 11th 2011 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: RPB to check that VHS, VIKING and VIDEO have required neighbours specified for UKIDSS-LAS and VVV for UKIDSS-GPS, VIDEOxUKIDSSUDS Discharged ACTION: RGM to seek clarification from Koreans on plans for their enhanced slice of WFCAM time. Discharged; no response as yet. ACTION: NJC to draft a concise summary email to the PSPIs asking for clarification over WFAU phase III deliveries for VHS and VIKING, noting the requirements for VMC and VIDEO, and the default "do nowt" mode for UV and VIDEO. Discahrged; response (not always as clear as we might have hoped) have been received from all PIs. Actions partly discharged but continuing: ----------------------------------------- ACTION: NCH to ask RPB to look into the perennial functions permissions problem. Discussed during this meeting; DRs 6 & 7 done while DRs4 and before pending; RPB will check on the necessary T-SQL scripts/users and then communicate with RSC to insert into CU19 (ACTION RSC) Actions carried forward from 11/02/11 meeting --------------------------------------------- ACTION: ETWS/RPB to update WSA/VSA schemas for new keyword NICOMB Continues; decision made to wait for the next reprocessing version (1.1). ACTION: RPB to look into automated RAID monitoring across all servers. Continues; ramses9 (most recent SW installation) has such a facility; it was noted that on the linux side monitoring is working as normal. Specific points and new actions: -------------------------------- Project management: Nothing of note this week. WFCAM & VISTA updates: No updates this week. Comments and issues arising from CASU minutes: No new minutes this week. Networking: VISTA v1.1 will be available imminently, we understand. The team suspect there will be a chorus of requests to base WFAU-ESO deliveries on this process version, despite all the previous warnings that this is simply not possible... sigh. WSA/VSA Operations: ETWS noted: - Updated the browser parser to include all new external catalogues and created a new set of pages after the VMC release. - Started the yearly update of third party software after the upgrade of the operating systems by MSH. - Fixed problems with the slow creation of monitor webpages due to the large amount of VVV data. The team reviewed the status of the various VISTA surveys, particularly regarding the ESO deliverables: o VMC: curation DB now ready with ESO deliverable fields; PI is undertaking a review of these prior to the flat file products being created. In the meantime, FITS header issues are being worked on. A full-scale VSA release DB is now being prepared for the survey team also. o VHS: a tile-curated release DB is being prepared in order that the PI can do their own QC on their own deliverables. It was noted that this is a one-off duplication of the same data and processing period contained in an existing paw-print based DB, and that from v1.1 onwards the tile-based DB will be produced with additional linking info incorporated to enable any paw-print functionality required. It was also noted that the user interface was never really designed for bulk outgest of the source tables, so this may be an issue for the PI's plan for the source catalogue deliveries. o VVV: bandmerging and variability products are to be delivered for the current deadline; production of these is to start next week following extensive iterations with the co-PI on QC issues and the contents of the delivered products. o VIKING: there are some tiling issues still to sort out for this survey; some confusion in communications with the PI as to exactly what is required for the current deliveries to ESO. o VIDEO: somewhat late in the day the requirement to deliver the OB images/cats for "intermediate" stacks has become apparent; rather than doing this from WFAU when all other such deliveries are being done by CASU, we suggested that VIDEO requested CASU to oblige. Latest communications seem to indicate that CASU are being most accomodating over this rather tardy request. As for requirements for products from VIDEO's own deep stacks, WFAU offered to make and deliver band-merged catalogues but were told this was not required at this stage; a further request to clarify if any other band-merged products are required to be delivered is yet to be answered. o UltraVISTA: SEP. Hardware and Systems: Latest debian upgrades have been successfully applied across all curation servers. MSH noted that there is now a plan to migrate all data holdings on server local mass filestores to NAS, starting with those on djedefre since one of its RAID arrays is currently degraded. NCH asked about the status of backups, and RPB noted that there is a routine weekly incremental backup of the ingest DBs, but that it is probably time to do a full one. It was also noted that DB F287 needs a backup, and that ideally we still need backups enabled from the linux curation servers to the Windows-hosted LTO4 tape library unit. ACTION: RPB to investigate LTO4 tape library backups from the linux client side. Software: RSC noted: "The specification for the ESO release products have been continually evolving as a greater understanding (or misunderstanding as the case may be) of the requirements has been achieved. There's been a lot of work in determining what the metadata requirements of the FITS headers should be and figuring out how best to obtain/calculate this metadata for each product as part of the overall curation process. Most of this is now implemented and I've been testing the catalogue (both detection and source) outgests for the two VMC fields to track down memory management issues in the Python script. There has been a lot of firefighting over the first deep tiled VISTA releases, with us getting in a mess over WFAU-product versioning again, which required some software fixes as well as remedial work to the database. For peace of mind such manual fixes to the database, when done via the pySQL.py script, are now fully logged both to help us track down the source of any later problems that may arise and to have a complete record of changes between backups. We also returned to investigating the re-occuring problem of CU19 detection table outgests slowing down / hanging / crashing in various ways. Suspicion is that the execution query plan has changed either due to the new version of SQL Server or else due to the additional amount of data/metadata. The problem seems to have been solved by an index on one of ProgrammeFrame primary key columns combined with a slightly different SQL query (one that we originally used, but changed for performance reasons for the GPS DR3 release), but we need to keep an eye on this in future. Finally, the SyncDb script has been modified to always perform dirty reads of the load database, for both software testing and metadata mirrors, in the hope that the BCP processes won't lock tables and interfere with operations again." This fortnight NJC has tested the new versions of CASU tools that Jim Lewis has kindly provided and sorted out the tilig code to work with these. NJC has been writing the ESO releases software to modify tile headers and outgest light curves. NJC has also been working with PS PIs on the ESO release requirements and has been fixing issues with the tiling pipeline when the data quality is low. NJC has also been looking at memory issues with the variability code and thinking about v1.1 issues. Survey Data Release: MSH asked when UKIDSS GPS/UDS go world readable at DR5 since an enquiry has been received for SIAP services. The proprietary release was Aug 28th 2009, which means the world release should be on Feb 28th (i.e. next Monday)? Non-survey Data Release: RSC noted that WFCAM open time non-survey projects from 09B should be made world public now. ACTION: RPB to make WFCAM 09B non-survey data world readable. Astrogrid deployment & Data Analysis services: No news this week. Miscellaneous: Nowt else this week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ============================================================= -- Scanned by iCritical.