From nch@roe.ac.uk Mon Jun 28 11:48:41 2010 Date: Fri, 25 Jun 2010 16:40:44 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren , Tom Kerr Subject: WFAU VDFS Science Archive weekly project meet minutes 25/06/10 Minutes of WFAU VDFS Science Archive meeting: June 25th 2010 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: ETWS, RPB, NJC, MAR, MSH, NCH, RGM Apologies: JPE, KTN, AL, RSC DoNM: Friday July 2nd in the Vista Hut Actions discharged this week: ----------------------------- ACTION: ALL to review new hardware requirements/procurement over the next couple of weeks Discharged; specs with KTN for tendering ACTION: NCH to fill out a Q2 2010 plan and continue with the progress monitoring for the time being. Deprecated as no longer necessary ACTION: NJC to continue to try to find out what photometric corrections will be applied to tiles. Discharged; JRL and STH have been contacted at CASU Actions partly discharged but continuing: ----------------------------------------- ACTION: RSC & NJC to document new operational procedures on the TWiki Continues (nearly done - NJC needs to add some new UDS/SExtractor things) ACTION: ETWS to try some test ingests of VIRCam tiles. Continues; new keywords need to be added to the schema and then tests done; then some small changes will likely be necessary when the finalised version 1.0 products are delivered. All noted also that provenance issues and QC scripts to auto-deprecate any incomplete tile products will need close attention once tiles are successfully ingested. ACTION: RPB/KTN to order several crates of LTO-4 tapes. Continues ACTION: ETWS to enhance the early CUs to cope better with CASU's process versioning Continues; nearly done except auto-deprecation of previous processed versions; this nasty hard-wired piece of SQL (by NCH) probably needs refactored into a flexible Puthon method. ACTION on NJC to test SExtractor on shepseskaf and finish off CU13 upgrades Continues; ETWS noted this should be tested on menkaure also. ACTION NCH to decide on frame set tolerance for VVV source merging as their is a disparity between the documentation and the database. Discharged; NCH contacted VVV PIs in the week and the clear response was a requirement for a (pairing, not frame-set) tolerance of 1.0 arcsec. RPB kindly agreed to make the required changes in the VSA & setup script. Actions carried forward from 17/06/10 meeting --------------------------------------------- ACTION: TBD to draft and MOU on behalf of PSPIs for ESO-SAF deliveries Continues; RGM volunteered to email F. Comeron at ESO-SAF to get the ball rolling. ACTION NJC to suggest workshop at next PSPI meeting. Specific points and new actions: -------------------------------- Project management: Nothing news-worthy this week. WFCAM & VISTA updates: No updates this week. Comments and issues arising from CASU minutes: No new minutes this week. Networking: KTN has attended the SuperJANET upgrade/user meeting in Newport Pagnell this week. WSA/VSA Operations: RPB reported: "VSA data processing: all P85 data to the end of April is now fully ingested apart from two days: one with the tiles, and one with a broken fits file. CASU have been informed. Non-Surveys: Some non-surveys have been processed this week with a roughly 50% success rate. NJC is looking into the problems. Backups: Windows backups decided to stop working, apparently due to services running as the administrator user. This is now fixed, though explaining why all the previous backups worked just fine is more difficult. VMC: New VMC release database put live yesterday. Hopefully this should fix the problem of missing deep stacks caused by the b0rken end date saga." ETWS noted -- Apart from some final tests regarding the deprecation of former versions, non-integer version numbers are now integrated in all the code. -- Installed the latest version of 3rd party software on menkaure after MSH upgraded the operating system. The team noted that a planned site power outage has been delayed until possibly next Tuesday, and discussed protecting our servers and informing users of the possible interuption to services. Hardware and Systems: Debian "squeeze" (?!) updates to the curation servers are taking place, MSH noted. Software: NJC noted "The extinction code has been producing crazy values this last week, including some Nan values. I tried tracking it down yesterday and then turned my brain on this morning and realised that one week ago Ross made a change to the src/galabs/Makefile: CFLAGS = -O3 -D__LINUX was changed to CFLAGS = -O3 -D__LINUX -std=c89 Reverting this, it now works on khafre etc. So the good news is that little if anything has been screwed up. The bad news is, we will not be able to run it on menkuare or shepseskaf." The team discussed at length the latest new requirements to come out of the extensive email interactions with some of the survey PIs (particularly the VMC) and the necessary software modifications: o QC flagging of pawprints not to be included in deepstacks but requiring to be kept for epoch photometry - this requires definition of a new QC deprecation code and hardwiring in relevant curation codes o saturation correction: MJI's nifty technique as detailed in UKIRT newsletter #25 has been requested for the VMC but seems like a good idea generally for both WFCAM and VIRCam data, so the intention is to implement asap ACTION: NCH to ask MJI for a code snippet just so that we can be sure we implement at the archive end exactly the recommended CASU algorithm for saturated flux recovery o freeform SQL access prior to release of static catalogue products (e.g. for QC purposes): MAR and RPB have been working on this, defining views and configuring access credentials to protect proprietary rights when accessing the monolithic ingest DB; it was noted that the online documentation will need enhancing to avoid confusing the survey teams, and/or final end-users of static releases ACTION: MAR to add a new web page detailing the main points of difference when access the ingest VSA as opposed to static survey release DBs ACTION: ETWS to enhance the schema browser parser to include a distinct "VSA QC" entry that excludes objects not accessible to any, but includes views (e.g. vmcMultiframe etc.) and details of the partitioned detection table schema as employed in the ingest DB. Survey Data Release: No news this week Non-survey Data Release: NJC noted that there were two non-surveys released this last week. One had no extinction terms in it (too low galactic latitude), but the other u09a4 did and it has strange values of extinction A_K etc (see above). This should be rerun at some stage and give the PI a heads up. Astrogrid deployment & Data Analysis services: Nothing to report this week. Miscellaneous: MSH announced that he's had enough and is off down to the beer festival in the Chambers St. Assembly Rooms to get p***ed. MSH now holds the record for the earliest of early-doors. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================