From nch@roe.ac.uk Fri Dec 4 16:26:38 2009 Date: Fri, 4 Dec 2009 16:07:12 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes 4/12/09 Minutes of WFAU VDFS Science Archive meeting: December 4th 2009 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: PMW, NCH, RSC, ETWS, NJC, RPB Apologies: JPE, JMS, RGM, AL, MSH, MAR DoNM: 10am, Friday December 11th 2009 in the VISTA Hut Actions discharged this week: ----------------------------- ACTION: RSC to review quality bit flagging for the VSA Discharged; RSC has put notes up on the internal Wiki pages at http://apache.roe.ac.uk/twiki/bin/view/WFAU/QualityBitFlags#Results_AN1 for internal consumption. The team discussed additional flags for detections in underexposed strips of standard 6-pawprint macrodithered tiles, and peak flux close-to-saturation. ACTION: RSC to profile C++ codes (eg. exnumeric) to diagnose seg faults Discontinued as the problem seems to have gone away, but NCH suggested that if a simple check can be done on the C/C++ codes for basic memory access violations, it might be a good idea to do so. RSC volunteered to look into this. NCH thanked RSC for his meticulous work on the quality error bits once again. Actions partly discharged but continuing: ----------------------------------------- ACTION: RPB: backup all DB-driven products after UDS DR5. STILL awaiting delivery of SCSI card; RPB suggested we cannibalise ahmose's now that the tape drive has been shifted to the cluster. Scrapheap Challenge here we come. Actions carried forward from 27/11/09 meeting: ---------------------------------------------- None this week Specific points and new actions: -------------------------------- Project management: The team filled in the progress charts for November. PMW noted that the final GBFR report is conspicuous by it's absence... WFCAM & VISTA updates: Updates received from JPE concerning availability of SV/PR/dry-run VISTA data to PSPIs this week, and a schedule for ESO's "preliminary acceptance" of VISTA. Not willing to quote anything specific here lest the Stasi get wind of it though. Comments and issues arising from CASU minutes: The team noted the minutes of the meeting of 19th November. NJC noted that WFAU look forward to receiving the "final" VISTA versions of the pipeline tools (stack, cataloguing etc.) for integration into the archive end for database-driven ops like deep staking etc., and that he has also asked JPE for information concerning all the VISTA filter bandpasses. Networking: ETWS noted that MJI communicated the availability of "version_0.6" VISTA SV/PR data (i.e. data from 27th September onwards) and this will be transfered next after the WFCAM October data (also release to WFAU this week as communicated by MJI - thanks!). WSA/VSA Operations: RPB noted: "Investigating database stats. Pretty sure that auto create stats won't work on read-only databases, so have been investigating the possibility of listing existing stats on older data releases to recreate by hand on DR6. Suggest we also have a log of any "nostats" warning given by the user interface so that we can periodically check if we need to create any more. CAL release finished first part of autocurate, then CU2, 3, & 4 were run and ingested. Second part of autocurate fell over at the CU8 stage. Not sure what the problem was, but this needs to be restarted once we're sure that killing the process off hasn't corrupted any files. ETWS highlighted a problem with the read-only data releases. Occasionally it's necessary to update file locations in older DRs when things get moved around. I think I can automate a great deal of the process of unmounting databases and remounting them as read/write to make this easier. Needs more work." ETWS noted: - Transfer of 09B October data is going ahead. - WCS reprocessed GPS data has been transferred and reingested. - Asked IT support to update the Starlink software to the latest possible version, so that the AST WCS table key problem would be fixed in a more generic way. Hardware and Systems: RSC asked if there was a problem with some NAS volumes since pami seemed to be giving intermittent access problems on some disks. Software: RSC noted: "I've been testing the archive agnostic implementation of the post-CU4 scripts now that we have some detections from stacks ingested into the VSA. Both regression tests for WSA data and VSA tests, in test databases. Also: - regressions tests are all passed apart from a mysterious bug in mkmerge apparently caused by the VISTA developments to the C++ code that means it's not producing results (and not producing any errors either). - three detection quality flags are working fine for the VSA: detections in a dither region, with deblended images and with bad pixels in the default aperture. The close to saturation flag will need further investigation to see if the somewhat arbitrary limit chosen for WFCAM is applicable to VIRCAM. Tests of flagging detections in the underexposed strip of tiles await ingests of tile metadata and detections. - CU7 merge logs have been successfully created for the VSA to help MAR test out the user interface. - CU7 source merging tests on the VSA have revealed a schema discrepancy between the release detection tables and the split-for-curation detection tables." ETWS noted that JRL advised that wcslib is being used at CASU for compatibility with FITS binary table "T" wcs keywords, and that ETWS has been looking into support (or lack thereof) of the same in Starlink AST. The latest release compatible with our Debian linux version (which is somewhat out of date) is Lehuakona, which doesn't solve the problem. In any case, it's not a show-stopper since WFAU's codes are being cludged to provide a work-around. Survey Data Release: UKIDSS survey head SJW has been in touch concerning the putative DR7 schedule for LAS, GCS and DXS. MAR noted that he can probably get QC1 done for these within the next week or so, so that fits in with the proposed eyeballing deadline of end of the year. The team discussed possible enhancements still outstanding regarding multi-epoch visits in one passband (e.g. Jx2 in the LAS, Kx2 in the GPS and GCS) i.e. deeper stacks and proper motions where the data exist. Given likely VISTA pressures, the team has agreed that DR7 by the end of Jan 2010 is doable, and will endeavour to include J deeps but no guarantees on the latter. Proper motion implementation will be deferred until a later release in all probability. Non-survey Data Release: No news this week. Astrogrid deployment & Data Analysis services: No news this week. Miscellaneous: Nothing else this week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================