From nch@roe.ac.uk Fri Jul 30 18:02:08 2010 Date: Fri, 30 Jul 2010 17:37:20 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren , Tom Kerr Subject: WFAU VDFS Science Archive weekly project meet mins, 30/07/10 Minutes of WFAU VDFS Science Archive meeting: July 30th 2010 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NJC, MAR, NCH, RSC, RPB, ETWS Apologies: JPE, KTN, AL, MSH, RGM DoNM: Friday August 6th in the Vista Hut Actions discharged this week: ----------------------------- ACTION on NJC to test SExtractor on shepseskaf and finish off CU13 upgrades Deprecated ACTION: RGM to contact JPE concerning WFAU's role in ESO survey deliveries. Discharged ACTION: NCH to chase up the PSPIs for responses concerning possible schema changes with respect to apermags. Discharged; one further response from VIDEO PI, who voted the same as VHS/VVV. Looks like the compromise solution wins, so when catalogue ingests start (next week) the appropriate changes will be made to the template schema. ACTION: ETWS to check that distortion correction is switched OFF for VIRCam tiles in TAN projection in CU4 Discharged ACTION: ETWS (with help from NJC where necessary) to implement aper mag saturation correction in CU4/exnumeric Discharged (see issues arising from CASU minutes below). ACTION: ETWS to enhance the schema browser parser to include a distinct "VSA QC" entry that excludes objects not accessible to any, but includes views (e.g. vmcMultiframe etc.) and details of the partitioned detection table schema as employed in the ingest DB. Discharged ACTION: RSC to test tiles QBits including finalising the saturation bit consistent with the correction procedure noted previously, and also CU7 if any tiles overlap, etc. Discharged ACTION: RSC to make a new release branch of the curation software. Discharged ACTION: ETWS to CU3/4/ingest reprocessed UDS J data 05A-09A in preparation for inclusion of UDS in DR8. Discharged Actions partly discharged but continuing: ----------------------------------------- ACTION: RSC & NJC to document new operational procedures on the TWiki Continues (nearly done - NJC needs to add some new UDS/SExtractor things) ACTION: NCH to assign actions on the following next time: - Modify entire code base to work with Tiles (Partly discharged - but needs fully testing on the new V1 data). This includes checking code works here on deep stacks, checking quality control/ QBF, ingest, CU7, CU6. Also do we need to do better links between tile detections and pawprint detections or will Provenance etc be good enough. - Change schema / CU4 for new aperture mag names and columns for corrections - VVV SynopticSource (allowing both SourceXSynopticSourceBestMatch and SourceXDetectionBestMarch) - Updating Filter table with A/E(B-V) and VegaToAB (Need values from STH) Continues Actions carried forward from 23/07/10 meeting --------------------------------------------- ACTION: TBD to draft and MOU on behalf of PSPIs for ESO-SAF deliveries Continues; reply received from F.Comeron and discussions on-going amongst PSPIs and ESO concerning the exact requirements. More news (see below) from ESO concerning "Phase 3" rules this week. ACTION: NJC to suggest workshop at next PSPI meeting. ACTION: NJC to check that distortion correction is switched OFF for VIRCam tile in TAN projection in CU13 Specific points and new actions: -------------------------------- Project management: Nothing of note this week. WFCAM & VISTA updates: No updates this week. Comments and issues arising from CASU minutes: The team noted the minutes of the meeting of 28th July. Concerning MJI's worries over blanket application of the saturation correction, it was noted that VVV and VMC have asked for this to be so; and also that the correction value will be materialised in the *Detection tables so can easily be removed at query time if desired. The team noted that the correction will presently only be applied for the VSA; a retrospective fix in the WSA for the final release will be done subject to the wishes of UKIDSS. The team discussed whether to apply only to apermag3, or all apermags ... the consensus was that it should be applied to all, but advice from MJI would be greatly appreciated. The team noted and understand that the CASU nebulosity filtering and tiling SW need further enhancement/checking, and hope that the PSPIs are similarly patient when it comes to archive-end implementation for DB-driven products. Finally, regarding SunOS XFS/ZFS, RPB noted that we tried out ZFS a couple of years ago and had a similarly negative experience - even the dude from Sun Microsystems (but Megaprices) couldn't get the damn thing to work. So regarding "does our ZFS need a good tune ?" we suggest Delilah by Tom Jones - that's a damn fine tune. Networking: VISTA v1.0 product transfers have been roaring up north at 25 Mbyte/s. WSA/VSA Operations: RPB reported: "Vista V1.0 data copying up from CASU. So far copied up to the end of January. Current v1.0 release to 14th Feb should be finished by Monday. CU3 and CU2 also running. Should be finished by early next week. WSA DR8 proceeding. LAS ready for release. GCS currently source merging. DXS awaiting the running of part 2 of AutoCurate." ETWS noted -- Included tiles as separate entries in the monitor pages for VISTA. -- Made good progress in including the saturation correction for VISTA apermags. CU3 metadata ingests for the first batch of v1.0 data are half-done; RSC noted that it might be a good idea to run a quick auto-check on the schema-to-FITSkeys match to check we're not missing anything new and vital ACTION: ETWS to run his auto fitskeys check on the current schema to make sure no vital metadata is missing from the VSA Hardware and Systems: Nowt to report this week. Software: RSC noted: "Revised detection quality flags in preparation for VISTA version 1 data. The saturation flag is now implemented for the VSA (using same bit number as for the WSA) by using the same criterion as the new saturation correction. This flagged about 1% of VHS detections, which is a similar level to the WSA flag. A new bit (23, the most severe one implemented yet at one more than the flag for the underexposed dither region of stacks) is now assigned to flagging detections in the underexposed strips of VISTA tiles (or the ears as they are otherwise known). Whilst testing the implementation of this new flag, was somewhat surprised to discover that the x/y-axes of the tile images were swapped when compared to the description at www.vista.ac.uk. This implementation will need to be tested again to account for any positional offsets in the construction of deep-stacked tiles once we receive the necessary software tools to do this from CASU." MAR noted that he has made an enhancement to the UI functionality in that users will be able to upload SQL tables to crossmatch with static tables; (this done at the request of several users). Survey Data Release: DR8 preparations nearly complete as noted above; UDS is on hold pending delivery of stacks/cats next week from Nottingham. Non-survey Data Release: RSC noted that the 09A data proprietary period has expired for non-surveys, and that we need to do another non-survey world release including the 09A data into WFCAMOPENTIME. ACTION: RPB to update WFCAMOPENTIME for 09A projects no longer inside their proprietary period. Astrogrid deployment & Data Analysis services: Nothing to report this week. Miscellaneous: That's all for this week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================