From nch@roe.ac.uk Wed Jul 28 13:18:46 2010 Date: Wed, 28 Jul 2010 12:37:32 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren , Tom Kerr Subject: WFAU VDFS Science Archive weekly project meet mins 23/07/10 Minutes of WFAU VDFS Science Archive meeting: July 23rd 2010 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NJC, MAR, MSH, NCH, RSC, RGM Apologies: JPE, KTN, AL, ETWS, RPB DoNM: Friday July 30th in the Vista Hut Actions discharged this week: ----------------------------- ACTION: MAR to update the data access documentation to reflect larger download limits. Discharged Actions partly discharged but continuing: ----------------------------------------- ACTION: RSC & NJC to document new operational procedures on the TWiki Continues (nearly done - NJC needs to add some new UDS/SExtractor things) ACTION on NJC to test SExtractor on shepseskaf and finish off CU13 upgrades Continues; ETWS noted this should be tested on menkaure also. ACTION: ETWS to check that distortion correction is switched OFF for VIRCam tiles in TAN projection in CU4 Discharged but needs testing now that v1.0 tiles have arrived. ACTION: ETWS to CU3/4/ingest reprocessed UDS J data 05A-09A in preparation for inclusion of UDS in DR8. CU3s all done; CU4s are next once some reprocessed GPS data (for DR7) are in. ACTION: ETWS (with help from NJC where necessary) to implement aper mag saturation correction in CU4/exnumeric ACTION: ETWS to enhance the schema browser parser to include a distinct "VSA QC" entry that excludes objects not accessible to any, but includes views (e.g. vmcMultiframe etc.) and details of the partitioned detection table schema as employed in the ingest DB. Continues; MAR noted that this is nearly done. ACTION: NCH to assign actions on the following next time: - Modify entire code base to work with Tiles (Partly discharged - but needs fully testing on the new V1 data). This includes checking code works here on deep stacks, checking quality control/ QBF, ingest, CU7, CU6. Also do we need to do better links between tile detections and pawprint detections or will Provenance etc be good enough. - Change schema / CU4 for new aperture mag names and columns for corrections - VVV SynopticSource (allowing both SourceXSynopticSourceBestMatch and SourceXDetectionBestMarch) - Updating Filter table with A/E(B-V) and VegaToAB (Need values from STH) Continues Actions carried forward from 16/07/10 meeting --------------------------------------------- ACTION: TBD to draft and MOU on behalf of PSPIs for ESO-SAF deliveries Continues; reply received from F.Comeron and discussions on-going amongst PSPIs and ESO concerning the exact requirements. More news (see below) from ESO concerning "Phase 3" rules this week. ACTION: NJC to suggest workshop at next PSPI meeting. ACTION: NJC to check that distortion correction is switched OFF for VIRCam tile in TAN projection in CU13 ACTION: RSC to test tiles QBits including finalising the saturation bit consistent with the correction procedure noted previously, and also CU7 if any tiles overlap, etc. Continues; RSC noted he needs some input concerning the threshold from which to apply bit flagging. ACTION: RSC to make a new release branch of the curation software. Stalled because new CU6 codes is out of sync with the schema. Specific points and new actions: -------------------------------- Project management: More official info concerning ESO Phase 3 (delivery of VISTA survey products to ESO-SAF) this week, via JPE. Official "year zero" is April 1st 2010, but dry run (Nov'09 to Mar'10) data are included in the delivery requirements. Data are to be routinely delivered 6 months after a tile dataset has been acquired, rather than in bulk for a given ESO observing period, once reduction procedures have reach steady state. The latter is assumed to be Sept 30 2010 since ESO expect first data product deliveries are expected by March 31st 2011 at the latest. NCH noted that this is all useful info, but the big question facing us now is: who is doing what. It seems that all of the PSPIs, CASU and WFAU could be delivering stuff to the archive, and there is no clear organisation as to which products should be delivered from where. NCH suggested that the time to sort this out is when representatives of all (including ESO) are in the same room, and maybe the autumn phase-3 meeting in Garching is the place to thrash it out. RGM thought it should be done in the PSPIs telecons, but NJC and MAR both noted that this had indeed been initiated a few times by JPE but without a clear outcome. RGM volunteered to contact JPE about this organisational issue. ACTION: RGM to contact JPE concerning WFAU's role in ESO survey deliveries. WFCAM & VISTA updates: No updates this week. Comments and issues arising from CASU minutes: No new minutes this week. NJC has also been in touch with CASU concerning archive-end implementation of standalone pipeline tools, noting that "we have imcore, imcore_list, wcsfit and imstack. I think that CASU may have made some changes to imcore/imcore_list to fix a bug in the half-light radius. We also need the tools that CASU use to make tiles (including the image filtering tool). I don't know if there are any changes to imcore or imstack for version 1 either. If there are any other tools that CASU think would be useful then please tell us." Networking: NCH noted that he can't invoke VSA v1.0 transfers (data was released to WFAU earlier in the week) for some reason - possibly to do with non-integer version numbers and automatic tools refusing to overwrite the v0.8 stuff. ETWS has been contacted; maybe RPB can help on his return on Monday. Of course, enquiries have already started to come in as to when the v1.0 data will be available through the archive; the answer is asap for flat-files, and then catalogue ingests will need to be tested for the latest schema changes for tiles, apermag attributes etc. MSH noted that there will be interuptions on any transfers from Cambridge next Wednesday owing to scheduled Janet maintenance on lightpath equipment at Cambridge University. WSA/VSA Operations: MAR closed out DR8 QC1 for the LAS on Thurs, so NCH is holding the fort with quality-bit flagging in LAS/GCS/GPS and stacking of the DXS this weekend. Hopefully may get the LAS source merging running as well. The release branch is being used for these, on servers menkaure and shepseskaf. Hardware and Systems: Just a fan replacement on a NAS box this week; otherwise all quiet on the hardware front. Software: NJC noted: "I have started improving ProgrammeBuilder to group pawprints into tiles. I have almost finished updating CU13 for the changes to run with SExtractor and I have added a new option to UpdateDetectionTables to create unique programmeIDs when multiple programme detection tables are added together." Concerning apermag schema adjustments for the VISTA PS, so far only VHS and VVV have replied, expressing a preference for the compromise plan. NCH noted that it's interesting (and somewhat worrying) that the main extragalactic survey PIs (esp. VIKING and VIDEO) have not replied yet, so ACTION: NCH to chase up the PSPIs for responses concerning possible schema changes with respect to apermags. A few adjustments to the UI logins have been requested this week, and MAR is looking into making the required changes. MAR noted that this week he also enhanced, broke and then fixed some of the multi-get and colour image facilities in the UI. Survey Data Release: DR8 preparations continue apace; NCH noted that Nottingham have indicated delivery of the UDS DR8 stacks will be the week after next. MAR raised a question concerning the likely contents of the first VISTA PS DRs, and it was noted that ESO's position seems to be that "dry run" data should be included as well as P85 survey data "proper". Our assumption is that, unless otherwise indicated, dry-run data will be included in the first public releases. MAR noted finally that he intended to close out GPS DR7 QC (i.e. the QC runs following ingest of the reprocessed filtered data) by the end of the week. Non-survey Data Release: Nothing to report this week. Astrogrid deployment & Data Analysis services: Nothing to report this week. Miscellaneous: MAR noted that UKIDSS survey head SJW had asked for deep J stacks to be incorporated into the LAS at DR10... the optimism is most admirable. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================