From nch@roe.ac.uk Mon Jul 19 12:23:09 2010 Date: Fri, 16 Jul 2010 14:02:45 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren , Tom Kerr Subject: WFAU VDFS Science Archive weekly project meeting mins, 16/07/2010 Minutes of WFAU VDFS Science Archive meeting: July 16th 2010 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: ETWS, RPB, NJC, MAR, MSH, NCH, RSC Apologies: JPE, KTN, AL, RGM DoNM: Friday July 16th in the Vista Hut Actions discharged this week: ----------------------------- ACTION: ETWS (with help from NJC where necessary) to implement aper mag saturation correction in CU4/exnumeric ETWS and NCH got a little confused over the details in MJI's f77 snippet, so NJC is being consulted. An email to MJI uncovered a missing line of code, and NJC says he's now confident he understands the procedure. ACTION: NCH to send a potted version of the apermag conflab to all PSPIs to solicit opinions and try to reach agreement. Discharged; thanks to VHS PSPI RGMcM for taking the initiative in distilling NCH's blether into three clear options and for proposing a fair voting scheme for the survey PIs. So far, only the VHS vote is in, and the compromise/pragmatic solution is in the lead (one half of VVV has indicated this would be their preference also). ACTION: ETWS to add in to the VSA template schema new individual deltaMag attributes for illumination, distortion and saturation corrections. Discharged; will be committed to trunk when the apermag changes have been agreed and applied. ACTION: NJC to contact WJS concerning the latter's bright star proximity studies in VIRCam images (cc RSC) Discharged Actions partly discharged but continuing: ----------------------------------------- ACTION: RSC & NJC to document new operational procedures on the TWiki Continues (nearly done - NJC needs to add some new UDS/SExtractor things) ACTION on NJC to test SExtractor on shepseskaf and finish off CU13 upgrades Continues; ETWS noted this should be tested on menkaure also. ACTION: ETWS to check that distortion correction is switched OFF for VIRCam tiles in TAN projection in CU4 Discharged but needs testing when the v1.0 tiles arrive. ACTION: ETWS to CU3/4/ingest reprocessed UDS J data 05A-09A in preparation for inclusion of UDS in DR8. CU3s all done; CU4s are next once some reprocessed GPS data (for DR7) are in. Actions carried forward from 09/07/10 meeting --------------------------------------------- ACTION: TBD to draft and MOU on behalf of PSPIs for ESO-SAF deliveries Continues; reply received from F.Comeron and discussions on-going amongst PSPIs and ESO concerning the exact requirements. It was noted from the PSPI teleconf (see below) that the likely deadline for submission of P85 products is March 2011. ACTION NJC to suggest workshop at next PSPI meeting. ACTION: ETWS to enhance the schema browser parser to include a distinct "VSA QC" entry that excludes objects not accessible to any, but includes views (e.g. vmcMultiframe etc.) and details of the partitioned detection table schema as employed in the ingest DB. Continues; needs a hack of the existing DB-driven approach for the special case of the VSA ingest DB. ACTION: NJC to check that distortion correction is switched OFF for VIRCam tile in TAN projection in CU13 ACTION: MAR to update the data access documentation to reflect larger download limits. Specific points and new actions: -------------------------------- Project management: Nothing of note this week. WFCAM & VISTA updates: No updates this week. Comments and issues arising from CASU minutes: No new minutes this week. Networking: NCH noted that UKIDSS LAS PI SJW has enquired about the possibility of linking in future DRs with a putative SDSS-DR8. Apparently there are plans to create such a catalogue release that incorporates all sorts of acronyms like BOSS and SEGUE etc. NCH noted that if a static SQL DB is available, WFAU will do the usual trick of copying into over into the system and setting up the crossmatching. VIRCam v1.0 transfers still await the green light from CASU. WSA/VSA Operations: ETWS noted -- Transfered the reprocessed DR7 GPS data. -- Ingested metadata for the reprocessed UDS and GPS data and created JPGs. -- Updated CU4 to write the illumination, distortion and saturation corrections into new attributes in the detection tables. -- Made progress on the VSA QC schema browser. Hardware and Systems: MSH noted that new memory is on order for menkaure which is currently in a forgetful state, having only 3GB. The motherboard has 8 DIMM slots, currently the DIMMs are 1GB. Potential expandability (with 2GB units) to 16GB, but this is felt unnecessary at this point. Software: RSC reported: "I've implemented the proposed changes to merged source positions for VISTA surveys to use weighted mean positions rather than masters selected by the detection in the shortest wavelength. This is applied to all surveys that have a source table schema that does not include separate columns for each observation epoch, i.e. everything other than the UKIDSS shallow surveys (which will eventually be changed to source positions derived from full astrometric solutions of proper motions). Full details are in trac ticket:151. Also, I've been catching up on the recent major software developments to CU6, CU7 as well as the new deprecation code implementation. I reran my earlier performance profile tests on CU6 now that it is aided by pre-calculated half-spaces in an auxiliary table, and confirmed that there is between a 2x and 6x overall performance advantage with a minimal initial overhead, whilst identifying future potential improvements. I've tested the change to frame set association by fieldID in CU7 for the cases of automatically setup programmes, and have ensured it's implemented consistently throughout the software design. Lastly, I've identified some minor bugs in the deprecation code selection codes, which have now been fixed, and have suggested this selection should be via constants rather than functions for clarity. Finally, I've sorted out all the inconsistencies between the SVN branches ready for a new release branch to be made; release_17. This will include the new wrapper interface for the new version of PyFITS." NCH reported: "This week I have finished off the changes to speed up best matching for synoptic tables. The slowest part, determining whether an object should have been observed has now been sped up by up to 30 times and is no longer the slowest step. I have compiled a list of changes to be made by v1.0 for the VSA. This may not be complete, but are: Modify entire code base to work with Tiles (Partly discharged - but can't be fully until we receive tiling code and more V1 data). This includes checking code works here on deep stacks, checking quality control/ QBF, ingest, CU7, CU6. Also do we need to do better links between tile detections and pawprint detections or will Provenance etc be good enough. Change schema / CU4 for new aperture mag names and columns for corrections Add in saturated magnitude corrections VVV SynopticSource (allowing both SourceXSynopticSourceBestMatch and SourceXDetectionBestMarch) Updating Filter table with A/E(B-V) and VegaToAB (Need values from Simon Hodgkin) Speed ups to CU6 (Almost complete, but a little polishing)" ACTION: NCH to assign actions on the above next time. RSC suggested that there are some further tile-related issues that can be tested with the prototype products, as ETWS has ingested them into the VSA DB. ACTION: RSC to test tiles QBits including finalising the saturation bit consistent with the correction procedure noted previously, and also CU7 if any tiles overlap, etc. RSC suggested also that a new release branch of the curation SW should be made to tag some of the recent software developments ACTION: RSC to make a new release branch of the curation software. Survey Data Release: The VMC PI reported that the vmcSources from deep stacks seemed to be missing a few obvious detections; the problem was traced to the thresholds applied at the source extraction stage. MJI/JRL were consulted, and the former noted that "All VISTA catalogues from v1.0 onward should be using 1.25 sigma for threshold" so this is being followed at the archive end too to keep it consistent with the CASU nightly pipeline. The change will take effect in the next catalogue DB releases at v1.0 UKIDSS-DR8 preparation is underway following delivery of the LAS eyeball deprecations. Some useful discussions this week also with the UDS mosaicing dudes from Nottingham; the stack will be arringing shortly enabling a synchronised (for a change!) UKIDSS DR including LAS, GCS, DXS and UDS. Non-survey Data Release: RPB noted that further non-surveys and another "uber" non-survey (ie. several programmes bolted together to create a single survey-like release database, like) have been released to PIs this week, covering 08B (late registered) and 09A/B. Astrogrid deployment & Data Analysis services: Nothing to report this week. Miscellaneous: Early doors at the Old Bell tonight to welcome WFAU Gaia visitor Javier Castaneda. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================