From nch@roe.ac.uk Tue Dec 6 13:12:32 2011 Date: Mon, 5 Dec 2011 12:29:45 +0000 From: Nigel Hambly To: Eckhard Sutorius , Nicholas Cross , Rob Blake , Mike Read , Ross Collins , Clive Davenhall , Mark Holliman Cc: Stelios Voutsinas , Lorenzo Rimoldini , Jim Emerson , Keith Noddle , Mike Irwin , Norman Gray , Tom Shanks , Andy Lawrence at ROE , Tom Kerr , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archives team meeting minutes, 2/12/2011 Minutes of WFAU VDFS Science Archive meeting: December 2nd 2011 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: ETWS, RPB, NJC, ACD, MAR, RSC, RGM, NCH, STV Apologies: JPE, KTN, AL, MSH, C37, 89X DoNM: 10am Friday December 16th 2011 in the VH (MAR in the chair) Actions discharged this time: ----------------------------- ACTION: RSC to test temporary ingest file writing to high bandwidth shares between curation clients and DB servers. Discharged; RSC noted that a linux-side share mounted by an SQL Server had been extensively tested following a few software mods. Ingest/outgest operations were found to be 2x faster (probably because of the separation of r/w disks more than anything else). Other processes have been found to be at least as fast as under the old configuration. Follow-up action noted below. ACTION: RPB (or whoever communicates with the PIs) to keep up the pressure on world public release DB preparation. Superseded (see below) ACTION: MAR to look into the SQL needed. ** Discharged by RSC, whatever it was that was needed... ACTION: NJC to look at VVV DR1 contents Discharged; see below Actions partly discharged but continuing: ----------------------------------------- ACTION: RPB to roll out latest curation client / DB server share configuration for routine ops use. ACTION: RPB to try monthly detection table partitions for VVV. CU4 ingest of VVV data into separate partitions implemented, but further tests required Nearly done; some issues over database-driven products being knitted into the monthly processed OB-level catalogues, but otherwise appears to be working well. Partitioned views need to be automated and fully tested. ACTION: NJC to check how we are handling this and update schema as required ** (i.e. issue of the deblend column being replaced by average confidence values in the standard catalogue output) Partly done; NJC noted there were some issues of propagating this into the source tables in the VSA schema, but the catalogue detection tables are fine. RSC volunteered to check out the implications for the WSA. Actions carried forward from 18/11/11 meeting --------------------------------------------- ** ACTION: Minute taker to make sure actions are fully explained with self-contained text descriptions ACTION: RPB to set up WSA mirror/sync for flat file DBs using MS SQL replication. Continues ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues ACTION: RGM, with input from NCH, to contact VISTA PIs with suggested timescale for public releases. Continues (see below) Specific points and new actions: -------------------------------- Project management: Nothing of note this time. WFCAM, VISTA and VST updates: Nothing of note. Comments and issues arising from CASU minutes: The team noted the minutes of the meeting of 26/10/11; there were no comments. Networking: Nothing of note this time. WSA/VSA/OSA Operations: (NB: OSA = OmegaCam Science Archive, for VST/ATLAS and consistency with WSA = WFCAM Science Archive, and VSA = VISTA Science Archive ... yes alright, the camera isn't called VISTA) ETWS reported: -- Wrote the SQL necessary to create the OSA database and worked with ACD and NJC on the table schemas. -- Started to update CU3 to be able to ingest the metadata into the OSA. -- Created the browser pages for the OSA. -- Finished CU4 on the re-processed GPS non-surveys. Apart from 10 broken files, of which I've sent a list to CASU but haven't heard anything back yet, all the catalogues are ingested and we are ready to proceed. -- Still working on the segmentation fault in CU4 on the new servers. It was noted that operational responsibilities for the newest addition to VDFS, the OSA for VST surveys (primarily ATLAS), is becoming clearer following lots of communications between the various parties involved. ESO deliveries will be handled by CASU. ATLAS survey consortium DB releases in the OSA will be required at the end of each 6-month observing period, while the first public release is anticipated for spring 2013 incorporating observing periods P87, P88 and P89. Hardware and Systems: Nothing of note this time. Software: NJC reported: "This week I have fixed the problems with pyfits version 3.0.x when creating stacks and tiles. I have also looked into the issues of the zeropoint discrepancies in VIDEO and found differences with the way we ran SExtractor and the VIDEO team did. I also found problems in the calculation of the seeing and AB magnitude limit. The AB mag limit was fixed by a small correction in the code to make it more robust, the seeing was fixed by different selection of stars. Changing the selection of stars made a small change to the aperture corrections. Changing selections of stars in the aperture correction code on CASU tiles can produce differences of the same order of magnitude as between us and the VIDEO team, but I have been unable to reproduce their ZP. Removing all aperture corrections gets us much closer. Ihave decided that this is a small enough concern to move on for now - we use the VIDEO teams ZP for the mosaics and it will just affect the ZPs of the OB frames and the aperture corrected magnitudes in the deeps, although most users will be interested in the galaxies. I have also produced the programme tables for the OSA: OSA_atlasSchema.sql, OSA_atlasNeighboursSchema.sql etc by creating OSA template schemas and running the surveyparser code." Survey Data Release: The bulk of the meeting was taken up with a long discussion concerning the release of world-public DBs now that ESO have announced the availability of phase-3 "DR1" badged datasets for 4 out of the 6 public surveys in the ESO-SAF. Following much discussion to help RGM formulate communiques to the PIs, it was agreed that the following would be advised for each individual case: o VVV: P85, containing enhanced DB-driven variability products and otherwise consistent with the recently accepted VVV-DR1 A&A paper o VIDEO: One CMM/LSS field "XMM3" created from an existing consortium release DB, and incorporating the recently fettled sky-subtracted products o VMC: 2 complete fields as delivered for phase-3 and created from the existing consortium release DB from August 2011 o VHS: P85 release with views vhsDesSource/vhsGpsSource/vhsAtlasSource generated from the existing (tile-based) consortium release DB and including the tile-pawprint linked photometry Also (despite no officially badged phase-3 release): o VIKING: CFHT-W1 from existing consortium release DB RGM will draft the messages to the PI immediately for vetting by the ops team prior to sending out. Non-survey Data Release: Nothing to report this time. Astrogrid deployment & Data Analysis services: MSH communicated earlier in the week: "The CDS folks are working on a footprint service and requested UKIDSS coverage files. Their footprint service is still in the works, and looks quite basic at this point, but you can see it here: http://alasky.u-strasbg.fr/footprints/ The footprint files are generated from density maps of the source coordinates from all the primary sources in the respective surveys. Also, we should have a think if we want to produce some more specific ones (i.e. by waveband, or waveband combinations, etc), and then post them on the WSA and/or VSA websites as appropriate (to compliment the already existing coverage plots)." Miscellaneous: VST-ATLAS meeting next week in Durham; ACD, MAR and ETWS attending. Internal WFAU "quarterly" meeting next Wednesday, incorporating updates from the various project teams (VDFS, Gaia, strategic management, ...) to inform all as to what's going on. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ============================================================= -- Scanned by iCritical.