From nch@roe.ac.uk Fri Jul 13 20:28:51 2012 Date: Fri, 13 Jul 2012 13:48:21 +0100 From: Nigel Hambly To: Eckhard Sutorius , Nicholas Cross , Rob Blake , Ross Collins , Mike Read , Clive Davenhall , Mark Holliman Cc: Lorenzo Rimoldini , Bob Mann , Jim Emerson , Keith Noddle , Norman Gray , Tom Shanks , Stephen Warren , Simon Dye , Stelios Voutsinas , Tom Kerr , Mike Irwin , Andy Lawrence at ROE Subject: WFAU VDFS Science Archives team meeting, 13 July 2012 Present: NCH, RSC, NJC, ETWS, RPB, MSH Apologies: JPE, AL, KTN, STV, RGM, NG, MAR DoNM: 10am Friday July 27th 2012 in the Vista Hut (subject to non-demolition); MAR in the chair. Actions discharged this time: ----------------------------- ACTION: RPB to set up WSA mirror/sync for flat file DBs using MS SQL replication. Discharged ACTION: RPB to complete the LTO-5 tape backups of VVV and SSA DBs and SSS flat file datasets. And VVV DR1. And WISE-ASS (snig). Discharged; SSA+SSS tapes to be labelled and stashed in the firesafe Actions partly discharged but continuing: ----------------------------------------- ACTION: RSC to scope out the areas/situations where we'd like to test MonetDB i.e. where do the current bottlenecks lie? Continues; some discussions concerning test cases resulted in the suggestion that current VVV curation stats should be assembled as a baseline, and that example end-user trawl queries from the web logs and from the WSA and VSA papers should be used also. Actions carried forward from 22/06/12 meeting --------------------------------------------- ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues; this one still has low priority. Specific points and new actions: -------------------------------- Project management: The grant renewal case (Q2 2013 - Q1 2016) has been completed and RGM was doing a final check-over this morning. NCH thanked everybody for their contributions. Plan is to submit later today, and then not hold our breath waiting for any outcome given the imminent STFC programmatic review. WFCAM, VISTA and VST updates: Nothing to report this time. Comments and issues arising from CASU minutes: The team noted the minutes of the meeting of 27/05/2012. It was noted that i) WFCAM data employing the new FeII narrow-band filter is about to arrive; ii) CASU are gamely undertaking to do some automated DAOPHOT-like PSF fitted image analysis experiments to appease the GPS and VVV teams; iii) VST v1.0 data are now available for transfer and ingest. ETWS noted that only a few schema changes have been necessary to accomodate the latest data, and that WFAU will employ "fixed" catalogues for detection table ingests. Regarding i) ACTION: ETWS (with help from NJC as necessary) to add FeII 1.644 micron filter details into the WSA. Networking: ETWS noted: -- After UKLight has been fixed, transfers are up to their old speed of 20-25MB/s again. WFCAM transfers are halfway through May, VISTA data is transferred up till the end of April, and the VST data re-transfer is also under way. ETWS and MSH noted that it was not clear exactly where the problem lay with the network, but IT Support Group up the hill said it was a problem "down the hill" - whether this means at JCMB in King's Buildings, or further afield on the network, is not known. Anyway, good to see things are back to normal; especially gratifying is that parallel transfers of infrared (WFCAM or VISTA) and optical (VST) data can be launched and both streams achieve the full transfer rate. WSA/VSA/OSA/GES Operations: ETWS noted: -- As a speed-up for catalogue ingests I wrote a script FinaliseCatalogueIngest which includes creating the constraints on the monthly VVV tables and deprecation of old detections. The latter was a part of CU4 and slowed it down when run for each ingest. RSC noted that he has been checking the latest GES data against the current schema, but that we need to do wholesale transfers/ingests to make sure everything goes in OK. ACTION: NCH to nudge JRL and ask if VDFS-like bulk slurps of GES data can be set up. Hardware and Systems: MSH noted: "SSD testing is progressing well. Query tests show full trawls take 1/4 of the execution time when compared to current hardware, and trawls of indexed columns are about 30% faster. Bulk ingests show only limited improvement (10-15%), though full scale testing of larger ingests have just begun. Details will be posted to the wiki shortly." MSH also noted that Gaia server kakai has a new motherboard in an attempt to cure it's unstable behaviour. Survey Data Release: UKIDSS DR10: SJW's eyeball QC has thrown up an issue whereby hole-plugging with single paw-prints is causing sky subtraction issues in that the pipeline uses the paw-sky method, and large extended objects (>10") are adversely affected. The agreed approach is for a new deprecation code (68) to be defined to deprecate any paw-sky subtracted images containing affected galaxies so as to exclude from DR10 while the problem is further investigated and a solution found, hopefully in time for the final UKIDSS DR. SJW has also noted another low-level issue in microstepped frames (which has been present all along but only just noticed) where systematic offsets in photometry caused by trailing in a constituent frame can lead to contamination in colour-selected samples. The agreed solution is to do a phase-II QC cut proper in DR10, employing a first-pass source merged catalogue to detect the problem using two-colour plots. Hence DR10 preparations will pause after source merging (CU7) and any new frame deprecations incorporated by a quick second-pass of CU7 before continuing with later CUs and the release. Regarding the UDS in DR10, ACTION: NJC to communicate with Nottingham concerning incorporation of UDS in DR10. The continuing Nordic saga that is the ESO-SAF phase-III deliverables: VIKING: done and uploaded (albeit the previous tranche, not the current catalogue tranche...) VMC: nearly there (some issues, but will be uploaded next week) VIDEO: ditto VVV: ditto (except uploading may take more time of course). VHS: some interaction with survey PI, but uncertainty over exactly what needs doing, so ACTION: NJC to clarify with RGMcM if he needs anything from WFAU for the current ESO deliverables NJC noted that he has made some procedural and software mods to keep track of what's being done and where regarding ESO releases, to aid in future deliveries. It was noted also that the VVV DR1 has not been made world-readable yet as the co-PIs have not yet sanctioned the release. ACTION: NJC to contact VVV team to see if they're happy with world release of DR1. Software: RSC noted: "In preparation for the forthcoming VVV DR2 curation runs I've been rearranging CU16 to be more disk space efficient and investigating the possibility of optimisations for matches with extragalactic source catalogues (beyond the default option of excluding them)." MSH noted: "Testing of Redmine integration with Mercurial is ongoing with various levels of success. The hope is that an integrated Redmine project management site with a mercurial repository will be used for WFAU's Euclid development in the future." GPS PI has been in touch regarding some low-level systematics in astrometry with regard to proper motion calculation; NCH will attempt to placate. Non-survey Data Release: None. Astrogrid deployment & Data Analysis services: MSH noted: "VO Service metadata is getting updated to enable VAO Portal access. Previous additions and changes didn't solve the problem of VAO portal clients being unable to query our services, but I've coordinated with them to locate and fix the problem. Deployment VO services for all public released VISTA databases will proceed next week (currently only VHS is available)." Miscellaneous: NJC noted that he's in the process of revising the VSA paper; NCH has put a darft up on the VSA. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ============================================================= -- Scanned by iCritical.