From mar@roe.ac.uk Fri Nov 13 15:04:36 2015 Date: Fri, 13 Nov 2015 15:04:30 +0000 From: Mike Read To: VDFS Science Archive Team -- Clive Davenhall , dmr@roe.ac.uk, Eckhard Sutorius , Mark Holliman , Nicholas Cross , Nigel Hambly , Rob Blake , Ross Collins , Stelios Voutsinas Cc: CCs for VDFS -- Alastair Edge , Andrew Lawrence , Bob Mann , Dante Minniti , James Dunlop , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Maria-Rosa Cioni , matt.jarvis@astro.ox.ac.uk, Mike Irwin , Nigel Metcalfe , Norman Gray , Phil Lucas , Richard McMahon at IOA , Simon Dye , Stephen Warren , Tom Kerr , Tom Shanks Subject: WFAU VDFS Science Archives progress meeting minutes, 13 November 2015 WFAU VDFS Science Archives progress meeting minutes, 13 November 2015 Present: MSH, ETWS, MAR, NJC, RSC Apologies: KTN, AL, RGM, RPB, ACD, STV, DMR DoNM: 10am Friday 4 December 2015 in the Villas meeting room, ACD in the chair (room is booked). Actions discharged this time: ---------------------------- None. Actions carried forward from 30/10/2015 meeting: ----------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues: this action still has low priority. ACTION: NJC noted that VISTA mosaicing crashes with normal filtering, which he should fix. Continues. NJC had implemented a fix and checked it into SVN. However, when it was run 'in anger' it still crashed. NJC to investigate. ACTION: NJC noted that he had received a report of a problem with the ESO VVV DR1 release which he should investigate. In one pointing the merged band catalogue contained fewer sources than expected. Continues. This action was low on NJC's priority list. ACTION MSH to pass to DMR his list of UCD 1 / UCD 1+ equivalences and a complete list of columns in all the WFAU databases. Actions resulting from the VVV DR3 brainstorming meeting held on 24/2/15: ------------------------------------------------------------------------ ACTION: RPB to experiment with ingest via partitioned tables, along the lines of the SSIS paper. Deferred (changed back from 'continues'). This action has now been overtaken by events: the VVV DR3 processing would now be done differently. However, the technique was still potentially useful for future work and should be investigated, but not as a matter of urgency. ACTION: RGM (was RSC, RGM): (i) RSC to write a succinct summary of the ingest and outgest problem that is the principal bottleneck with the VVV processing (briefly the ingest and outgest speed is limited to approx. 100 Mbit/sec for no apparent reason) and (ii) RGM to compose an e-mail message around this summary and send it to Ani Thakar. Continues. RSC had composed a succinct summary and sent it to RGM. RGM had emailed Ani Thakar but no reply to date. RGM had sent a reminder but had still received no response. RGM was planning to contact Alex Szalay on another matter and would take the opportunity to raise the issue with him. Specific points and new actions: -------------------------------- Project management: * Nothing to report. WFCAM, VISTA and VST updates: * Nothing to report Networking: * The network was now working again after some downtime following the server room works. WSA/VSA/OSA/GES Operations: * ETWS reported that the latest VSA and OSA data had been transferred and had been CU'ed 2 and 3 but further progress and ATLAS list driven ingests were being blocked by the VMC and VPHAS 0releases. Some previously missing 14B/15A WSA data had been curated. VVV database files were currrently being transferred to the VVV load server. ETWS suggested that we should think about running our releases differently so as not to block load server operations i.e. by copying data to a different server first and then running the release procedures there. * The homogenised GES iDR4 full parameters and abundances files had been transferred up from CASU. Hardware and Systems: * MSH reported that the server room works had been moved back a week to 27-29th November. This wasn't ideal as he was unavailable on that Saturday. The plan was to start shutting down machines on the Friday, RPB could probably come in on the Saturday and MSH the Sunday. Software: * NJC noted that he'd been working on quality bit flagging, ESO release code, vvvSource content and looking into using SWarp following the meeting with Alastair Edge and Simon Driver. * MAR had updated some of the VSA VVV services so as to access the synopticSource table. Review of Deadlines: * Using printer problems as an excuse, this item was skipped. http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * Following problems/delays during the ppErrBit flagging phase the VMC release was now progressing and currently processing deep stacks. Non-survey Data Release: * None. Astrogrid deployment, VO & Data Analysis services: * Nothing to report. Miscellaneous: * WFAU quarterly and Xmas lunch Thursday 26th November.