From mar@roe.ac.uk Fri Apr 24 15:06:08 2015 Date: Mon, 20 Apr 2015 15:33:48 +0100 From: Mike Read To: VDFS Science Archive Team -- Clive Davenhall , dmr@roe.ac.uk, Eckhard Sutorius , Mark Holliman , Nicholas Cross , Nigel Hambly , Rob Blake , Ross Collins , Stelios Voutsinas Cc: CCs for VDFS -- Alastair Edge , Andrew Lawrence , Bob Mann , Dante Minniti , James Dunlop , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Maria-Rosa Cioni , matt.jarvis@astro.ox.ac.uk, Mike Irwin , Nigel Metcalfe , Norman Gray , Phil Lucas , Richard McMahon at IOA , Simon Dye , Stephen Warren , Tom Kerr , Tom Shanks Subject: WFAU VDFS minutes 10 April 2015 WFAU VDFS Science Archives progress meeting minutes, 10 April 2015 Present: NJC, RPB, MAR, RSC, MSH, ACD Apologies: RGM, NCH, ETWS,STV, KTN, AL DoNM: 10am Friday 24 April 2015 in the Villas meeting room, ACD in the chair (room is booked). Actions discharged this time: ----------------------------- ACTION: NJC to ask Matt Jarvis whether there will be a VIDEO P93 Team Release. NJC noted that we are expecting there to be a VIDEO P93 release. ACTION: ACD. In a couple of months NJC would be presenting an overview of the WFAU archives relevant to Galactic astronomy at a conference. ACD should consider what NJC might say about the GES archive at this meeting. Discharged. ACTION: MAR to speak to Mike Irwin about whether any re-processing of the VVV data is anticipated. CASU currently have no firm plans for wholesale re-processing VVV data but may produce revised tile catalogues in the future. Actions carried forward from 13/3/2015 meeting: ----------------------------------------------- ACTION: NJC (was RPB; before that MAR) to progress [was: 'have a look at'] the status of non-survey releases for registered projects. Continues. MAR had compiled a list of non-survey projects that could be released and sent it to RPB. RPB had started creating releases for some 8-12 non-survey releases. However, they had crashed about half way through this preparation because of what appeared to be a schema problem. RPB would attempt to re-run. ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues: this action still has low priority. ACTION: ETWS/NCH purchase a new license for mxODBC. Continues. ETWS had raised this with NCH and was waiting for a reply. ETWS should remind NCH. The licence required is for an upgrade, so the action is not urgent, but nonetheless it should be progressed. ACTION: NJC noted that VISTA mosaicing crashes with normal filtering, which he should fix. Continues. NJC had implemented a fix and checked it into SVN. However, it had not yet been run 'in anger'. ACTION: RPB to compile a Web page listing all the completed releases. Continues. ACTION: NJC to produce a roadmap for future VSA releases in order to plan for hardware requirements. Continues. ACTION: NJC noted that he had received a report of a problem with the ESO VVV DR1 release which he should investigate. In one pointing the merged band catalogue contained fewer sources than expected. Continues. This action was low on NJC's priority list. Actions resulting from the VVV DR3 brainstorming meeting held on 24/2/15: ------------------------------------------------------------------------ ACTION: MAR to analyse the load and query performance on the public servers (to assess the feasibility of using lower performance machines for hosting public archives than for curation). Continues. ACTION: NJC to speak to the VVV PIs, notably Dante, about the schedule for and expected number of remaining VVV releases. Continues. ACTION: RPB to experiment with ingest via partitioned tables, along the lines of the SSIS paper. Continues. ACTION: RGM (was RSC, RGM): (i) RSC to write a succinct summary of the ingest and outgest problem that is the principal bottleneck with the VVV processing (briefly the ingest and outgest speed is limited to approx. 100 Mbit/sec for no apparent reason) and (ii) RGM to compose an e-mail message around this summary and send it to Ani Thakar. Continues. RSC had composed a succinct summary and sent it to RGM. RGM had emailed Ani Thakar but no reply to date. Specific points and new actions: -------------------------------- Project management: * Nothing for the minutes. WFCAM, VISTA and VST updates: * Nothing for the minutes. Networking: * Nothing really to report. WSA/VSA/OSA/GES Operations: * VVV CU6 was about a third of the way through. * ACD reported that in response to a request from Cambridge an additional, interim GES release was prepared comprising just the metadata for the iDR4 Giraffe manyspec files. This release is now available to the Consortium as gesiDR4GiraffeSpectra. In conformance with Consortium Policy the spectra themselves were not included in the release. Hardware and Systems: * MSH reported that we now had a new webserver (niuserre), a new NAS box (/disk52-56 approx 150Tb) and a new database node (ramses16). Disk15 had been copied from khafre to the new NAS box. RPB noted that some disks had needed replacing on pedubaste, which was currently re-building. Software: * NJC reported working on programme builder with a view to creating LAS deep stacks. He'd also starting looking at VVV proper motions. Work on the ATLAS ESO release should be prioritized over the next few weeks. Review of Deadlines: * Progress towards forthcoming deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * Five survey releases had been made: - VHS P99 team release, - VIDEO DR4 public release, - VIKING DR4 public release. - VHS DR3 public release. - GES gesiDR4GiraffeSpectra preliminary team release. Non-survey Data Release: * None. Several are still in preparation. Astrogrid deployment, VO & Data Analysis services: * MSH noted he would be deploying VO services for the new public releases. Miscellaneous: Nothing for the minutes