From acd@roe.ac.uk Tue May 5 15:21:15 2015 Date: Tue, 5 May 2015 14:10:24 +0000 From: Clive Davenhall To: Bob Mann , Andy Lawrence , "Read, Mike" , Eckhard Sutorius , Mark S. Holliman , Nicholas Cross , Nigel C. Hambly , Rob Blake , Ross Collins , Stelios Voutsinas , Dave Morris , Clive Davenhall Cc: "dante@astro.puc.cl" , James Dunlop , "j.p.emerson@qmul.ac.uk" , ktn keithnoddle.org , "Lorenzo.Rimoldini@unige.ch" , "mcioni@aip.de" , "matt.jarvis@astro.ox.ac.uk" , "mike@ast.cam.ac.uk" , "nigel.metcalfe@durham.ac.uk" , "norman@astro.gla.ac.uk" , "pwl@star.herts.ac.uk" , "rgm@ast.cam.ac.uk" , "Simon.Dye@nottingham.ac.uk" , "s.j.warren@ic.ac.uk" , "tkerr@jach.hawaii.edu" , "tom.shanks@durham.ac.uk" Subject: WFAU VDFS Science Archives progress meeting minutes, 24 April 2015 WFAU VDFS Science Archives progress meeting minutes, 24 April 2015 Present: RSC, ETWS, MAR, RGM, RPB, NJC, MSH, ACD Apologies: STV, KTN, AL DoNM: 10am Friday 8 May 2015 in the Villas meeting room, MAR in the chair (room is booked). Actions discharged this time: ----------------------------- None. Actions carried forward from 13/3/2015 meeting: ----------------------------------------------- ACTION: RPB (having been in inverse order: NJC, RPB, MAR) to progress [was: 'have a look at'] the status of non-survey releases for registered projects. Continues. MAR had compiled a list of non-survey projects that could be released and sent it to RPB. RPB had started creating releases for some 8-12 non-survey releases. However, they had crashed about half way through this preparation because of what appeared to be a schema problem. NJC had fixed the problem and the software appeared to run correctly. However, there was insufficient disk space on the public server for the new releases. RPB should investigate the possibilities for either freeing-up disk space or using alternative space. ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues: this action still has low priority. ACTION: ETWS/NCH purchase a new license for mxODBC. Continues. ETWS had checked with NCH. The licence was available on an offer with without any time restrictions. The purchase can and should proceed. ACTION: NJC noted that VISTA mosaicing crashes with normal filtering, which he should fix. Continues. NJC had implemented a fix and checked it into SVN. However, when it was run 'in anger' it still crashed. NJC to investigate. ACTION: RPB to compile a Web page listing all the completed releases. Continues. RPB had made a start compiling this list. However, the exercise revealed that a number of databases had been misplaced and he was investigating their whereabouts. ACTION: NJC to produce a roadmap for future VSA releases in order to plan for hardware requirements. Continues. ACTION: NJC noted that he had received a report of a problem with the ESO VVV DR1 release which he should investigate. In one pointing the merged band catalogue contained fewer sources than expected. Continues. This action was low on NJC's priority list. NJC noted that a number of bugs in DR2 had been reported which where more pressing. Actions resulting from the VVV DR3 brainstorming meeting held on 24/2/15: ------------------------------------------------------------------------ ACTION: MAR to analyse the load and query performance on the public servers (to assess the feasibility of using lower performance machines for hosting public archives than for curation). Continues. ACTION: NJC to speak to the VVV PIs, notably Dante, about the schedule for and expected number of remaining VVV releases. Continues. NJC had spoken to the PIs but the results had been inconclusive; he should pursue the matter further. ACTION: RPB to experiment with ingest via partitioned tables, along the lines of the SSIS paper. Continues. ACTION: RGM (was RSC, RGM): (i) RSC to write a succinct summary of the ingest and outgest problem that is the principal bottleneck with the VVV processing (briefly the ingest and outgest speed is limited to approx. 100 Mbit/sec for no apparent reason) and (ii) RGM to compose an e-mail message around this summary and send it to Ani Thakar. Continues. RSC had composed a succinct summary and sent it to RGM. RGM had emailed Ani Thakar but no reply to date. RGM would send a reminder to Ani. Specific points and new actions: -------------------------------- Project management: * RGM would shortly be speaking to Colin Vincent about a 3-year grant, to start on 1/4/16 to continue the VDFSS archiving. It was expected to be about half the size of the current grant. The recent meeting with Lockheed Martin seemed to have gone well and possibilities for future collaboration looked promising. WFCAM, VISTA and VST updates: * Nothing to report. Networking: * Nothing to report. WSA/VSA/OSA/GES Operations: * ETWS reported that he had: - prepared ingest lists for the re-transferred OSA data and was deprecating the old data. - Created browser pages and metadocs for the latest VSA releases. Hardware and Systems: * MSH reported: - the new processing machines were on-line, - the heat fluctuations in the server room had been fixed, - a new server rack had been installed, - some spare disks were urgently needed as the current spares had largely been used to replace disks that had failed. Software: * NJC noted that he been working on an OSA ESO release. He had also worked on a program-builder for deep stacks and experimented with software for a correlated multi-epoch survey. * ETWS had developed scripts to copy/deprecate old data in which only catalogue files where re-transferred and to create file lists for easy CU3/4 ingest of the new data. Review of Deadlines: * Progress towards forthcoming deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * The VIKING P93 Team release had been made. Non-survey Data Release: * None. Several are still in preparation. Astrogrid deployment, VO & Data Analysis services: * MSH noted that he had updated the UCDs to version 1+. ACTION: RGM to again remind Norman Gray about completing and releasing UCD+. Miscellaneous: * MSH noted that additional backup tapes were urgently required. ACD, Original: 30/4/15, Revised: 5/5/15.