From mar@roe.ac.uk Fri May 22 13:10:11 2015 Date: Tue, 19 May 2015 15:28:52 +0100 From: Mike Read To: VDFS Science Archive Team -- Clive Davenhall , dmr@roe.ac.uk, Eckhard Sutorius , Mark Holliman , Nicholas Cross , Nigel Hambly , Rob Blake , Ross Collins , Stelios Voutsinas Cc: CCs for VDFS -- Alastair Edge , Andrew Lawrence , Bob Mann , Dante Minniti , James Dunlop , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Maria-Rosa Cioni , matt.jarvis@astro.ox.ac.uk, Mike Irwin , Nigel Metcalfe , Norman Gray , Phil Lucas , Richard McMahon at IOA , Simon Dye , Stephen Warren , Tom Kerr Subject: WFAU VDFS Science Archives progress meeting minutes, 8 May 2015 WFAU VDFS Science Archives progress meeting minutes, 8 May 2015 Present: RSC, ETWS, MAR, RGM, RPB, NJC, MSH Apologies: STV, KTN, AL, ACD DoNM: 10am Friday 22 May 2015 in the Villas meeting room, ACD in the chair (room is booked). Actions discharged this time: ----------------------------- ACTION: RPB (having been in inverse order: NJC, RPB, MAR) to progress [was: 'have a look at'] the status of non-survey releases for registered projects. Discharged, RPB had carried out tests on non-survey DBs mounted on a NAS box. Performance was not noticeably worse, consequently some of the larger older release had been moved to the NAS box thereby freeing up space for new releases. The new releases were now progressing. Actions carried forward from 24/4/2015 meeting: ----------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues: this action still has low priority. ACTION: RSC to test the new version of mxODBC (was ETWS/NCH purchase a new license for mxODBC. Continues. RSC was currently testing the new version under an evaluation license. ACTION: NJC noted that VISTA mosaicing crashes with normal filtering, which he should fix. Continues. NJC had implemented a fix and checked it into SVN. However, when it was run 'in anger' it still crashed. NJC to investigate. ACTION: RPB to compile a Web page listing all the completed releases. Continues. RPB had made a start compiling this list. However, the exercise revealed that a number of databases had been misplaced and he was investigating their whereabouts. ACTION: NJC to produce a roadmap for future VSA releases in order to plan for hardware requirements. Continues, NJC had contacted the PIs. ACTION: NJC noted that he had received a report of a problem with the ESO VVV DR1 release which he should investigate. In one pointing the merged band catalogue contained fewer sources than expected. Continues. This action was low on NJC's priority list. NJC noted that revised VVV DR2 content had been uploaded to ESO. Actions resulting from the VVV DR3 brainstorming meeting held on 24/2/15: ------------------------------------------------------------------------ ACTION: MAR to analyse the load and query performance on the public servers (to assess the feasibility of using lower performance machines for hosting public archives than for curation). Discharged, MAR had looked at the VVV user query load on the public servers. Percentage load was not high and the vast majority of sensible queries completed quickly. Following discussion it was agreed that the new DB server would become the VVV load server and DR3 would be released on the current load server (rasmes14). ACTION: NJC to speak to the VVV PIs, notably Dante, about the schedule for and expected number of remaining VVV releases. Continues. NJC had spoken to the PIs but the results had been inconclusive; he should pursue the matter further. ACTION: RPB to experiment with ingest via partitioned tables, along the lines of the SSIS paper. Continues. ACTION: RGM (was RSC, RGM): (i) RSC to write a succinct summary of the ingest and outgest problem that is the principal bottleneck with the VVV processing (briefly the ingest and outgest speed is limited to approx. 100 Mbit/sec for no apparent reason) and (ii) RGM to compose an e-mail message around this summary and send it to Ani Thakar. Continues. RSC had composed a succinct summary and sent it to RGM. RGM had emailed Ani Thakar but no reply to date. RGM would send a reminder to Ani. Still awaiting a response. Specific points and new actions: -------------------------------- Project management: * Nothing for the minutes. WFCAM, VISTA and VST updates: * WFCAM was back on UKIRT. Networking: * Nothing to report. WSA/VSA/OSA/GES Operations: * ETWS reported that VSA CU3 and CU4 were running on data from Oct-Dec 2014. Reprocessed WSA LAS J2 data was being transferred. Reprocessed ATLAS data, mainly list driven products needed to be ingested into the OSA. VVV CU6 was still running. Hardware and Systems: * Ramses16 and Nuiserre had new RAID cards fitted and were back online. Spare 2Tb disks had arrived. LTO tapes had been ordered. Software: * NJC noted that he been working on ATLAS ESO release code. Initial validation had required updating a lot of OSA UCDs. The latest VVV DR2 ESO release issues had been fixed. * ETWS had been installing the 3rd party software on Teti. ACTION: NJC to remove dependencies on old libraries. Review of Deadlines: * Progress towards forthcoming deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * None. Non-survey Data Release: * VSERV1 and U13A06v20150506, U13AD1v20150506, U13AKASI1v20150507, U13AKASI3v20150507 Astrogrid deployment, VO & Data Analysis services: * MSH reported that cone-search, SIAP and TAP services were now available for all the latest VSA public releases. Miscellaneous: * MAR noted that some indicies and statistics were missing from vvvSource in VVV DR2. ACTION: RPB/MAR to generate the missing VVV DR2 indicies and statistics.