From mar@roe.ac.uk Thu Jul 23 14:52:50 2015 Date: Thu, 23 Jul 2015 14:35:01 +0100 From: Mike Read To: VDFS Science Archive Team -- Clive Davenhall , dmr@roe.ac.uk, Eckhard Sutorius , Mark Holliman , Nicholas Cross , Nigel Hambly , Rob Blake , Ross Collins , Stelios Voutsinas Cc: CCs for VDFS -- Alastair Edge , Andrew Lawrence , Bob Mann , Dante Minniti , James Dunlop , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Maria-Rosa Cioni , matt.jarvis@astro.ox.ac.uk, Mike Irwin , Nigel Metcalfe , Norman Gray , Phil Lucas , Richard McMahon at IOA , Simon Dye , Stephen Warren , Tom Shanks Subject: WFAU VDFS Science Archives progress meeting minutes, 17 July 2015 WFAU VDFS Science Archives progress meeting minutes, 17 July 2015 Present: RGM, ETWS, MSH, NJC, RPB, RSC, ACD, MAR Apologies: STV, KTN, AL, DoNM: 10am Friday 31 July 2015 in the Villas meeting room, ACD in the chair (room is booked). Actions discharged this time: ----------------------------- None Actions carried forward from 3/7/2015 meeting: ----------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues: this action still has low priority. ACTION: ACD to obtain the details of obtaining an mxODBC licence from ETWS and progress obtaining a sufficient number of such licences. Continues. There are currently no issues in using the version of mxODBC we have. ACTION: NJC noted that VISTA mosaicing crashes with normal filtering, which he should fix. Continues. NJC had implemented a fix and checked it into SVN. However, when it was run 'in anger' it still crashed. NJC to investigate. ACTION: RPB to compile a Web page listing all the completed releases. Continues. RPB had made a start compiling this list. However, the exercise revealed that a number of databases had been misplaced and he was investigating their whereabouts. ACTION: NJC noted that he had received a report of a problem with the ESO VVV DR1 release which he should investigate. In one pointing the merged band catalogue contained fewer sources than expected. Continues. This action was low on NJC's priority list. ACTION RPB: for VVV investigate whether the way in which extractions are allocated to machines could be rejigged to improve performance. Continues. ACTION NJC: check that CU19 is outgesting a specific list from TilePawprints' tables. Continues. Actions resulting from the VVV DR3 brainstorming meeting held on 24/2/15: ------------------------------------------------------------------------ ACTION: RPB to experiment with ingest via partitioned tables, along the lines of the SSIS paper. Deferred. RPB was investigating various options and commencement of work on the present action would depend on the results of these investigations. ACTION: RGM (was RSC, RGM): (i) RSC to write a succinct summary of the ingest and outgest problem that is the principal bottleneck with the VVV processing (briefly the ingest and outgest speed is limited to approx. 100 Mbit/sec for no apparent reason) and (ii) RGM to compose an e-mail message around this summary and send it to Ani Thakar. Continues. RSC had composed a succinct summary and sent it to RGM. RGM had emailed Ani Thakar but no reply to date. RGM had sent a reminder but had still received no response. ACTION: NJC to remove dependencies on old libraries. Continues. This was mainly done but NJC to check on what if anything was still needed for VOTable handling. Specific points and new actions: -------------------------------- Project management: * RGM noted that the grant renewal would be submitted in late September and that those involved in writing the various sections would be fleshing out the outline in the coming weeks. WFCAM, VISTA and VST updates: * Nothing to report; all the telescopes were believed to be operating nominally. Networking: * Nothing to report; the networks were performing nominally. WSA/VSA/OSA/GES Operations: * ETWS reported that he had: - Transferred new/fixed OSA data and these were currently being ingested. - Produced a new WSA schema browser for the new domain. The outgest of the detection table for VVV DR3 was approximately half way through. Hardware and Systems: The webserver, seshat, had suffered from several spontaneous shutdowns without logging any issues. It had managed to stay up for about a week but as it was off maintenance services were being moved to niuserre. RPB noted that he had some issues getting all the data off the archival UKIRT tapes but might have only been accessing the first tar file on a given tape. MSH noted that he would attempt to install an exabyte drive on one of our old servers to see if the SuperCOSMOS tapes were readable. One of RGM's students had brought thoth to a standstill querying the SDSS database and running MPI code. An account would be set-up on userkaf for her so to avoid blocking the gateway server. It was discussed whether other IfA servers could be used instead, possibly employing an SSH tunnel to connect to our DB servers. This method had been suggested and set-up during Andy Connolly's recent visit, but it was not known how well/if it was working. ACTION: MAR to contact AJC to see if he was able to query our databases remotely via SSH tunnel. Software: * NJC reported: - further work on putting all archives and programmes on the same automated set-up - looking into the setup and schemas for further variability and difference imaging analysis. * ACD noted that he done more work on modifying the GES ingest software to handle single spec files * MAR had been getting the seshat web services running on niuserre. * ETWS had made some minor software mods for the OSA and had installed additional software on userkaf for Maria Manolopoulou. Review of Deadlines: * ACD had yet to update the deadlines twiki based on the discussion at the previous VDFS meeting. He would be contacting the relevant individuals and updating the page. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * None since the last meeting. Non-survey Data Release: * None since the last meeting. Astrogrid deployment, VO & Data Analysis services: * Nothing to report; all systems operating nominally. Miscellaneous: RSC noted that he would shortly be submitting an abstract for the GES workshop (Vilnius, December 1-4, 2015). NJC and ETWS would be attending the ESO workshop, Rainbows in the Southern Sky, in (Garching, October 5-9, 2015). MSH reported that there would be a Euclid system team meeting in Edinburgh, December 14-15 2015.