From acd@roe.ac.uk Thu Mar 19 14:55:08 2015 Date: Thu, 19 Mar 2015 14:46:16 +0000 From: Clive Davenhall To: Bob Mann , Andy Lawrence , "Read, Mike" , Eckhard Sutorius , Mark S. Holliman , Nicholas Cross , Nigel C. Hambly , Rob Blake , Ross Collins , Stelios Voutsinas , Dave Morris , Clive Davenhall Cc: "dante@astro.puc.cl" , James Dunlop , "j.p.emerson@qmul.ac.uk" , ktn keithnoddle.org , "Lorenzo.Rimoldini@unige.ch" , "mcioni@aip.de" , "matt.jarvis@astro.ox.ac.uk" , "mike@ast.cam.ac.uk" , "nigel.metcalfe@durham.ac.uk" , "norman@astro.gla.ac.uk" , "pwl@star.herts.ac.uk" , "rgm@ast.cam.ac.uk" , "Simon.Dye@nottingham.ac.uk" , "s.j.warren@ic.ac.uk" , "tkerr@jach.hawaii.edu" , "tom.shanks@durham.ac.uk" Subject: WFAU VDFS Science Archives progress meeting minutes, 13 March 2015 WFAU VDFS Science Archives progress meeting minutes, 13 March 2015 Present: NJC, RPB, ETWS, RSC, MSH, ACD Apologies: RGM, NCH, MAR,STV, KTN, AL DoNM: 10am Friday 27 March 2015 in the Villas meeting room, MAR in the chair (room is booked). Actions discharged this time: ----------------------------- None. Actions carried forward from 27/2/2015 meeting: ----------------------------------------------- ACTION: NJC (was RPB; before that MAR) to progress [was: 'have a look at'] the status of non-survey releases for registered projects. Continues. MAR had compiled a list of non-survey projects that could be released and sent it to RPB. RPB had started creating releases for some 8-12 non-survey releases. However, they had crashed about half way through this preparation because of what appeared to be a schema problem. NJC had investigated this problem but not yet completed a fix. ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues: this action still has low priority. ACTION: ETWS/NCH purchase a new license for mxODBC. Continues. ETWS had raised this with NCH and was waiting for a reply. ETWS should remind NCH. The licence required is for an upgrade, so the action is not urgent, but nonetheless it should be progressed. ACTION: NJC noted that VISTA mosaicing crashes with normal filtering, which he should fix. Continues. NJC had implemented a fix and checked it into SVN. However, it had not yet been run 'in anger'. ACTION: RPB to compile a Web page listing all the completed releases. Continues. ACTION: NJC to produce a roadmap for future VSA releases in order to plan for hardware requirements. Continues. ACTION: NJC noted that he had received a report of a problem with the ESO VVV DR1 release which he should investigate. In one pointing the merged band catalogue contained fewer sources than expected. Continues. This action was low on NJC's priority list. ACTION: NJC to ask Matt Jarvis whether there will be a VIDEO P93 Team Release. Continues. ACTION: ACD. In a couple of months NJC would be presenting an overview of the WFAU archives relevant to Galactic astronomy at a conference. ACD should consider what NJC might say about the GES archive at this meeting. Continues. Actions resulting from the VVV DR3 brainstorming meeting held on 24/2/15: ------------------------------------------------------------------------ ACTION: MAR to analyse the load and query performance on the public servers (to assess the feasibility of using lower performance machines for hosting public archives than for curation). Continues. ACTION: NJC to speak to the VVV PIs, notably Dante, about the schedule for and expected number of remaining VVV releases. Continues. ACTION: MAR to speak to Mike Irwin about whether any re-processing of the VVV data is anticipated. Continues. CASU currently have no firm plans for re-processing VVV data but may produce TAR catalogues in the future. ACTION: RPB to experiment with ingest via partitioned tables, along the lines of the SSIS paper. Continues. ACTION: RGM (was RSC, RGM): (i) RSC to write a succinct summary of the ingest and outgest problem that is the principal bottleneck with the VVV processing (briefly the ingest and outgest speed is limited to approx. 100 Mbit/sec for no apparent reason) and (ii) RGM to compose an e-mail message around this summary and send it to Ani Thakar. Continues. RSC had composed a succinct summary and sent it to RGM. RGM had not yet composed a message around it and sent it to Ani Thakar. Specific points and new actions: -------------------------------- Project management: * ACD reported on new grant awards and consequent contract announcements. Award letters had been received for the post-rolling-grant VDFSS work and the Gaia DFS, so staff working on these projects would have their contracts extended in line with these grants. The EUCLID grant had been announced but no award letter was yet available. Consequently staff working on EUCLID would be offered interim extensions to cover the period until the awards were available. ACD noted that the late progress with this issue and interim nature of some of these arrangements was less than ideal and apologised to the staff affected. WFCAM, VISTA and VST updates: * WFCAM was currently removed from UKIRT and was expected to return to operation on 22/4/15. VISTA and VST were believed to be operating normally. Networking: * Nothing really to report. WSA/VSA/OSA/GES Operations: * ETWS reported: - Ingest of the VVV P92-P93 catalogue data has finished. The objID update is currently in progress. - Processing and ingest of OSA P92-P94 has resumed. - Transfer of the latest WSA data up to Dec 2014 has finished. - Transfer of the latest VSA data up to Dec 2014 has started. - Schema browser pages and metadocs for latest VSA and OSA releases have been created. * RPB reported: - The VHS P93 release is progressing and is currently nearing the end of Tile/PawPrint table creation. - VVV DR3 had crashed while running the new SynopticSource code and NJC is currently investigating the problem. Hardware and Systems: * MSH reported that the recent procurement from Viglen was not proceeding smoothly. Some of the components needed to assemble the order had been inadvertently sent to the ROE rather than to Viglen's premises. They had now been forwarded to Viglen. The completed order would not now be shipped until mid-April. When MSH pointed out that the tender had specified a delivery before the end of March the sales rep. that he was speaking to had been surprised. MSH also noted that the kit ordered from Eclipse was expected to be delivered next week. * MSH had migrated all the data from the oldest NAS boxes which could now be shut down. * Catherine Heymans had approached MSH about WFAU (i) hosting her storage kit and (ii) installing an instance of Astro-WISE for her. MSH had referred her to RGM in both instances. Software: * NJC had written new code to match the VVV survey to one or more extinction tables. Initially the facility was limited to the VVV survey but it had been written to make it easily extensible to other surveys. Spherical coordinates are handled using a KD-tree. He had also made some adjustments to the software for generating releases of non-survey databases. * ETWS had enhanced the objID updater to update correctly even when data are distributed over multiple databases. Review of Deadlines: * Progress towards forthcoming deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * Three survey releases had been made: - ATLAS P91 (DR2) Team Release, - VVV DR2 ESO Release, - VMC P93 Team Release. Non-survey Data Release: * None. Several are in preparation, but a few problems have been encountered over the past few weeks. These issues are now nearly resolved and the releases should be completed shortly. Astrogrid deployment, VO & Data Analysis services: * MSH reported that Google had announced that the 'Google Code' repository hosting service was to close. They had provided a tool to migrate projects using it to GitHub. However, GitHub is unsuitable for hosting the AstroGrid repository and other SVN hosting services should be investigated. Miscellaneous: * Several people would be away next week (starting Mon. 16/3/15). KTN, MSH and RPB would be at a EUCLID meeting. ACD would be on leave until Wed. ACD, Original: 13/3/15, Revised: 19/3/15.