From acd@roe.ac.uk Thu Mar 12 13:49:00 2015 Date: Mon, 9 Mar 2015 17:52:44 +0000 From: Clive Davenhall To: Bob Mann , Andy Lawrence , "Read, Mike" , Eckhard Sutorius , Mark S. Holliman , Nicholas Cross , Nigel C. Hambly , Rob Blake , Ross Collins , Stelios Voutsinas , Dave Morris , Clive Davenhall Cc: "dante@astro.puc.cl" , James Dunlop , "j.p.emerson@qmul.ac.uk" , ktn keithnoddle.org , "Lorenzo.Rimoldini@unige.ch" , "mcioni@aip.de" , "matt.jarvis@astro.ox.ac.uk" , "mike@ast.cam.ac.uk" , "nigel.metcalfe@durham.ac.uk" , "norman@astro.gla.ac.uk" , "pwl@star.herts.ac.uk" , "rgm@ast.cam.ac.uk" , "Simon.Dye@nottingham.ac.uk" , "s.j.warren@ic.ac.uk" , "tkerr@jach.hawaii.edu" , "tom.shanks@durham.ac.uk" Subject: WFAU VDFS Science Archives progress meeting minutes, 27 February 2015 WFAU VDFS Science Archives progress meeting minutes, 27 February 2015 Present: RSC, RGM, MAR, RPB, ETWS, NJC, ACD Apologies: MSH, NCH, STV, KTN, AL DoNM: 10am Friday 13 March 2015 in the Villas meeting room, ACD in the chair (room is booked). Actions discharged this time: ----------------------------- ACTION: ACD to schedule a meeting to review progress with the various VVV brainstorming tasks from the previous meetings, RSC to chair. Done. The meeting had been arranged and had now happened. The wiki page for the meeting had been updated. The actions ensuing from this meeting have been added to actions from previous meetings listed here to ensure that they do not get overlooked. ACTION: ACD to check with JAC whether they have dispatched all the tapes or there are more to be sent. Done. All the tapes and have been dispatched and subsequently received. Actions carried forward from 13/2/2015 meeting: ----------------------------------------------- ACTION: NJC (was RPB; before that MAR) to progress [was: 'have a look at'] the status of non-survey releases for registered projects. Continues. MAR had compiled a list of non-survey projects that could be released and sent it to RPB. RPB had started creating releases for some 8-12 non-survey releases. However, they had crashed about half way through this preparation because of what appeared to be a schema problem. NJC should investigate this problem. ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues: this action still has low priority. ACTION: ETWS/NCH purchase a new license for mxODBC. Continues. ETWS had raised this with NCH and was waiting for a reply. ETWS should remind NCH. ACTION: NJC noted that VISTA mosaicing crashes with normal filtering, which he should fix. Continues. NJC had implemented a fix but had yet to check it into SVN. ACTION: RPB to compile a Web page listing all the completed releases. Continues. ACTION: NJC to produce roadmap for future VSA releases so as to plan for hardware requirements. Continues. Actions resulting from the VVV DR3 brainstorming meeting held on 24/2/15: ------------------------------------------------------------------------ ACTION: MAR to analyse the load and query performance on the public servers (to asses the feasibility of using lower performance machines for hosting public archives than for curation). ACTION: NJC to speak to the VVV PIs, notably Dante, about the schedule for and expected number of remaining VVV releases. ACTION: MAR to speak to Mike Irwin about whether any re-processing of the VVV data is anticipated. ACTION: RPB to experiment with ingest via partitioned tables, along the lines of the SSIS paper. ACTION: RSC, RGM: (i) RSC to write a succinct summary of the ingest and outgest problem that is is principal bottleneck with the VVV processing (briefly the ingest and outgest speed is limited to approx. 100 Mbit/sec for no apparent reason) and (ii) RGM to compose an e-mail message around this summary and send it to Ani Thakar. Specific points and new actions: -------------------------------- Project management: * RGM inquired about the rapidity with which lto tapes could be purchased because there may be a small amount of money that needs to be spent quickly at the end of March. The mellifluous RPB replied that he thought they could be bought quite speedily. WFCAM, VISTA and VST updates: * WFCAM has now been removed from UKIRT for a couple of months. CASU were up-to-date with their processing. Networking: * Nothing really to report; the networks were working. There had been a short, scheduled outage lasting a couple of hours. WSA/VSA/OSA/GES Operations: * ETWS reported: - Ingest of the major batch of VVV P92/93 data has finished. The files/dates missing due to space restrictions on the production servers now need to be determined and ingested. - The objID update for the VVV DR3 stacks and tiles has finished, Calculation of the constraints is in progress but taking longer than expected. * RPB reported mellifluously: - The VMC release should be finished and released in a few weeks. - ATLAS was having problems creating neighbour tables, but the problems were silly programming issues rather than anything serious. Hardware and Systems: * The ever-mellifluous RPB noted that there had been an unexplained temperature warning in C2. It had disappeared once the (open, wire-frame) door to the cabinet containing the thermometer had been opened. Software: * NJC had been working on integrating the new extinction maps into the WFAU surveys. He planned to write-up this work and report to Oscar Gonzalez (the author of the extinction surveys). There had been some discussion of putting all the extinction maps in a separate database called (with stunning originality) 'extinction'. In addition he had made a few fixes to various items. * ETWS noted that he too had made some minor updates. He had also finalised the Higher-CU monitor page creator. Review of Deadlines: * Progress towards forthcoming deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines ACTION: NJC noted that he had received a report of a problem with the ESO VVV DR1 release which he should investigate. In one pointing the merged band catalogue contained fewer sources than expected. ACTION: NJC to ask Matt Jarvis whether there will be a VIDEO P93 Team Release. ACTION: ACD. In a couple of months NJC would be presenting an overview the WFAU archives relevant to Galactic astronomy at a conference. ACD should consider what NJC might say about the GES archive at this meeting. Survey Data Release: * None had had been made but VMC and ATLAS releases were imminent. * The GESiDR2iDR3 Consortium release had been announced. Non-survey Data Release: * A bunch were largely prepared but were waiting on NJC fix the schema (see above). Astrogrid deployment, VO & Data Analysis services: * Nothing to report. Miscellaneous: * The next WFAU Quarterly Meeting would be held on Tuesday 24 March. ACD, Original: 27/2/15, Revised: 27/2/15.