From acd@roe.ac.uk Tue Feb 2 12:01:07 2016 Date: Tue, 2 Feb 2016 12:01:03 +0000 From: Clive Davenhall To: Bob Mann , Andy Lawrence , "Read, Mike" , Eckhard Sutorius , Mark S. Holliman , Nicholas Cross , Nigel C. Hambly , Rob Blake , Ross Collins , Stelios Voutsinas , Dave Morris , Clive Davenhall Cc: "dante@astro.puc.cl" , James Dunlop , "j.p.emerson@qmul.ac.uk" , ktn keithnoddle.org , "Lorenzo.Rimoldini@unige.ch" , "mcioni@aip.de" , "matt.jarvis@astro.ox.ac.uk" , "mike@ast.cam.ac.uk" , "nigel.metcalfe@durham.ac.uk" , "norman@astro.gla.ac.uk" , "pwl@star.herts.ac.uk" , "rgm@ast.cam.ac.uk" , "Simon.Dye@nottingham.ac.uk" , "s.j.warren@ic.ac.uk" , "tom.shanks@durham.ac.uk" Subject: WFAU VDFS Science Archives progress meeting minutes, 22 January 2016 WFAU VDFS Science Archives progress meeting minutes, 22 January 2016 Present: MSH, ETWS, RPB, STV, DMR, ACD Apologies: RGM, MAR, NJC, RSC, KTN, AL DoNM: 10am Friday 5 February 2016 in the Villas meeting room, MAR in the chair (room is booked). The meeting was held on a auspicious day for ETWS. It was his birthday and moreover this birthday marked his completion of a single round of the Mayan calendar, with the culmination of a single combined cycle of their 365 and 260 days calendars. By this measure he is now both one year old and becomes an elder. Actions discharged this time: ---------------------------- ACTION: MAR to arrange telecon with CASU. Done. A telecon had been arranged, and had been held on ?????. It had been useful and productive. Actions carried forward from 8/1/2016 meeting: --------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues: this action still has low priority. ACTION: NJC noted that VISTA mosaicing crashes with normal filtering, which he should fix. Continues. NJC had implemented a fix. Though it completed successfully when he ran some tests it crashed when run 'in anger' on a complete ingest dataset. NJC should revisit and correct his fix. ACTION: NJC noted that he had received a report of a problem with the ESO VVV DR1 release which he should investigate. In one pointing the merged band catalogue contained fewer sources than expected. Continues. This action was very low on NJC's priority list and unlikely to be implemented in the foreseeable future. ACTION: MAR, NJC. Either MAR or NJC should go to the VMC meeting to be held at Keele University during June 2016. Continues. NJC volunteered to go. Actions resulting from the VVV DR3 brainstorming meeting held on 24/2/15: ------------------------------------------------------------------------ ACTION: RPB to experiment with ingest via partitioned tables, along the lines of the SSIS paper. Deferred (changed back from 'continues'). This action has now been overtaken by events: the VVV DR4 (was DR3) processing would now be done differently. However, the technique was still potentially useful for future work and should be investigated before the next VVV release, which is time away. ACTION: RGM (was RSC, RGM): (i) RSC to write a succinct summary of the ingest and outgest problem that is the principal bottleneck with the VVV processing (briefly the ingest and outgest speed is limited to approx. 100 Mbit/sec for no apparent reason) and (ii) RGM to compose an e-mail message around this summary and send it to Ani Thakar. Continues. RSC had composed a succinct summary and sent it to RGM. RGM had emailed Ani Thakar but no reply to date. RGM had sent a reminder but had still received no response. RGM had also contacted Alex Szalay and not received a reply from him either. He would consider how to proceed. Specific points and new actions: -------------------------------- Project management: * The review panel for the grant application to continue the VSA/WSA work were meeting on the same day as the present meeting (Friday 22 January). RGM and MAR were in London attending this meeting. WFCAM, VISTA and VST updates: * Nothing to report. Networking: * Nothing to report; everything appeared to be working nominally. WSA/VSA/OSA/GES Operations: * ETWS reported: - Following the release of additional data by Mike Irwin data transfer was now notionally complete as follows: OSA to November 2015; VSA to September 2015. However, some OSA ingests failed and ETWS is still tracking down the various problems. - Curation tasks CU2 and CU3 had been completed for the VSA and CU4 would be run next week. - Pyfits and astropy are incompatible in the same python script and astropy should be used throughout. Hardware and Systems: * MSH noted: - The new NAS boxes had been installed and were working. - The new AstroTROP kit would be installed next week. - The renovations and alterations to the server room were almost complete. - Some idiot had lost MSH's 'tool kit' box of nuts and bolts etc. for the storage racks and Linux installation CD-ROMs, which was most annoying. - No further progress had been made obtaining additional IP addresses; discussions were now believed to be proceeding at a senior level. Software: * NJC reported: - some problems with recent ESO releases had been sorted out, - some patches had been made to VIKING, - synoptic ??? was ready for the VVV release, - and a few other things. * ETWS added that the third-party software had been upgraded on all machines. He reiterated that all instances of pyfits needed to replaced with astropy. Review of Deadlines: * Progress towards forthcoming deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * VHS P94 had been released to the Consortium, but little feedback had been received. All the P94 releases were now out apart from VIDEO. Effort would now concentrate on P95. * A VPHAS release had been made. ACTION: DMR, NJC, RPB ensure that the new-style OSA release has access to all the database tables that it needs; specifically resolve the problems encountered with neighbour tables. ACTION: MSH, DMR install and configure the Nagios[1] system monitoring package to monitor the availability of our database services. Non-survey Data Release: * RPB reported: - 7 non-survey releases had been attempted, of these: 3 worked: U14BUA2, U14BUA10, U14BUA17; 4 did not: U14BUA15, U14BUA16, U14BUA18, U14BUA19. ACTION: RPB to pass the errors to NJC. - The generic 'schema change' message that had plagued earlier non-survey releases has now been resolved. Astrogrid deployment, VO & Data Analysis services: * MSH reported that a new bug had appeared in the AstroGrid registry, making a registry TAP service increasingly desirable. Such a service was still under development and initially a test registry would be set up. ACTION: DMR, STV MSH to liaise over setting up a test registry TAP service. Miscellaneous: * NCH noted that significant modifications were needed to the OSA Web pages to correspond to the new-style interface. ACD, 22/1/16. [1] - https://www.nagios.org/