From acd@roe.ac.uk Tue Feb 16 12:06:00 2016 Date: Tue, 16 Feb 2016 12:05:51 +0000 From: Clive Davenhall To: Bob Mann , Andy Lawrence , "Read, Mike" , Eckhard Sutorius , Mark S. Holliman , Nicholas Cross , Nigel C. Hambly , Rob Blake , Ross Collins , Stelios Voutsinas , Dave Morris , Clive Davenhall Cc: "dante@astro.puc.cl" , James Dunlop , "j.p.emerson@qmul.ac.uk" , ktn keithnoddle.org , "Lorenzo.Rimoldini@unige.ch" , "mcioni@aip.de" , "matt.jarvis@astro.ox.ac.uk" , "mike@ast.cam.ac.uk" , "nigel.metcalfe@durham.ac.uk" , "norman@astro.gla.ac.uk" , "pwl@star.herts.ac.uk" , "rgm@ast.cam.ac.uk" , "Simon.Dye@nottingham.ac.uk" , "s.j.warren@ic.ac.uk" , "tom.shanks@durham.ac.uk" Subject: WFAU VDFS Science Archives progress meeting minutes, 5 February 2016 WFAU VDFS Science Archives progress meeting minutes, 5 February 2016 Present: ETWS, RSC, RGM (part), ACD Apologies: MAR, NJC, MSH, RPB, STV, DMR, KTN, AL Additional text supplied for the minutes: MSH DoNM: 10am Friday 19 February 2016 in the Villas meeting room, MAR in the chair (room is booked). Actions discharged this time: ---------------------------- ACTION: MAR, NJC. Either MAR or NJC should go to the VMC meeting to be held at Keele University during June 2016. Done. NJC had volunteered to go. Actions carried forward from 5/2/2016 meeting: --------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues: this action still has low priority. ACTION: NJC noted that VISTA mosaicing crashes with normal filtering, which he should fix. Continues. NJC had implemented a fix. Though it completed successfully when he ran some tests it crashed when run 'in anger' on a complete ingest dataset. NJC should revisit and correct his fix. ACTION: NJC noted that he had received a report of a problem with the ESO VVV DR1 release which he should investigate. In one pointing the merged band catalogue contained fewer sources than expected. Continues. This action was very low on NJC's priority list and unlikely to be implemented in the foreseeable future. ACTION (evolved from previous action): NJC register for the VMC meeting to be held at Keele University during June 2016. Continues. ACTION: DMR, NJC, RPB ensure that the new-style OSA release has access to all the database tables that it needs; specifically resolve the problems encountered with neighbour tables. Continues. ACTION: MSH, DMR install and configure the Nagios[1] system monitoring package to monitor the availability of our database services. Continues. ACTION: RPB to pass to NJC the errors encountered while trying to make non-survey releases U14BUA15, U14BUA16, U14BUA18 and U14BUA19. Continues. ACTION: DMR, STV MSH to liaise over setting up a test registry TAP service. Continues. Actions resulting from the VVV DR3 brainstorming meeting held on 24/2/15: ------------------------------------------------------------------------ ACTION: RPB to experiment with ingest via partitioned tables, along the lines of the SSIS paper. Deferred (changed back from 'continues'). This action has now been overtaken by events: the VVV DR4 (was DR3) processing would now be done differently. However, the technique was still potentially useful for future work and should be investigated before the next VVV release, which is time away. ACTION: RGM (was RSC, RGM): (i) RSC to write a succinct summary of the ingest and outgest problem that is the principal bottleneck with the VVV processing (briefly the ingest and outgest speed is limited to approx. 100 Mbit/sec for no apparent reason) and (ii) RGM to compose an e-mail message around this summary and send it to Ani Thakar. Continues. RSC had composed a succinct summary and sent it to RGM. RGM had emailed Ani Thakar but no reply to date. RGM had sent a reminder but had still received no response. RGM had also contacted Alex Szalay and not received a reply from him either. He would consider how to proceed. Specific points and new actions: -------------------------------- Project management: * Nothing to report. WFCAM, VISTA and VST updates: * Mike Irwin had mentioned that, contrary to previous practice, WFCAM would remain on UKIRT for the whole of period 2016A, rather than being removed for maintenance. Networking: * Nothing to report; everything appeared to be working nominally. WSA/VSA/OSA/GES Operations: * ETWS reported: - OSA: ingest of the uncorrected data for P93 is in progress. - VVV: started processing the May 2015 data; ingest of the April 2015 data is nearly finished. - VSA (other): the latest data have been transferred (to October 2015). - WSA: the latest metadata is ingested up to June 2015. * (Note added to the minutes) NJC reported that the VVV DR4 and VIKING v20151230 ESO releases were both in preparation but still at a preliminary stage. The VIKING release is likely to require several more weeks and the VVV DR4 release about six weeks. Hardware and Systems: * MSH noted: - Ramses12 died due to a power distributor failure. Eclipse have fixed the problem (though the system is not on maintenance). - Refurbishment of C2 is nearly complete and the official handover is expected to take place sometime next week (while MSH and RPB are away). We will soon need to start moving all WFAU out of the Crawford lab. back into C2, and also to schedule an uplift for the retired machines. Software: * ETWS had made some minor fixes and enhancements of to create file lists that can be used directly for the different OSA ingests (uncorrected, fixed and list-driven data). Review of Deadlines: * Progress towards forthcoming deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * None. Non-survey Data Release: * None. Astrogrid deployment, VO & Data Analysis services: * Nothing to report. Miscellaneous: * ACD noted that he would be on leave 9-15 February inclusive. ACD, 5/2/16. [1] - https://www.nagios.org/