From acd@roe.ac.uk Sun Dec 6 14:13:42 2015 Date: Sun, 6 Dec 2015 14:13:43 +0000 From: Clive Davenhall To: Bob Mann , Andy Lawrence , "Read, Mike" , Eckhard Sutorius , Mark S. Holliman , Nicholas Cross , Nigel C. Hambly , Rob Blake , Ross Collins , Stelios Voutsinas , Dave Morris , Clive Davenhall Cc: "dante@astro.puc.cl" , James Dunlop , "j.p.emerson@qmul.ac.uk" , ktn keithnoddle.org , "Lorenzo.Rimoldini@unige.ch" , "mcioni@aip.de" , "matt.jarvis@astro.ox.ac.uk" , "mike@ast.cam.ac.uk" , "nigel.metcalfe@durham.ac.uk" , "norman@astro.gla.ac.uk" , "pwl@star.herts.ac.uk" , "rgm@ast.cam.ac.uk" , "Simon.Dye@nottingham.ac.uk" , "s.j.warren@ic.ac.uk" , "tkerr@jach.hawaii.edu" , "tom.shanks@durham.ac.uk" Subject: WFAU VDFS Science Archives progress meeting minutes, 4 December 2015 WFAU VDFS Science Archives progress meeting minutes, 4 December 2015 Present: MSH, ETWS, RPB, MAR, NJC, RGM, ACD Apologies: RSC, KTN, AL, STV, DMR DoNM: 10am Friday 18 December 2015 in the Villas meeting room, MAR in the chair (room is booked). Actions discharged this time: ---------------------------- ACTION MSH to pass to DMR his list of UCD 1 / UCD 1+ equivalences and a complete list of columns in all the WFAU databases. Done. Actions carried forward from 30/10/2015 meeting: ----------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues: this action still has low priority. ACTION: NJC noted that VISTA mosaicing crashes with normal filtering, which he should fix. Continues. NJC had implemented a fix. He had run some tests, which completed successfully. However, the software had still to be run 'in anger' on a complete ingest dataset. ACTION: NJC noted that he had received a report of a problem with the ESO VVV DR1 release which he should investigate. In one pointing the merged band catalogue contained fewer sources than expected. Continues. This action was very low on NJC's priority list. Actions resulting from the VVV DR3 brainstorming meeting held on 24/2/15: ------------------------------------------------------------------------ ACTION: RPB to experiment with ingest via partitioned tables, along the lines of the SSIS paper. Deferred (changed back from 'continues'). This action has now been overtaken by events: the VVV DR3 processing would now be done differently. However, the technique was still potentially useful for future work and should be investigated, but not as a matter of urgency. ACTION: RGM (was RSC, RGM): (i) RSC to write a succinct summary of the ingest and outgest problem that is the principal bottleneck with the VVV processing (briefly the ingest and outgest speed is limited to approx. 100 Mbit/sec for no apparent reason) and (ii) RGM to compose an e-mail message around this summary and send it to Ani Thakar. Continues. RSC had composed a succinct summary and sent it to RGM. RGM had emailed Ani Thakar but no reply to date. RGM had sent a reminder but had still received no response. RGM had also contacted Alex Szalay and not received a reply from him either. He would consider how to proceed. Specific points and new actions: -------------------------------- Project management: * RGM would shortly be holding Annual Reviews for all WFAU staff. A date had been agreed for each member of staff's review. WFCAM, VISTA and VST updates: * Nothing to report. Networking: * The network was operating nominally. ETWS also mentioned that JANET had experienced a serious outage in the Manchester area during the previous week. It was caused by a DDOS attack from multiple sources outside the JANET network. WSA/VSA/OSA/GES Operations: * RPB noted the following: - VMC P94 was in the final phase of curation and hence nearly complete. It should be be released during the week commencing Mon. 7/12/15. - Neighbour tables were currently being created for the VPHAS release and hence it should complete soon. ACTION: MAR, NJC. Either MAR or NJC should go to the VMC meeting to be held at Keele University during June 2016. - A fix was being implemented for the problem encountered generating the VVV DR4 synoptic source tables. - Preparation of a WSA non-survey release crashed because of problem in the Astropy library. This problem only occurred on some servers, not all of them. Hardware and Systems: * MSH noted: - As part of the continuing upgrade to the server room all WFAU systems had been moved to new racks. Most machines had now been restored to normal operation. The work to the server room was scheduled to complete in mid-December. There were currently cooling problems in the server room, but it was hoped that these were temporary and associated with the temporary partitions in place while construction work was underway. The previous KVM switch no longer works. A replacement was needed as a matter of some urgency and should be Internet-enabled. ACTION: ACD to speak to KTN about funding a new KVM switch. - Discussions about upgrading the site network continued. Software: * NJC reported: - He had made numerous minor fixes to a variety of pieces of software. - He had started to develop routines to convert files into a format suitable for ingest to the SWarp software for re-sampling and co-adding images. (For SWARP see: http://www.astromatic.net/software/swarp) * ETWS noted that NCH has bought mxODBC license upgrades and ETWS has installed these on all servers except those currently curating releases. Lockheed Martin: * MSH noted that obtaining SQL Server for Lockheed Martin was turning out to be problematic and expensive, basically because they are not an academic institution and hence do not qualify for an academic licence. It would be necessary to obtain SQL Server 2014, which has not been tried with WFAU software. Licensing might also be complicated and expensive. It was not obvious whether the licences would be per user, per core or per machine. An mxODBC licence would also be needed, which again would not qualify for an academic discount. Review of Deadlines: * Progress towards forthcoming deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines ACTION: ACD would organise a meeting to discuss the dead-locking that results from attempting to run more than one different curation activity simultaneously on the same server. Attendees should include ETWS, RPB, MAR, NJC and RSC. Survey Data Release: * The ATLAS DR2 ESO release had been completed. Non-survey Data Release: * None. Astrogrid deployment, VO & Data Analysis services: * The replacement TAP service currently under development was expected to become available shortly. Miscellaneous: * MSH noted that two EUCLID meetings would be held during December: a EUCLID System Team meeting on 15-16/12/15 and a EUCLID UK meeting on 17-18/12/15. * NJC noted that he had recently attended the ESO/ESA Science Operations meeting and would circulate notes shortly. ACD, original: 4/12/15, revised: 6/12/15.