From mar@roe.ac.uk Wed Mar 23 11:37:39 2016 Date: Wed, 23 Mar 2016 11:37:27 +0000 From: Mike Read To: VDFS Science Archive Team -- Clive Davenhall , dmr@roe.ac.uk, Eckhard Sutorius , Mark Holliman , Nicholas Cross , Nigel Hambly , Rob Blake , Ross Collins , Stelios Voutsinas Cc: CCs for VDFS -- Alastair Edge , Andrew Lawrence , Bob Mann , Dante Minniti , James Dunlop , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Maria-Rosa Cioni , matt.jarvis@astro.ox.ac.uk, Mike Irwin , Nigel Metcalfe , Norman Gray , Phil Lucas , Richard McMahon at IOA , Simon Dye , Stephen Warren , Tom Kerr , Tom Shanks Subject: WFAU VDFS Science Archives progress meeting minutes, 18 March 2016 WFAU VDFS Science Archives progress meeting minutes, 18 March 2016 Present: RSC, MAR, MSH, ACD, NJC, RPB Apologies: RGM, STV, DMR, KTN, AL, ETWS DoNM: 10:00 am Friday 1st April 2016 in the Villas meeting room. ACD in the chair (room is booked). Actions discharged this time: ---------------------------- ACTION: MSH, ETWS, RPB to investigate intermittent error running VVV on ramses16: 'there is insufficient memory available in the buffer pool'. Discharged (for now) as the issue had not reoccurred. Actions carried forward from 4/3/2016 meeting: --------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues: this action still has low priority. ACTION: NJC noted that VISTA mosaicing crashes with normal filtering, which he should fix. Continues. NJC had implemented a fix. Though it completed successfully when he ran some tests it crashed when run 'in anger' on a complete ingest dataset. NJC should revisit and correct his fix. ACTION: NJC noted that he had received a report of a problem with the ESO VVV DR1 release which he should investigate. In one pointing the merged band catalogue contained fewer sources than expected. Continues. This action was very low on NJC's priority list and unlikely to be implemented in the foreseeable future. ACTION (evolved from previous action): NJC register for the VMC meeting to be held at Keele University during June 2016. NJC had emailed Maria-Rosa but not yet registered. Continues. ACTION: DMR, NJC, RPB ensure that the new-style OSA release has access to all the database tables that it needs; specifically resolve the problems encountered with neighbour tables. Continues, RPB was fixing up VPHAS ACTION: MSH, DMR install and configure the Nagios[1] system monitoring package to monitor the availability of our database services. Continues. Progress is stalled until additional IP addresses are made available to WFAU. ACTION: NJC to fix the errors encountered while trying to make non-survey releases U14BUA15, U14BUA16, U14BUA18 and U14BUA19. Continues, NJC thought the issues had been resolved. RPB to try the releases again. ACTION: NJC to update Twiki page on future VISTA surveys. Continues. Actions resulting from the VVV DR3 brainstorming meeting held on 24/2/15: ------------------------------------------------------------------------ ACTION: RPB to experiment with ingest via partitioned tables, along the lines of the SSIS paper. Deferred (changed back from 'continues'). This action has now been overtaken by events: the VVV DR4 (was DR3) processing would now be done differently. However, the technique was still potentially useful for future work and should be investigated before the next VVV release, which is time away. ACTION: RGM (was RSC, RGM): (i) RSC to write a succinct summary of the ingest and outgest problem that is the principal bottleneck with the VVV processing (briefly the ingest and outgest speed is limited to approx. 100 Mbit/sec for no apparent reason) and (ii) RGM to compose an e-mail message around this summary and send it to Ani Thakar. Continues. RSC had composed a succinct summary and sent it to RGM. RGM had emailed Ani Thakar but no reply to date. RGM had sent a reminder but had still received no response. RGM had also contacted Alex Szalay and not received a reply from him either. He would consider how to proceed. Specific points and new actions: -------------------------------- Project management: * Attempts were being made to schedule the next WFAU quarterly. Everyone should ensure their absences are kept up to date on the Twiki. WFCAM, VISTA and VST updates: * Nothing to report. Networking: * Nothing to report; everything appeared to be working nominally. WSA/VSA/OSA/GES Operations: * RPB noted that all VVV data were now together in a single database. Quality bit flagging and tile-pawprints could now be run. ACTION: RPB to backup VVV. ACTION: RPB to restore VSA replication and check OSA replication. ACTION: MAR to contact Phil/Dante about VVV QC. * The P95 ATLAS release was running. Hardware and Systems: * Apart from ramses12 needing a re-start there had been no problems. Software: * NJC reported further work on the list driven pipeline, mosaicing and some fixes for the ATLAS release. Review of Deadlines: * Progress towards forthcoming deadlines was reviewed offline. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * The initial UHS release had been made. * The P95 VMC database had been released. Non-survey Data Release: * None. * The Nial Tanvir VISTA project release had finished but space needed to be freed up to host it on the public server. Astrogrid deployment, VO & Data Analysis services: * Nothing to report Miscellaneous: * Nothing for the minutes.