From acd@roe.ac.uk Tue Apr 4 11:28:38 2017 Date: Tue, 4 Apr 2017 10:28:26 +0000 From: Clive Davenhall To: Bob Mann , Andy Lawrence , "Read, Mike" , Eckhard Sutorius , Mark S. Holliman , Nicholas Cross , Nigel C. Hambly , Rob Blake , Ross Collins , Stelios Voutsinas , Dave Morris , Clive Davenhall Cc: "dante@astro.puc.cl" , James Dunlop , "j.p.emerson@qmul.ac.uk" , ktn keithnoddle.org , "Lorenzo.Rimoldini@unige.ch" , "mcioni@aip.de" , "matt.jarvis@astro.ox.ac.uk" , "mike@ast.cam.ac.uk" , "nigel.metcalfe@durham.ac.uk" , "norman@astro.gla.ac.uk" , "pwl@star.herts.ac.uk" , "rgm@ast.cam.ac.uk" , "Simon.Dye@nottingham.ac.uk" , "s.j.warren@ic.ac.uk" , "tom.shanks@durham.ac.uk" Subject: WFAU VDFS Science Archives progress meeting minutes, 24 March 2017 WFAU VDFS Science Archives progress meeting minutes, 24 March 2017 Present: NJC, MSH, MAR, RPB, RSC, ACD Apologies: RGM, ETWS, STV, DMR, KTN, AL, NCH DoNM: 10:00 am Friday 14 April 2017, Villas Meeting Room MAR in the chair (room is booked). Actions discharged this time: ----------------------------- ACTION: MSH to set up a meeting with the SDSS team to discuss SQL Server on Linux. Done. A date had been set. ACTION: RPB to rerun the non-survey databases which failed previously but have now been fixed. Done. ACTION: RPB to configure the AMANDA backup system to e-mail a copy of the log file that it generates during each run to ACD. Done. ACD was now receiving copies of AMANDA's log files. ACTION: RPB to check if the AMANDA backup system has an option to generate a checksum (or similar) for the datasets that it backs up. Done. AMANDA does not have an option to generate checksums. ACTION: NJC to check that the latest browser for the VIKING survey, VIKINGZYSELJv20170124 is correct and that the corresponding schema is checked in to SVN. This browser is available on disk50. Done. ACTION: MSH to generate coverage maps for the VPHASv20170222 release. Done. Actions carried forward from 3/2/2017 meeting: ---------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan DR13 is installed but not yet cross-mounted. However, we still need to decide which SDSS releases to keep so the present action should remain. ACTION: MAR to write some H-ATLAS documentation once H-ATLAS is available. Continues. ACTION: RPB to check with CASU whether they had any WFCAM observations made during the gap in the Edinburgh tapes. It was suggested we might like to check here first! Continues. Actions carried forward from the Backup Strategy Meeting held on 27/1/2017: -------------------------------------------------------------------------- ACTION: RPB to make a full tape backup of VVV promptly after it completes and henceforth make a copy annually. Continues. ACTION: RPB to investigate the options for incremental backups of VVV while it is being created, either monthly and/or at convenient junctures between curation steps. Continues. RPB had tried to make a complete backup but it failed and processing had restarted before he could identify the problem. Therefore the database was currently at risk. Another backup would be attempted once the stacks had been ingested. ACTION: NJC, In the context of the VVV, to check which curation steps can be paused without misadventure and hence compile a list of points at which backups may be made during the ingest. The list should be added to the twiki. Continues. NJC had a list of suitable pause points in his head. He would write it down in a twiki page. ACTION: NJC (was NJC, ETWS, MSH, MAR) to create a twiki page comprising a list of all the external datasets that WFAU hosts. The information included for each dataset should include: its name, size, where and when it originates and what WFAU uses it for. Continues. ETWS had created a page which is largely complete. NJC would check and add any remaining details. Once NJC is satisfied with the list the action would be complete but henceforth ETWS would periodically check the list. ACTION: RPB to create a twiki page documenting the backup procedure. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. Tuesday 28 February 2017 was subsequently agreed as a target date for completing these notes. Continues. ACTION: RPB to create a twiki page documenting the procedure to restore datasets from a backup. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. Continues. ACTION: ACD to obtain an estimate of the likely cost of a fire safe. Continues. ACD was awaiting an estimate of the volume of the tapes to be stored from RPB. ACTION: RGM to ask Steven Duffield to review the WFAU backup procedures. Continues. This action could not progress until RPB had written notes describing the backup and recovery procedures. ACTION: RPB to create a tape backup of the Gaia commissioning data at terra:/disk1/gaia/KAKAI_OLD and hand the tape to NCH for safe-keeping. Continues. ACTION: ETWS to estimate the disk space required for new files during the next the next couple of months and pass it to MSH. Continues. ACTION NJC: Check and update the required neighbours tables for recent releases. Continues. In particular all databases should have Gaia neighbour tables. ACTION RPB: Generate a neighbours table for the most recent WSA release. Continues. ACTION NJC: Check with Maria-Rosa Cioni that the standard Gaia neighbours table would be suitable for her purposes. Continues. Specific points and new actions: -------------------------------- Project management: * The scheduling of VDFSSA meetings was briefly discussed. No strong preferences for an alternative day or time emerged, though it was noted that Monday morning, Wednesday afternoon and Thursday morning were unsuitable because of other commitments. Provisionally it was decided to continue with the current time of 10:00 am on Friday morning but to enforce the fortnightly cadence more strictly in order to avoid clashes with LSST meetings. However, any final decision would be made by RGM whose availability was more constrained. WFCAM, VISTA and VST updates: * Nothing to report. Networking: * ROE will be having a planned network outage starting at 8:00 am on Tuesday 28/3/17 which is likely to last for a few minutes. MSH plans to utilise this lacuna to move the WFAU machines to the new 10Gbe external switch, though this may take longer than a few minutes. WSA/VSA/OSA/GES Operations: * RPB reported: - VSA: All P97 data had been copied except VVV and Quality Control could proceed. Preparation of the VMS P97 release had started and preparation could start for the other surveys, VVV excepted. - OSA: the VPHAS team release was available and NJC could start work on the ESO release. * RSC noted the March 2017 GES Newsletter had recently been circulated and that it intimated that the iDR5 would be available from WFAU by mid-July. Consequently the data should be available for ingest by WFAU from late June or early July. ACD noted that a GES Consortium all-hands meeting would be held in Catania during 4-8 September 2017. ACTION RSC: check the schedule for availability of iDR5 data with Clare Worley and Anna Hourihane. * ETWS was not present and so could not report. Hardware and Systems: * MSH reported that three new machines had been delivered and were now operational: - khaba, a new gateway server, - pepy, a processing server, - ramses19 a storage server. Data were being copied from ramses3 (which is due to retire) to ramses19. Finally, unas was back on-line. Review of Software Development Deadlines: * ACD and NJC briefly discussed the software items that NJC is currently working on. The prioritised list of software items is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/PrepareSoft Other Software: * NJC noted that he had worked on several items, mostly fixes to ESO releases. Review of Survey Release Deadlines: * Progress towards forthcoming survey deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * The following surveys had been released: - VMCDR4: a public release corresponding to team release VMCv20160701, - VMCv20170109: a team release including data up to and including P96. Non-survey Data Release: * The following non-survey releases had been made: U/15B/NA04 U/16A/H02 U/16A/H07 U/16A/H06 U/16A/H04 U/15B/H01 U/15B/UA02A U/16A/H08 U/15B/UA14 The following non-survey release had failed and NJC should investigate: U/16A/UA12 Astrogrid deployment, VO & Data Analysis services: * MSH noted that the CDS had requested that WFAU include tag lines in its releases but these would have to await the new TAP service. Miscellaneous: * A WFAU leaving meal for Marcus Ebert was planned for Wednesday 29/3/17. * RSC noted that he would be in the Observatory all week for the week beginning Monday 27/3/17 but would then be away for 2 weeks. ACD, 24/3/17.