From mar@roe.ac.uk Fri Jul 28 14:18:22 2017 Date: Fri, 28 Jul 2017 14:18:14 +0100 From: Mike Read To: VDFS Science Archive Team -- Clive Davenhall , dmr@roe.ac.uk, Eckhard Sutorius , Mark Holliman , Nicholas Cross , Nigel Hambly , Rob Blake , Ross Collins , Roy Williams , Stelios Voutsinas Subject: WFAU VDFS Science Archives progress meeting minutes, 21 July 2017 WFAU VDFS Science Archives progress meeting minutes, 21 July 2017 Present: MAR, RPB, ETWS, RDW, ACD Apologies: MSH, RSC, DMR, KTN, AL, NCH, STV, NJC, RGM DoNM: 10:00 am Friday 4th August 2017, Villas Meeting Room ACD in the chair (room is booked). Actions discharged this time: ----------------------------- ACTION ACD: check to establish the state of the order for the new NAS box. Done, NAS box had been ordered. Actions carried forward from 23/6/2017 meeting: ---------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan DR13 is installed but not yet cross-mounted. However, we still need to decide which SDSS releases to keep so the present action should remain. ACTION: MAR to write some H-ATLAS documentation once H-ATLAS is available. Continues. ACTION: RPB to check with CASU whether they had any WFCAM observations made during the gap in the Edinburgh tapes. It was suggested we might like to check here first! Continues. ACTION: MAR to arrange the date of the next CASU/WFAU telecon with Mike Irwin. Continues. ACTION: MAR to check that any data on RAMSES3 and RAMSES5 which is still required is copied elsewhere. Continues. This item now requires urgent attention. Actions carried forward from the Backup Strategy Meeting held on 27/1/2017: -------------------------------------------------------------------------- ACTION: RPB to make a full tape backup of VVV promptly after it completes and henceforth make a copy annually. Continues. ACTION: NJC, In the context of the VVV, to check which curation steps can be paused without misadventure and hence compile a list of points at which backups may be made during the ingest. The list should be added to the twiki. Continues. NJC had a list of suitable pause points in his head. He would write it down in a twiki page. ACTION: NJC (was NJC, ETWS, MSH, MAR) to create a twiki page comprising a list of all the external datasets that WFAU hosts. The information included for each dataset should include: its name, size, where and when it originates and what WFAU uses it for. Continues. ETWS had created a page which is largely complete. However, a few queries remain and NJC will liaise with RPB and MAR to resolve them. Once NJC is satisfied with the list the action would be complete but henceforth ETWS would periodically check the list. ACTION: RPB to create a twiki page documenting the backup procedure. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. Tuesday 28 February 2017 was subsequently agreed as a target date for completing these notes. A stub page for RPB to complete is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/BackupsProcedures Continues. ACTION: RPB to create a twiki page documenting the procedure to restore datasets from a backup. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. A stub page for RPB to complete is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/BackupsRecovery Continues. ACTION: ACD to obtain an estimate of the likely cost of a fire safe. Continues. ACD now had an estimate of the volume of tapes to be stored. The next step is to investigate suitable safes. ACTION: RGM to ask Steven Duffield to review the WFAU backup procedures. Continues. This action could not progress until RPB had written notes describing the backup and recovery procedures. ACTION: RPB to create a tape backup of the Gaia commissioning data at terra:/disk1/gaia/KAKAI_OLD and hand the tape to NCH for safe-keeping. RPB had staged the data to another disk ready for tape backup. Continues. ACTION (amended): RPB Generate a Gaia DR1 neighbours table for the most recent WSA release and the VVV DR4 release. Continues. ACTION (amended): MSH, RSC, NJC (was all) run further tests of SQL server on Linux. Continues. Informal testing has continued, but more formal tests are needed. ACTION (amended): MSH (was all) set up a meeting to review tests of SQL server on Linux prior to the next telecon with John Hopkins. Continues. Specific points and new actions: -------------------------------- Project management: * Nothing for the minutes. WFCAM, VISTA and VST updates: * All the instruments were working as expected. Networking: * All WFAU networks were working nominally. WSA/VSA/OSA/GES Operations: Following on from the VMC telecon it was noted that the last 3 months of P98 VISTA processed data would shortly be made available by CASU. * ETWS reported that following a few issues the OSA had been backed up and the final CU4s and illumination corrections were being applied before the SQL scripts generated on the OSAP95 database were run on the load database to get them back in sync. It was thought that the LAS and GCS deep stacking had reached completion and was awaiting NJC's input to progress. The issue with VVV tiles was still being investigated. * ACD reported that the latest GES spectra had been received. Hardware and Systems: * RPB reported that ramses12 (OSA) had frozen over the weekend reporting errors with the system disk. It was back up and running but it's probably time to think about moving the OSA to another server. Status of Backups: * RPB reported that he been examining the backups over the last week or so and it seems to be working fine. There was a slight issue in that AMANDA tries to do regular incremental backups. This makes sense if you're backing up normal data, but doesn't make sense for databases which should be immutable. Something obviously did change, because some of the incrementals were quite large, thereby wasting disk space. However, I figured out how to disable these, so backups should be more efficient in future. Review of Software Development Deadlines: * As NJC was not present this item was skipped. The prioritised list of software items is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/PrepareSoft Other Software: * ETWS reported that he had worked on some OSA updates. He also noted that Astropy v2.0 had been released and he proposed installing it on unas initially, so that ACD could test it out. ACTION: ETWS to install new version of Astropy on unas. Review of Survey Release Deadlines: * The progress towards forthcoming survey deadlines was skipped. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * VHS and VIKING P97 releases had been made * The VIKING ESO release had been patched up and re-submitted at ESO. Non-survey Data Release: * None to report. Astrogrid deployment, VO & Data Analysis services: * STV and DMR would be at ROE next week. Miscellaneous: * ACD noted that Nathalie is leaving the department. From 1 August claims should be sent to the Physics and Astronomy finance team in JCMB.