From mar@roe.ac.uk Mon Feb 20 17:39:31 2017 Date: Mon, 20 Feb 2017 17:39:21 +0000 From: Mike Read To: VDFS Science Archive Team -- Clive Davenhall , dmr@roe.ac.uk, Eckhard Sutorius , Mark Holliman , Nicholas Cross , Nigel Hambly , Rob Blake , Ross Collins , Stelios Voutsinas Cc: CCs for VDFS -- Alastair Edge , Andrew Lawrence , Bob Mann , Dante Minniti , James Dunlop , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Maria-Rosa Cioni , matt.jarvis@astro.ox.ac.uk, Mike Irwin , Nigel Metcalfe , Norman Gray , Phil Lucas , Richard McMahon at IOA , Simon Dye , Stephen Warren , Tom Kerr , Tom Shanks Subject: WFAU VDFS Science Archives progress meeting minutes, 17 February 2017 WFAU VDFS Science Archives progress meeting minutes, 17 February 2017 Present: ETWS, RPB, MSH, RGM, RSC, MAR Apologies: STV, DMR, KTN, AL, NCH, ACD, NJC DoNM: 10:00 am Friday 3 March 2017, Villas Meeting Room ACD in the chair (room is booked). Actions discharged this time: ----------------------------- ACTION: MSH to investigate the speed of disk44 Switching over the network cable had fixed the issue. ACTION: RPB, ACD to complete and record the Candlemas 2017 backup recovery test. Discharged. ACTION: RPB to create the Gaia DR1 cross-matches for existing databases. Discharged (done for ATLASDR1 & DR3). Do others as required. Actions discharged from the Backup Strategy Meeting held on 27/1/2017: --------------------------------------------------------------------- None Actions carried forward from 3/2/2017 meeting: ---------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan DR13 is installed but not yet cross-mounted. However, we still need to decide which SDSS releases to keep so the present action should remain. ACTION: MAR to write some H-ATLAS documentation once H-ATLAS is available. Continues. ACTION: RPB to check with CASU whether they had any WFCAM observations made during the gap in the Edinburgh tapes. It was suggested we might like to check here first! Continues. ACTION: MSH to set up a meeting with the SDSS team to discuss SQL Server on Linux. Continues. ACTION: RPB to rerun the non-survey databases which failed previously but have now been fixed. Actions carried forward from the Backup Strategy Meeting held on 27/1/2017: -------------------------------------------------------------------------- ACTION: RPB to make a full tape backup of VVV promptly after it completes and henceforth make a copy annually. Continues. ACTION: RPB to investigate the options for incremental backups of VVV while it is being created, either monthly and/or at convenient junctures between curation steps. Continues. ACTION: NJC, In the context of the VVV, to check which curation steps can be paused without misadventure and hence compile a list of points at which backups may be made during the ingest. The list should be added to the twiki. ACTION: NJC, ETWS, MSH, MAR to create a twiki page comprising a list of all the external datasets that WFAU hosts. The information included for each dataset should include: its name, size, where and when it originates and what WFAU uses it for. Continues. ETWS had created a page and added all the databases he is aware of, though the details for some of the them are incomplete. NJC, MSH and MAR should check and add any further databases and details. ACTION: RPB to configure the AMANDA backup system e-mail a copy of the log file that it generates during each run to ACD. Continues. ACTION: RPB to create a twiki page documenting the backup procedure. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. Tuesday 28 February 2017 was subsequently agreed as a target date for completing these notes. Continues. ACTION: RPB to create a twiki page documenting the procedure to restore datasets from a backup. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. Continues. ACTION: RPB to check if the AMANDA backup system has an option to generate a checksum (or similar) for the datasets that it backs up. Continues. ACTION: ACD to obtain an estimate of the likely cost of a fire safe. Continues. ACTION: RGM to ask Steven Duffield to review the WFAU backup procedures. Continues. This action could not progress until RPB had written notes describing the backup and recovery procedures. ACTION: RPB to create a tape backup of the Gaia commissioning data at terra:/disk1/gaia/KAKAI_OLD and hand the tape to NCH for safe-keeping. Continues. Specific points and new actions: -------------------------------- Project management: * RGM noted that he would complete the write-ups of the recent annual reviews soon. WFCAM, VISTA and VST updates: * WFCAM is off the telescope for a month or so. Networking: * Nothing to report; the networks were operating nominally. WSA/VSA/OSA/GES Operations: * ETWS reported: -- The Aug 2016 VISTA data has been transferred and ingested completely (CU1-4). -- The Oct, Nov, Dec 2016, Jan, Feb 2017 WFCAM data has been transferred (CU1). Metdata ingests are in progress. -- The Jun, Jul 2016 VVVX catalogue data has been ingested (CU4); Aug is in progress. -- VPHAS reseaming has finished and neighbour tables are in progress. -- Created new Browser Pages for WSA and OSA. * RPB noted that the VVV had finished creating deep stacks but had hit problems during tiling Hardware and Systems: * MSH noted that the new hardware orders had been received by Eclipse. Disk42 had been copied over to the new NAS box and had mysteriously increased in size, he would investigate. One of the NAS boxes had partially come off it's rails and would need re-fitting. Ramses5 was reporting a dead disk but this server was being replaced soon. The root password had been changed on all machines. The web server, niuserre, seems to have cured itself of issues it suffered the previous weekend, this might have been related to disk40 filling up. Review of Software Development Deadlines: * The list of software deadlines was skipped over. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/PrepareSoft Other Software: * ETWS noted: -- Removed userkaf from curation server list in SystemConstants. -- More updates for VVVX processing and other minor fixes. * MAR noted that he'd work modded the UI code to allow UHS community access to the upcoming UHSDR1 release. He also run-off 3/4 million ATLAS cut-outs for a user. Review of Survey Release Deadlines: * Progress towards forthcoming survey deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * Nothing to report. Non-survey Data Release: * Combined project, W/SERV/9, had been created and a release was being attempted Astrogrid deployment, VO & Data Analysis services: * Nothing to report. Miscellaneous: * MSH, RSC & RPB would be down at a Euclid meeting in London at the end of next week.