From acd@roe.ac.uk Wed Feb 15 10:53:56 2017 Date: Wed, 15 Feb 2017 10:53:47 +0000 From: Clive Davenhall To: Bob Mann , Andy Lawrence , "Read, Mike" , Eckhard Sutorius , Mark S. Holliman , Nicholas Cross , Nigel C. Hambly , Rob Blake , Ross Collins , Stelios Voutsinas , Dave Morris , Clive Davenhall Cc: "dante@astro.puc.cl" , James Dunlop , "j.p.emerson@qmul.ac.uk" , ktn keithnoddle.org , "Lorenzo.Rimoldini@unige.ch" , "mcioni@aip.de" , "matt.jarvis@astro.ox.ac.uk" , "mike@ast.cam.ac.uk" , "nigel.metcalfe@durham.ac.uk" , "norman@astro.gla.ac.uk" , "pwl@star.herts.ac.uk" , "rgm@ast.cam.ac.uk" , "Simon.Dye@nottingham.ac.uk" , "s.j.warren@ic.ac.uk" , "tom.shanks@durham.ac.uk" Subject: WFAU VDFS Science Archives progress meeting minutes, 3 February 2017 WFAU VDFS Science Archives progress meeting minutes, 3 February 2017 Present: ETWS, RPB, MSH, RGM, RSC, NJC, ACD Apologies: MAR, STV, DMR, KTN, AL, NCH DoNM: 10:00 am Friday 17 February 2017, Villas Meeting Room MAR in the chair (room is booked). Actions discharged this time: ---------------------------- ACTION: RPB to check the log files and inform NJC of the problems with the previously listed non-survey releases. Done. ACTION: NJC to investigate and fix the problems with the previously listed non-survey releases. Done. NJC had fixed the problems and RPB now needs to re-run the releases. ACTION: ACD to set-up a twiki page on MS SQL Server on Linux and all to add their input about what and how it needs to be tested for our uses. Done. A page had been set and some material had been added. Additional notes would be added as required. ACTION: RGM and ACD to discuss the WFAU backup strategy once the current set of backups are complete. Done. A meeting had been held and the actions resulting from that meeting are included below. Actions discharged from the Backup Strategy Meeting held on 27/1/2017: --------------------------------------------------------------------- ACTION: RPB to perform quarterly tests to restore a randomly chosen database from backup. Done. The first such test was in progress. Actions carried forward from 20/1/2017 meeting: ---------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan DR13 is installed but not yet cross-mounted. However, we still need to decide which SDSS releases to keep so the present action should remain. ACTION: MAR to write some H-ATLAS documentation once H-ATLAS is available. Continues. ACTION: RPB to check with CASU whether they had any WFCAM observations made during the gap in the Edinburgh tapes. It was suggested we might like to check here first! Continues. ACTION: MSH to investigate the speed of disk44 Continues. MSH noted that the whole NAS is slow but shows no errors and is set-up the same as our other boxes. He would try switching over the network cable. ACTION: MSH, RGM to set up a meeting with the SDSS team to discuss SQL Server on Linux. Continues. Actions carried forward from the Backup Strategy Meeting held on 27/1/2017: -------------------------------------------------------------------------- ACTION: RPB to make a full tape backup of VVV promptly after it completes and henceforth make a copy annually. Continues. ACTION: RPB to investigate the options for incremental backups of VVV while it is being created, either monthly and/or at convenient junctures between curation steps. Continues. ACTION: NJC, In the context of the VVV, to check which curation steps can be paused without misadventure and hence compile a list of points at which backups may be made during the ingest. The list should be added to the twiki. ACTION: NJC, ETWS, MSH, MAR to create a twiki page comprising a list of all the external datasets that WFAU hosts. The information included for each dataset should include: its name, size, where and when it orginates and what WFAU uses it for. Continues. ETWS had created a page and added all the databases he is aware of, though the details for some of the them are incomplete. NJC, MSH and MAR should check and add any further databases and details. ACTION: RPB to configure the AMANDA backup system e-mail a copy of the log file that it generates during each run to ACD. Continues. ACTION: RPB to create a twiki page documenting the backup procedure. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. Tuesday 28 February 2017 was subsequently agreed as a target date for completing these notes. Continues. ACTION: RPB to create a twiki page documenting the procedure to restore datasets from a backup. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. Continues. ACTION: RPB to check if the AMANDA backup system has an option to generate a checksum (or similar) for the datasets that it backs up. Continues. ACTION: ACD to obtain an estimate of the likely cost of a fire safe. Continues. ACTION: RGM to ask Sean McGeever to review the WFAU backup procedures. Continues. This action could not progress until RPB had written notes describing the backup and recovery procedures. ACTION: RPB to create a tape backup of the Gaia commissioning data at terra:/disk1/gaia/KAKAI_OLD and hand the tape to NCH for safe-keeping. Continues. Specific points and new actions: -------------------------------- Project management: * RGM noted that he would be completing and distributing the write-ups of the recent annual reviews over the next couple of weeks. WFCAM, VISTA and VST updates: * Nothing to report. Networking: * Nothing to report; the networks were operating nominally. WSA/VSA/OSA/GES Operations: * ETWS reported that he had: - run checks to ensure that WSA database entries are correct in the case where the source files no longer exist, - updated the WSA provenance for sky frames, - ingested metadata (CU3) for July 2016 into the VSA, - ingested catalogue data (CU4) for April to July 2016 into the VSA, - ingested the missing VPHAS data into the OSAP95 database and restarted the VPHAS release. * RPB noted that the VVV was still creating deep stacks; this operation is thought likely to complete in mid-February. NJC added that before starting to stack the tiles he would check the tile zeropoints, a few of which appeared to be anomalous. * The first of the new quarterly tests of the backup recovery was in progress (the Candlemas test). RPB had successfully recovered the VIKINGDR4 database to disk, but had yet to check that the retrieved version was complete and incorrupt. ACTION: RPB, ACD to complete and record the Candlemas 2017 backup recovery test. ACTION: RPB to create the Gaia DR1 cross-matches for existing databases. Hardware and Systems: * MSH noted that several new orders for hardware had been raised, but Eclipse had not yet received the purchase orders. He planned to chase progress with Finance later on the day of the meeting. The new NAS box had suffered a transient glitch but this had been solved by a reboot. Review of Software Development Deadlines: * The list of software deadlines was briefly discussed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/PrepareSoft Other Software: * NJC reported that he had fixed the problems with VPHAS re-seaming. He had also added an option to CU7 retain the order of framesets and made a few other minor changes. In addition he had done some work on the ATLAS ESO release. * ETWS noted that he had made some minor improvements including enabling objID updates for non-surveys only to be able to prioritise non-surveys. * ETWS and NJC had fixed the dual image mode for VIDEO P95/96. Review of Survey Release Deadlines: * Progress towards forthcoming survey deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * Nothing to report. Non-survey Data Release: * Nothing to report. ACTION: RPB to rerun the non-survey databases which failed previously but have now been fixed. Astrogrid deployment, VO & Data Analysis services: * Nothing to report. Miscellaneous: * ACD reminded the meeting of the EWASS 2017 conference to be held in Prague in June which would have a couple of sessions relevant to WFAU work. NJC noted that there would be a VVVX meeting in Edinburgh immediately after the EWASS meeting (the timing was deliberate and aimed to minimise inconvenience for attendees from other continents). ACD, 3/2/17.