From acd@roe.ac.uk Tue Mar 21 11:13:24 2017 Date: Tue, 21 Mar 2017 11:13:07 +0000 From: Clive Davenhall To: Bob Mann , Andy Lawrence , "Read, Mike" , Eckhard Sutorius , Mark S. Holliman , Nicholas Cross , Nigel C. Hambly , Rob Blake , Ross Collins , Stelios Voutsinas , Dave Morris , Clive Davenhall Cc: "dante@astro.puc.cl" , James Dunlop , "j.p.emerson@qmul.ac.uk" , ktn keithnoddle.org , "Lorenzo.Rimoldini@unige.ch" , "mcioni@aip.de" , "matt.jarvis@astro.ox.ac.uk" , "mike@ast.cam.ac.uk" , "nigel.metcalfe@durham.ac.uk" , "norman@astro.gla.ac.uk" , "pwl@star.herts.ac.uk" , "rgm@ast.cam.ac.uk" , "Simon.Dye@nottingham.ac.uk" , "s.j.warren@ic.ac.uk" , "tom.shanks@durham.ac.uk" Subject: WFAU VDFS Science Archives progress meeting minutes, 3 March 2017 WFAU VDFS Science Archives progress meeting minutes, 3 March 2017 Present: RGM, ETWS, NJC, MAR, MSH, RPB, RSC, ACD Apologies: STV, DMR, KTN, AL, NCH DoNM: 10:00 am Friday 17 March 2017, Villas Meeting Room MAR in the chair (room is booked). Actions discharged this time: ----------------------------- None Actions carried forward from 3/2/2017 meeting: ---------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan DR13 is installed but not yet cross-mounted. However, we still need to decide which SDSS releases to keep so the present action should remain. ACTION: MAR to write some H-ATLAS documentation once H-ATLAS is available. Continues. ACTION: RPB to check with CASU whether they had any WFCAM observations made during the gap in the Edinburgh tapes. It was suggested we might like to check here first! Continues. ACTION: MSH to set up a meeting with the SDSS team to discuss SQL Server on Linux. Continues. MSH had set up a doodle page to solicit a convenient date for the meeting but some invitees had yet to respond. ACTION: RPB to rerun the non-survey databases which failed previously but have now been fixed. Continues. Actions carried forward from the Backup Strategy Meeting held on 27/1/2017: -------------------------------------------------------------------------- ACTION: RPB to make a full tape backup of VVV promptly after it completes and henceforth make a copy annually. Continues. ACTION: RPB to investigate the options for incremental backups of VVV while it is being created, either monthly and/or at convenient junctures between curation steps. Continues. RPB had tried to make a complete backup but it failed and processing had restarted before he could identify the problem. Therefore the database was currently at risk. Another backup would be attempted once the tiles had been ingested. In the meantime copies of the FITS image files would be created on a second disk. ACTION: NJC, In the context of the VVV, to check which curation steps can be paused without misadventure and hence compile a list of points at which backups may be made during the ingest. The list should be added to the twiki. Continues. NJC had a list of suitable pause points in his head. He would write it down in a twiki page. ACTION: NJC, ETWS, MSH, MAR to create a twiki page comprising a list of all the external datasets that WFAU hosts. The information included for each dataset should include: its name, size, where and when it originates and what WFAU uses it for. Continues. ETWS had created a page and added all the databases he is aware of, though the details for some of the them are incomplete. NJC, MSH and MAR should check and add any further databases and details. ACTION: RPB to configure the AMANDA backup system to e-mail a copy of the log file that it generates during each run to ACD. Continues. RPB thought that he had done this but would check. ACTION: RPB to create a twiki page documenting the backup procedure. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. Tuesday 28 February 2017 was subsequently agreed as a target date for completing these notes. Continues. ACTION: RPB to create a twiki page documenting the procedure to restore datasets from a backup. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. Continues. ACTION: RPB to check if the AMANDA backup system has an option to generate a checksum (or similar) for the datasets that it backs up. Continues. ACTION: ACD to obtain an estimate of the likely cost of a fire safe. Continues. ACD was awaiting an estimate of the volume of the tapes to be stored from RPB. ACTION: RGM to ask Steven Duffield to review the WFAU backup procedures. Continues. This action could not progress until RPB had written notes describing the backup and recovery procedures. ACTION: RPB to create a tape backup of the Gaia commissioning data at terra:/disk1/gaia/KAKAI_OLD and hand the tape to NCH for safe-keeping. Continues. Specific points and new actions: -------------------------------- Project management: * RGM apologised that he had still to complete the write-ups of the recent annual reviews. He also noted that an advertisement for a new Project Manager would appear soon. This post was to replace ACD who would be retiring in the autumn. WFCAM, VISTA and VST updates: * WFCAM is still off the telescope. CASU were believed to be continuing to process WFCAM data. Networking: * Nothing to report; the networks were operating nominally. A great deal of data had been transferred recently, with transfer rates typically between 20 and 30MByte/sec. WSA/VSA/OSA/GES Operations: * ETWS reported: - WSA catalogue data were ingested up to February 2017. - VSA metadata were ingested up to September 2016 and catalogue data up to August 2016. The October 2016 data have been transferred. - VVVX data were ingested up to August 2016, but some files need to be reingested due to database errors. - OSA: VPHAS processing was complete. ACTION: ETWS to estimate the disk space required for new files during the next the next couple of months and pass it to MSH. * RPB noted that: - VVV stacks were being ingested, but the process crashed part way through reporting an unusual database message. The ingest needs to be restarted. - Vista tiles can now be created, which allowed a non survey for Nicloas Lodieu to be generated. * RSC reminded the meeting that Gaia neighbour tables should be created for new databases releases. ACTION NJC: Check and update the required neighbours tables for recent releases. ACTION RPB: Generate a neighbours table for the most recent WSA release. ACTION NJC: Check with Maria-Rosa Cioni that the standard Gaia neighbours table would be suitable for her purposes. Hardware and Systems: * MSH reported that there had been an issue with a (new-ish) NAS box that had caused it to erroneously report that it had no free space and which required a hard reboot to fix. A similar problem had occurred with a different NAS box a few weeks ago. He wondered if there was a rogue piece of software that was causing the problem and would investigate if the problem recurred. He also noted that he was chasing the new hardware order recently placed with Eclipse. Review of Software Development Deadlines: * ACD presented a summary of the software items currently being worked on, which he and NJC had recently reviewed. The prioritised list of software items is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/PrepareSoft ACTION: NJC to check that the latest browser for the VIKING survey, VIKINGZYSELJv20170124 is correct and that the corresponding schema is checked in to SVN. This browser is available on disk50. Other Software: * NJC noted that he had worked on numerous items. * ETWS reported making many minor changes, including: - updating the code to process VVVX data into the vvvDetection* tables, - fixing problems with two-monthly WSA illumination corrections which cross semester boundaries. Review of Survey Release Deadlines: * Progress towards forthcoming survey deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * VPHASv20170222 had been released. ACTION: MSH to generate coverage maps for this release. Non-survey Data Release: * Two non-survey releases had been made: - N097C0781 for Nicloas Lodieu, - WSERV9 for Serena Kim. Astrogrid deployment, VO & Data Analysis services: * Nothing to report. Miscellaneous: * MSH reported that he continued to test the Linux version of SQL Server. In particular, he was in active discussion with Microsoft about mounting databases readonly. * NJC noted that a VVV Science Team Workshop would be held at ROE during 3-5/7/17 and encouraged WFAU staff to attend. The main topic would be the VVVX extension to the survey. Details of the meeting are available at: http://www.oagonzalez.net/8th-vvv-science-meeting-edinburgh ACD, 3/3/17.