From mar@roe.ac.uk Mon May 22 16:21:07 2017 Date: Mon, 22 May 2017 16:20:56 +0100 From: Mike Read To: VDFS Science Archive Team -- Clive Davenhall , dmr@roe.ac.uk, Eckhard Sutorius , Mark Holliman , Nicholas Cross , Nigel Hambly , Rob Blake , Ross Collins , Stelios Voutsinas Cc: CCs for VDFS -- Alastair Edge , Andrew Lawrence , Bob Mann , Dante Minniti , James Dunlop , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Maria-Rosa Cioni , matt.jarvis@astro.ox.ac.uk, Mike Irwin , Nigel Metcalfe , Norman Gray , Phil Lucas , Richard McMahon at IOA , Simon Dye , Stephen Warren , Tom Kerr , Tom Shanks Subject: WFAU VDFS Science Archives progress meeting minutes, 12 May 2017 ***** NB this week's meeting is in Crawford 2 ***** WFAU VDFS Science Archives progress meeting minutes, 12 May 2017 Present: NJC, MSH, MAR, RPB, RSC, ACD, ETWS Apologies: RGM, STV, DMR, KTN, AL, NCH DoNM: 10:00 am Friday 26 May 2017, Crawford Meeting Room 2 ACD in the chair (room is booked). Actions discharged this time: ----------------------------- ACTION: RPB to investigate the options for incremental backups of VVV while it is being created, either monthly and/or at convenient junctures between curation steps. Done, RPB had created a backup of the VVV load server DB and would continue to do so. ACTION: ETWS to estimate the disk space required for new files during the next the next couple of months and pass it to MSH. Done, new NAS boc ordered. ACTION NJC: Check and update the required neighbours tables for recent releases. Done ACTION NJC: Check with Maria-Rosa Cioni that the standard Gaia neighbours table would be suitable for her purposes. Done (yes there were). Actions carried forward from 24/3/2017 meeting: ---------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan DR13 is installed but not yet cross-mounted. However, we still need to decide which SDSS releases to keep so the present action should remain. ACTION: MAR to write some H-ATLAS documentation once H-ATLAS is available. Continues. ACTION: RPB to check with CASU whether they had any WFCAM observations made during the gap in the Edinburgh tapes. It was suggested we might like to check here first! Continues. ACTION RSC: check the schedule for availability of iDR5 data with Clare Worley and Anna Hourihane. Actions carried forward from the Backup Strategy Meeting held on 27/1/2017: -------------------------------------------------------------------------- ACTION: RPB to make a full tape backup of VVV promptly after it completes and henceforth make a copy annually. Continues. ACTION: NJC, In the context of the VVV, to check which curation steps can be paused without misadventure and hence compile a list of points at which backups may be made during the ingest. The list should be added to the twiki. Continues. NJC had a list of suitable pause points in his head. He would write it down in a twiki page. ACTION: NJC (was NJC, ETWS, MSH, MAR) to create a twiki page comprising a list of all the external datasets that WFAU hosts. The information included for each dataset should include: its name, size, where and when it originates and what WFAU uses it for. Continues. ETWS had created a page which is largely complete. NJC would check and add any remaining details. Once NJC is satisfied with the list the action would be complete but henceforth ETWS would periodically check the list. ACTION: RPB to create a twiki page documenting the backup procedure. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. Tuesday 28 February 2017 was subsequently agreed as a target date for completing these notes. Continues. ACTION: RPB to create a twiki page documenting the procedure to restore datasets from a backup. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. Continues. ACTION: ACD to obtain an estimate of the likely cost of a fire safe. Continues. ACD was awaiting an estimate of the volume of the tapes to be stored from RPB. ACTION: RGM to ask Steven Duffield to review the WFAU backup procedures. Continues. This action could not progress until RPB had written notes describing the backup and recovery procedures. ACTION: RPB to create a tape backup of the Gaia commissioning data at terra:/disk1/gaia/KAKAI_OLD and hand the tape to NCH for safe-keeping. Continues. ACTION RPB: Generate a neighbours table for the most recent WSA release. Continues. Specific points and new actions: -------------------------------- Project management: * Nothing for the minutes. WFCAM, VISTA and VST updates: * Nothing to report. Networking: * Site network upgrade is scheduled for the end of the month. MSH noted our switch was in place. WSA/VSA/OSA/GES Operations: * ETWS noted that the VSA was ingested up to December 2016. VVV objID updates were running. OSA data were in up to January 2017. The VVV release was currently in the deep tile curation stage. The VHS P97 release was on hold while NJC looked into the set-up of the field pointings. Hardware and Systems: * MSH reported that Nekau had gone crazy but was all fixed now. The new NAS box would arrive next week. Review of Software Development Deadlines: ACD presented a summary of the software items currently being worked on The prioritised list of software items is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/PrepareSoft Other Software: * Phil Lucas had asked about the saturation flagging/correction in the WSA. It was agreed to implement the same flagging as per the VSA (based on individual frames' attributes rather than a global flag). Correction of magnitudes would probably not be included in DR11 as it would take a while to run but could be rolled out for DRFinal. ACTION: MAR to reset UKIDSS saturation flag. Review of Survey Release Deadlines: * Progress towards forthcoming survey deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * The following surveys had been released: - VMCv20170411 - VPHASv2017022 Non-survey Data Release: * None to report. Astrogrid deployment, VO & Data Analysis services: * MSH noted that the TAP service was currently under test. Miscellaneous: SQL server on Linux was briefly discussed. MSH urged all to try it out and post comments/results on the twiki. Another telecon might then be arranged as and when we had more experience. ACTION: All to test SQL server on Linux and update twiki. NJC asked about VVV DB speed improvements in light of the recent email exchanges. After some discussion/confusion over the exact nature of the bottleneck it was suggested that speed ups could be achieved more readily by better logistical planning. eg maybe running the DB release outgests of monthly detection tables ahead of the actual final CU19 release, whilst other curation tasks were being performed. VVV load server ingests were currently up to date (save for objId updates). However previously agreed actions to test potential improvements to data import/export should still be pursued.