From mar@roe.ac.uk Thu Jan 18 16:37:08 2018 Date: Thu, 18 Jan 2018 16:36:58 +0000 From: Mike Read To: VDFS Science Archive Team -- Clive Davenhall , dmr@roe.ac.uk, Eckhard Sutorius , Mark Holliman , Nicholas Cross , Nigel Hambly , Rob Blake , Ross Collins , Roy Williams , Stelios Voutsinas Cc: CCs for VDFS -- Alastair Edge , Andrew Lawrence , Bob Mann , Dante Minniti , James Dunlop , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Maria-Rosa Cioni , matt.jarvis@astro.ox.ac.uk, Mike Irwin , Nigel Metcalfe , Norman Gray , Phil Lucas , Richard McMahon at IOA , Simon Dye , Stephen Warren , Tom Shanks Subject: WFAU data services meeting minutes, 10 January 2018 WFAU data services meeting minutes, 10 January 2018 Present: MAR, RDW, RGM, MSH, ETWS, NJC, RSC, RPB Apologies: DMR, STV, NCH, AL, ACD DoNM: 11:00 am Wednesday 24th January 2018, Villas Meeting Room MAR in the chair (room is booked). Actions discharged this time: ----------------------------- ACTION: MSH to contact Ewan to see if there had been any progress on a process for automatic shutdown of ATC hardware in C2 in the event of a power outage. Discharged: No progress had been made on the ATC side. NEW ACTION: MSH to email RGM and Jim a summary of C2 shutdown concerns so that they could chase it up with the ATC. Actions carried forward from 13/12/2017 meeting: ---------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan DR13 is installed but not yet cross-mounted. However, we still need to decide which SDSS releases to keep so the present action should remain. ACTION: MAR to write some H-ATLAS documentation once H-ATLAS is available. Continues. ACTION: RPB to add note to twiki to note filegroup issue and fix. Continues Actions carried forward from the Backup Strategy Meeting held on 27/1/2017: -------------------------------------------------------------------------- ACTION: RPB to create a twiki page documenting the backup procedure. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. Tuesday 28 February 2017 was subsequently agreed as a target date for completing these notes. A stub page for RPB to complete is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/BackupsProcedures Continues. ACTION: RPB to create a twiki page documenting the procedure to restore datasets from a backup. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. A stub page for RPB to complete is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/BackupsRecovery Continues. ACTION: RGM to ask Steven Duffield to review the WFAU backup procedures. Continues. This action could not progress until RPB had written notes describing the backup and recovery procedures. ACTION (amended): MSH (was all) set up a meeting to review tests of SQL server on Linux prior to the next telecon with John Hopkins. Continues, awaits results of testing. Actions carried forward from the SQL Server on Linux Meeting 27/1/2017: --------------------------------------------------------------------------- ACTION: NJC to run the "testall" script as a basic sanity check. Check all file paths and change to Linux style. Should run in 30 minutes. This failed fairly quickly as XP_CMDSHELL not available under Linux. ACTION: RPB to check out how file permissions operate under MSSQL on Linux with a view to replacing XP_CMDSHELL with ssh operations. ACTION: RPB to run curation scripts on the WSA database to test re-doing a previous release. ACTION: MSH to check the crossmount/crossmatch/crossshare capabilities, connect two more servers, one Windows and one Linux. Candidate machines are ramses1 and ramses2. Continues. Specific points and new actions: -------------------------------- Project management: * RGM noted that most of the annual reviews had been completed and would be written up in the next week or so. We would be hosting a FIRE-OPAL camera at ROE as part of the new LM collaboration. RGM, RDW and MSH would shortly have a meeting to go through the budget for upcoming computing HW purchases. WFCAM, VISTA and VST updates: * AL has sent an email round notifying us of a visit by folk from USNO on 22nd March. WSA/VSA/OSA/GES Operations: * VSA - data transferred up to Nov 2017 (CU3'ed Oct 2017 * OSA - data transferred up to Sep 2017. * VIKING P98 being CU19'ed. * WSA - CU4 had been seg faulting. Still looking at issues with LAS and GCS mergelogs. * VVV detections copying over to release server in parallel with other curation. * ATLAS - after a series of crashes, the next ATLAS release was back running. ACTION: MSH & MAR to add non-survey gap data to VMC and add area of VIKING data to SHARKS (NJC has some code to to this, otherwise MAR will do it manually) Hardware, Systems and Network: * MSH noted that all machines would need updated/rebooted to apply the meltdown/spectre fixes, machine performance would likely be affected. Another NAS box was being retired next week. HW orders were needing chasing as accounts were now causing 6 week delays. ROE had a new firewall. ACTION: MSH to cost full SSD VVV server Status of Backups: * Proceeding well and finishing off list of databases following the Christmas hiatus. Possible issue with unas. Review of Software Release Deadlines: * The prioritised list of software items is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/PrepareSoft ACTION: NJC to contact H-ATLAS PIs to see what, if any, requirements they still had for a WFAU hosted data-release. Other Software: * ETWS had been installing 3rd party software on curation servers, proceeding well apart from pepy, which has a history of being difficult. ACTION: RPB look at doing a fresh system install on pepy. * NJC had been working on the issues with creating LAS/GCS mergelogs and re-seaming. * MAR had helped a user obtain 500,000 UKIDSS cut-outs. * Our HTMIDs were the same as Belfast's, so could proceed with supplying them with UKIDSS data. Review of survey deadlines: * Progress towards forthcoming survey deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * VHSDR5 (formerly VHSv20161007) had been made public. Non-survey Data Release: * MAR asked RPB if he could work through the supplied list of projects. Astrogrid deployment, VO & Data Analysis services: * A meeting would be arranged when STV was around next week to discuss generating and serving HIPS images for various surveys. Miscellaneous: Nothing for the minutes.