From mar@roe.ac.uk Thu Aug 24 17:14:02 2017 Date: Thu, 24 Aug 2017 17:13:48 +0100 From: Mike Read To: VDFS Science Archive Team -- Clive Davenhall , dmr@roe.ac.uk, Eckhard Sutorius , Mark Holliman , Nicholas Cross , Nigel Hambly , Rob Blake , Ross Collins , Roy Williams , Stelios Voutsinas Cc: CCs for VDFS -- Alastair Edge , Andrew Lawrence , Bob Mann , Dante Minniti , James Dunlop , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Maria-Rosa Cioni , matt.jarvis@astro.ox.ac.uk, Mike Irwin , Nigel Metcalfe , Norman Gray , Phil Lucas , Richard McMahon at IOA , Simon Dye , Stephen Warren , Tom Kerr , Tom Shanks Subject: WFAU data services meeting minutes, 18 August 2017 WFAU data services meeting minutes, 18 August 2017 Present: RSC, ETWS, MAR, MSH, NJC Apologies: RGM, DMR, STV, NCH, KTN, AL, ACD, RDW, RPB DoNM: 10:00 am Friday 1st September 2017, Villas Meeting Room MAR in the chair (room is booked). Actions discharged this time: ----------------------------- ACTION: ACD to organise a meeting to agree and schedule a programme of tests of SQL server on Linux. Attendees should be at least MSH, RSC, NJC and RDW. Discharged, meeting had taken place. ACTION (amended): MSH, RSC, NJC (was all) run further tests of SQL server on Linux. Specific actions had been agreed at the meeting and have been included below. ACTION: NJC to investigate and fix the problem with the combined release of WFCAM non-survey data. Discharged, fixed CU13. Actions carried forward from 4/8/2017 meeting: ---------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan DR13 is installed but not yet cross-mounted. However, we still need to decide which SDSS releases to keep so the present action should remain. ACTION: MAR to write some H-ATLAS documentation once H-ATLAS is available. Continues. ACTION: MAR to arrange the date of the next CASU/WFAU telecon with Mike Irwin. Continues. ACTION (amended): MAR, RPB to check that any data on RAMSES3 and RAMSES5 which is still required is copied elsewhere. Continues. Still a few DBs need to be copied/replicated. This item needs to be completed promptly. Actions carried forward from the Backup Strategy Meeting held on 27/1/2017: -------------------------------------------------------------------------- ACTION: RPB to make a full tape backup of VVV promptly after it completes and henceforth make a copy annually. Continues. ACTION: NJC, In the context of the VVV, to check which curation steps can be paused without misadventure and hence compile a list of points at which backups may be made during the ingest. The list should be added to the twiki. Continues. NJC had a list of suitable pause points in his head. He would write it down in a twiki page. ACTION: RPB to create a twiki page documenting the backup procedure. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. Tuesday 28 February 2017 was subsequently agreed as a target date for completing these notes. A stub page for RPB to complete is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/BackupsProcedures Continues. ACTION: RPB to create a twiki page documenting the procedure to restore datasets from a backup. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. A stub page for RPB to complete is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/BackupsRecovery Continues. ACTION: RGM to ask Steven Duffield to review the WFAU backup procedures. Continues. This action could not progress until RPB had written notes describing the backup and recovery procedures. ACTION (amended): RPB, ETWS, NJC Generate a Gaia DR1 neighbours table for the most recent WSA release and the VVV DR4 release. Continues. RPB had generated a neighbours table for VVV DR4. ETWS had still to regenerate the schema browser and NJC had still to update the other documentation. RPB had still to generate neighbours tables for the WSA releases (UHS was done at release). ACTION (amended): MSH (was all) set up a meeting to review tests of SQL server on Linux prior to the next telecon with John Hopkins. Continues, awaits results of testing. Actions carried forward from the SQL Servere on Linux Meeting 27/1/2017: --------------------------------------------------------------------------- ACTION: MSH to ensure that standard SQL login usernames are working on the Linux SQLserver test machine ramses7. ACTION: NJC to run the "testall" script as a basic sanity check. Check all file paths and change to Linux style. Should run in 30 minutes. ACTION: RPB to ingest the WSA on ramses7. Again, check the scripts for file paths. ACTION: RPB to run curation scripts on the WSA database. ACTION: MSH to check with Dave/Stelios to verify that ramses1 and ramses2 are not used by them. ACTION: MSH to check the crossmount/crossmatch/crossshare capabilities, connect two more servers, one Windows and one Linux. Candidate machines are ramses1 and ramses2. ACTION: MAR to move the HTM functionality outside the database (i.e. user's cross-id facility), so that there is no need to run the HTM Csharp code. Specific points and new actions: -------------------------------- Project management: * The new security passes would start to come into operation next week. WFCAM, VISTA and VST updates: * All the instruments were working as expected. Networking: * All WFAU networks were working nominally. WSA/VSA/OSA/GES Operations: * ETWS reported that he had: - [with NJC] Added new entries in curation tables for the new VISTA VISIONS survey. - Ingested the VISTA version 1.5 catalogues for Jan to Mar 2017 data. ACD had handed over GES ops to RSC. Hardware and Systems: * Ramses 10 had suffered some dead disks. Userkaf had become unresponsive. Status of Backups: * RPB not present to give update. Review of Survey Release Deadlines: * The prioritised list of software items is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/PrepareSoft Other Software: * ETWS recounted that he had: - Updated the latest observation times for VISTA and WFCAM data. - Adjusted the illumination table settings for VISTA March data since this month has a big gap in observations and therefor was processed with February illuminations. NJC had been working on including VVV team catalogues. There was some discussion over adding ppErrBits updates as part of finalizing monthly ingests. It was agreed was a good thing to do but there was a slight concern over this leading to missed detection flagging if the metadata was ingested but not the detections themselves. ACTION: RSC/NJC to check the consistency of the ppErrBits flagging/updates so as to avoid detections not being flagged. Review of survey deadlines: * Progress towards forthcoming survey deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * None to report. Non-survey Data Release: * Nothing to report. Astrogrid deployment, VO & Data Analysis services: * Dave & Stelios would be around next week. Miscellaneous: * Nothing to report.