From mar@roe.ac.uk Thu Dec 21 19:06:39 2017 Date: Thu, 21 Dec 2017 19:06:29 +0000 From: Mike Read To: VDFS Science Archive Team -- Clive Davenhall , dmr@roe.ac.uk, Eckhard Sutorius , Mark Holliman , Nicholas Cross , Nigel Hambly , Rob Blake , Ross Collins , Roy Williams , Stelios Voutsinas Cc: CCs for VDFS -- Alastair Edge , Andrew Lawrence , Bob Mann , Dante Minniti , James Dunlop , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Maria-Rosa Cioni , matt.jarvis@astro.ox.ac.uk, Mike Irwin , Nigel Metcalfe , Norman Gray , Phil Lucas , Richard McMahon at IOA , Simon Dye , Stephen Warren , Tom Kerr , Tom Shanks Subject: WFAU data services meeting minutes, 13 December 2017 WFAU data services meeting minutes, 13 December 2017 Present: MAR, RPB, RDW, RGM, MSH, ETWS, NJC Apologies: DMR, STV, NCH, AL, ACD, RPB, RSC DoNM: 11:00 am Wednesday 10th January 2018, Villas Meeting Room MAR in the chair (room is booked). Actions discharged this time: ----------------------------- ACTION: ALL to consider advantages/disadvantages of combining or keeping VVV/VVVX separate. Continues, NJC had had some discussions with the PIs, but no conclusion yet. Our preference is to keep separate with maybe one final combined release. NJC to discuss further with Phil Discharged, NJC had been in contact with the VVVX PIs. The initial release would be VVX only. Actions carried forward from 29/11/2017 meeting: ---------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan DR13 is installed but not yet cross-mounted. However, we still need to decide which SDSS releases to keep so the present action should remain. ACTION: MAR to write some H-ATLAS documentation once H-ATLAS is available. Continues. ACTION: RPB to add note to twiki to note filegroup issue and fix. Continues Actions carried forward from the Backup Strategy Meeting held on 27/1/2017: -------------------------------------------------------------------------- ACTION: RPB to create a twiki page documenting the backup procedure. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. Tuesday 28 February 2017 was subsequently agreed as a target date for completing these notes. A stub page for RPB to complete is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/BackupsProcedures Continues. ACTION: RPB to create a twiki page documenting the procedure to restore datasets from a backup. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. A stub page for RPB to complete is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/BackupsRecovery Continues. ACTION: RGM to ask Steven Duffield to review the WFAU backup procedures. Continues. This action could not progress until RPB had written notes describing the backup and recovery procedures. ACTION (amended): MSH (was all) set up a meeting to review tests of SQL server on Linux prior to the next telecon with John Hopkins. Continues, awaits results of testing. Actions carried forward from the SQL Server on Linux Meeting 27/1/2017: --------------------------------------------------------------------------- ACTION: NJC to run the "testall" script as a basic sanity check. Check all file paths and change to Linux style. Should run in 30 minutes. This failed fairly quickly as XP_CMDSHELL not available under Linux. ACTION: RPB to check out how file permissions operate under MSSQL on Linux with a view to replacing XP_CMDSHELL with ssh operations. ACTION: RPB to run curation scripts on the WSA database to test re-doing a previous release. ACTION: MSH to check the crossmount/crossmatch/crossshare capabilities, connect two more servers, one Windows and one Linux. Candidate machines are ramses1 and ramses2. Continues. Specific points and new actions: -------------------------------- Project management: * RGM noted that the annual reviews had started and most would be done before Christmas. The Higgs building would be handed over in January but office moves would not take place until around Easter WFCAM, VISTA and VST updates: * There had been a telecon with CASU on the 6th December. No firm timescale for likely GAIA DR2 astrometric re-processing of data. Following a warm-up, detector 4 on WFCAM had a new dead region though at least it's in the dither region. UHS-K is nearly 30% observed, the team would be visiting the UK in March. WSA/VSA/OSA/GES Operations: * The recent batch of WSA data had been CU4'ed * UHS-K detections were being ingested, * VVV source merging/re-seaming. Hardware, Systems and Network: * MSH noted that a new NAS box had been ordered. The two oldest NAS boxes were playing up and data were being copied over. * Test shutdown process early in 2018, probably retire ramses3-6 at that time. ACTION: MSH to contact Ewan to see if there had been any progress on a process for automatic shutdown of ATC hardware in C2 in the event of a power outage. Status of Backups: * RPB reported no issues with some tidying up still required. Review of Software Release Deadlines: * The prioritised list of software items is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/PrepareSoft Other Software: * NJC had fixed and issue with CU7 in append mode. Unfortunately this bug had a shown up in the ongoing ATLAS release and this release would need to be re-started. * ETWS had updated 3rd party software on various servers, the testAll script was currently failing on Khaba. * MAR had updated the VSA UI to include the new surveys. Test were being carried to see if the HTMIDs generated here were the same as at Belfast. Review of survey deadlines: * Progress towards forthcoming survey deadlines was skipped. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * VHSv20171207 had been released. The VHS P96 release would be made public. Non-survey Data Release: * RPB was working through a list. Astrogrid deployment, VO & Data Analysis services: * firethorn.py is almost ready for use in Jupyterhub. Miscellaneous: Nothing for the minutes.