From mar@roe.ac.uk Thu Feb 8 16:53:00 2018 Date: Thu, 8 Feb 2018 16:52:36 +0000 From: Mike Read To: VDFS Science Archive Team -- Clive Davenhall , dmr@roe.ac.uk, Eckhard Sutorius , Mark Holliman , Nicholas Cross , Nigel Hambly , Rob Blake , Ross Collins , Roy Williams , Stelios Voutsinas Cc: CCs for VDFS -- Alastair Edge , Andrew Lawrence , Bob Mann , Dante Minniti , James Dunlop , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Maria-Rosa Cioni , matt.jarvis@astro.ox.ac.uk, Mike Irwin , Nigel Metcalfe , Norman Gray , Phil Lucas , Richard McMahon at IOA , Simon Dye , Stephen Warren , Tom Shanks Subject: WFAU data services meeting minutes, 7 February 2018 WFAU data services meeting minutes, 7 February 2018 Present: MAR, RDW, MSH, ETWS, NJC, RSC, RPB Apologies: DMR, STV, NCH, AL, ACD, RGM DoNM: 11:00 am Wednesday 14th February 2018, M1 Meeting Room MAR in the chair (room is booked). Actions discharged this time: ----------------------------- ACTION: RPB to add note to twiki to note filegroup issue and fix. Discharged ACTION: MSH to email RGM and Jim a summary of C2 shutdown concerns so that they could chase it up with the ATC. Discharged: The ATC have set-up an automated shutdown of half of their systems but were still looking into how to do the rest. MSH would keep track of their progress. ACTION: RPB to check out how file permissions operate under MSSQL on Linux with a view to replacing XP_CMDSHELL with ssh operations. Discharged, RPB reported that file permissions are sensibly maintained. NEW ACTION: RPB to set up generic user for ssh operations NEW ACTION: RSC to progress changes to code base to add ssh alternatives where XP_CMDSHELL used. Actions carried forward from 10/1/2018 meeting: ---------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan DR13 is installed but not yet cross-mounted. However, we still need to decide which SDSS releases to keep so the present action should remain. ACTION: MAR to write some H-ATLAS documentation once H-ATLAS is available. Continues. ACTION: NJC & MAR to add non-survey gap data to VMC and add area of VIKING data to SHARKS (NJC has some code to to this, otherwise MAR will do it manually) Continues, data are now CU4'ed so can proceed. ACTION: NJC to contact H-ATLAS PIs to see what, if any, requirements they still had for a WFAU hosted data-release. Continues. ACTION: RPB look at doing a fresh system install on pepy. Actions carried forward from the Backup Strategy Meeting held on 27/1/2017: --------------------------------------------------------------------------- ACTION: RPB to create a twiki page documenting the backup procedure. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. Tuesday 28 February 2017 was subsequently agreed as a target date for completing these notes. A stub page for RPB to complete is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/BackupsProcedures Continues. ACTION: RPB to create a twiki page documenting the procedure to restore datasets from a backup. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. A stub page for RPB to complete is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/BackupsRecovery Continues. ACTION: RGM to ask Steven Duffield to review the WFAU backup procedures. Continues. This action could not progress until RPB had written notes describing the backup and recovery procedures. Actions carried forward from the SQL Server on Linux Meeting 27/1/2017: --------------------------------------------------------------------------- ACTION: NJC to run the "testall" script as a basic sanity check. Check all file paths and change to Linux style. Should run in 30 minutes. This failed fairly quickly as XP_CMDSHELL not available under Linux. ACTION: RPB to run curation scripts on the WSA database to test re-doing a previous release. ACTION: MSH to check the crossmount/crossmatch/crossshare capabilities, connect two more servers, one Windows and one Linux. Candidate machines are ramses1 and ramses2. Continues. ACTION: MSH to set up a meeting to review tests of SQL server on Linux prior to the next telecon with John Hopkins. Continues, awaits results of testing. Specific points and new actions: -------------------------------- Project management: Nothing for the minutes. WFCAM, VISTA and VST updates: * Telecon with Durham and CASU next Thursday. WSA/VSA/OSA/GES Operations: * VSA - CU4'ed up to Dec 2017. VVVX ingests proceeding. * OSA - CU3 and CU2'ing latest data. 16 nights would need re-processing to fix nightly and APASS ZPs. * WSA - CU4'ed up to Oct 2017. LAS sourceXdetection fell over with buffer pool error. * ATLAS - release finally CU19'ing after problems with SDSS neighbour tables. * VVV - detection tables being copied to release server. However a spontaneous server reboot had interrupted the process. Hardware, Systems and Network: * MSH noted that seshat was on its last legs. The new NAS box still required an OS install but new USB sticks were needed as the recent batch had been failing. Thw MSDN subscription had expired but MSH had submitted a helpdesk ticket to get this renewed via the university. Ramses21 was up and running. ACTION: MAR to move OSA web-services off seshat. Status of Backups: * Nothing for the minutes Review of Software Release Deadlines: * The prioritised list of software items is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/PrepareSoft Other Software: * NJC reported work on CU6 and sourceXDetection speed-ups. He'd also been looking at implementing a pperrBit to flag bright deep tile sources that had no neighbours on their pawprints. This had arisen when spurious sources had been noticed in the VMC. The origin of these sources seemed to lie in the filtered tile image. Recent correspondence with MJI (CASU) suggested that creating the filtered image without reference to the conf might improve the catalogue output. NJC also noted that two of the new VISTA surveys might require new curation: VEILS was based on half tiles and including VIKING data into SHARKS would require re-mosaicing. * MAR reported that he'd been experimenting with creating HiPS images and cats. Review of survey deadlines: * Progress towards forthcoming survey deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * Nothing for the minutes Non-survey Data Release: * Nothing for the minutes. Astrogrid deployment, VO & Data Analysis services: * Nothing for the minues Miscellaneous: * Nothing for the minutes.