From mar@roe.ac.uk Thu Feb 1 11:52:08 2018 Date: Thu, 1 Feb 2018 11:52:03 +0000 From: Mike Read To: VDFS Science Archive Team -- Clive Davenhall , dmr@roe.ac.uk, Eckhard Sutorius , Mark Holliman , Nicholas Cross , Nigel Hambly , Rob Blake , Ross Collins , Roy Williams , Stelios Voutsinas Cc: CCs for VDFS -- Alastair Edge , Andrew Lawrence , Bob Mann , Dante Minniti , James Dunlop , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Maria-Rosa Cioni , matt.jarvis@astro.ox.ac.uk, Mike Irwin , Nigel Metcalfe , Norman Gray , Phil Lucas , Richard McMahon at IOA , Simon Dye , Stephen Warren , Tom Shanks Subject: WFAU data services meeting minutes, 24 January 2018 WFAU data services meeting minutes, 24 January 2018 Present: MAR, RDW, RGM, MSH, ETWS, NJC, RSC, RPB Apologies: DMR, STV, NCH, AL, ACD DoNM: 11:00 am Wednesday 7th February 2018, Villas Meeting Room MAR in the chair (room is booked). Actions discharged this time: ----------------------------- ACTION: MSH to cost full SSD VVV server Discharged, around 90K which is likely beyond budget for now. Actions carried forward from 10/1/2018 meeting: ---------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan DR13 is installed but not yet cross-mounted. However, we still need to decide which SDSS releases to keep so the present action should remain. ACTION: MAR to write some H-ATLAS documentation once H-ATLAS is available. Continues. ACTION: RPB to add note to twiki to note filegroup issue and fix. Continues ACTION: MSH to email RGM and Jim a summary of C2 shutdown concerns so that they could chase it up with the ATC. Continues, there is an IT services meeting next week. ACTION: NJC & MAR to add non-survey gap data to VMC and add area of VIKING data to SHARKS (NJC has some code to to this, otherwise MAR will do it manually) Continues, waiting for data to be CU4'ed. ACTION: NJC to contact H-ATLAS PIs to see what, if any, requirements they still had for a WFAU hosted data-release. Continues. ACTION: RPB look at doing a fresh system install on pepy. Actions carried forward from the Backup Strategy Meeting held on 27/1/2017: --------------------------------------------------------------------------- ACTION: RPB to create a twiki page documenting the backup procedure. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. Tuesday 28 February 2017 was subsequently agreed as a target date for completing these notes. A stub page for RPB to complete is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/BackupsProcedures Continues. ACTION: RPB to create a twiki page documenting the procedure to restore datasets from a backup. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. A stub page for RPB to complete is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/BackupsRecovery Continues. ACTION: RGM to ask Steven Duffield to review the WFAU backup procedures. Continues. This action could not progress until RPB had written notes describing the backup and recovery procedures. Actions carried forward from the SQL Server on Linux Meeting 27/1/2017: --------------------------------------------------------------------------- ACTION: NJC to run the "testall" script as a basic sanity check. Check all file paths and change to Linux style. Should run in 30 minutes. This failed fairly quickly as XP_CMDSHELL not available under Linux. ACTION: RPB to check out how file permissions operate under MSSQL on Linux with a view to replacing XP_CMDSHELL with ssh operations. ACTION: RPB to run curation scripts on the WSA database to test re-doing a previous release. ACTION: MSH to check the crossmount/crossmatch/crossshare capabilities, connect two more servers, one Windows and one Linux. Candidate machines are ramses1 and ramses2. Continues. ACTION: MSH to set up a meeting to review tests of SQL server on Linux prior to the next telecon with John Hopkins. Continues, awaits results of testing. Specific points and new actions: -------------------------------- Project management: * RGM would be sending in the list of requested desks for the new building so that the ATC could order them. It was hoped that a tour of the new WFAU offices could be arranged shortly so that the layout of offices could be looked at and additional furniture/fixtures etc could be ordered via the IfA ahead of the move. * RDW would looking at purchasing an account for the WFAU group for the GOTO Meeting conferencing system. * RDW asked about CAUA V1.5 VISTA processing. It was thought that 2016 data was being checked over but as yet no backdated data had been released to us. WFCAM, VISTA and VST updates: * Nothing for the minutes. WSA/VSA/OSA/GES Operations: * VSA - being CU4'ed. VINROUGE had required setting up. * OSA - data up to 09/17 being CU3'ed. * WSA - CU4 seg faulting issues had been fixed. LAS/GCS source merging and deeps had gone through another iteration. * ATLAS - re-seaming had finished (much quicker since the re-start). * VVV - during the creation of sourceXdetection it was discovered that there were duplicate objIDs in the detection tables. A fix was being applied, with one more monthly table yet to be updated. Hardware, Systems and Network: * MSH noted that a few servers still required OS upgrades. The new DB node still required setting up. * RPB reported that occasionally khaba seemed to suffer network glitches. MSH had seem similar issues but it was proving difficult to trace the root cause. Status of Backups: * Proceeding well and finishing off last few DBSs before going on to WFAU flat file products. Review of Software Release Deadlines: * The prioritised list of software items is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/PrepareSoft Other Software: * NJC reported work on MAP and mosaicing. CU7 and CU13 improvements. Branch code changes had been merged back into trunk. * MAR reported that a sample of UKIDSS source table data and details on our S/G attributes had been supplied to Belfast. RDW noted that we might now run the context classifier here rather than send them all the data. Review of survey deadlines: * Progress towards forthcoming survey deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * VIKINGv20180115 had been released. Non-survey Data Release: * Nothing for the minutes. Astrogrid deployment, VO & Data Analysis services: * RDW, STV and MAR had met to discuss generating/hosting HiPS. Storage wise around 50-100Tb would be needed for 0.4 arcsec pixels with another 100Tb or so of scratch space for tmp files. Miscellaneous: Nothing for the minutes.