From mar@roe.ac.uk Fri Sep 1 18:59:26 2017 Date: Fri, 1 Sep 2017 18:59:14 +0100 From: Mike Read To: VDFS Science Archive Team -- Clive Davenhall , dmr@roe.ac.uk, Eckhard Sutorius , Mark Holliman , Nicholas Cross , Nigel Hambly , Rob Blake , Ross Collins , Roy Williams , Stelios Voutsinas Cc: CCs for VDFS -- Alastair Edge , Andrew Lawrence , Bob Mann , Dante Minniti , James Dunlop , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Maria-Rosa Cioni , matt.jarvis@astro.ox.ac.uk, Mike Irwin , Nigel Metcalfe , Norman Gray , Phil Lucas , Richard McMahon at IOA , Simon Dye , Stephen Warren , Tom Kerr , Tom Shanks Subject: WFAU data services meeting minutes, 1 September 2017 WFAU data services meeting minutes, 1 September 2017 Present: RPB, ETWS, MAR, MSH, NJC, RGM Apologies: DMR, STV, NCH, KTN, RSC, AL, ACD, RDW, RPB DoNM: 10:00 am Friday 15th September 2017, Villas Meeting Room MAR in the chair (room is booked). Actions discharged this time: ----------------------------- ACTION (amended): MAR, RPB to check that any data on RAMSES3 and RAMSES5 which is still required is copied elsewhere. Discharged. It is thought that all DBs were now copied. WFCAMpropriety still need to be published to ramses19 and ramses15. ACTION: RPB to make a full tape backup of VVV promptly after it completes and henceforth make a copy annually. Discharged ACTION: NJC, In the context of the VVV, to check which curation steps can be paused without misadventure and hence compile a list of points at which backups may be made during the ingest. The list should be added to the twiki. Discharged. ACTION: MSH to ensure that standard SQL login usernames are working on the Linux SQLserver test machine ramses7. Discharged. ACTION: RPB to ingest the WSA on ramses7. Again, check the scripts for file paths. Discharged. ACTION: MSH to check with Dave/Stelios to verify that ramses1 and ramses2 are not used by them. Discharged, ramses2 is needed but rames1 can be re-used. ACTION: MAR to move the HTM functionality outside the database (i.e. user's cross-id facility), so that there is no need to run the HTM Csharp code. Discharged and HTM C Charp does actually work on Linux. ACTION: RSC/NJC to check the consistency of the ppErrBits flagging/updates so as to avoid detections not being flagged. Discharged. Actions carried forward from 18/8/2017 meeting: ---------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan DR13 is installed but not yet cross-mounted. However, we still need to decide which SDSS releases to keep so the present action should remain. ACTION: MAR to write some H-ATLAS documentation once H-ATLAS is available. Continues. ACTION: MAR to arrange the date of the next CASU/WFAU telecon with Mike Irwin. Continues. Actions carried forward from the Backup Strategy Meeting held on 27/1/2017: -------------------------------------------------------------------------- ACTION: RPB to create a twiki page documenting the backup procedure. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. Tuesday 28 February 2017 was subsequently agreed as a target date for completing these notes. A stub page for RPB to complete is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/BackupsProcedures Continues. ACTION: RPB to create a twiki page documenting the procedure to restore datasets from a backup. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. A stub page for RPB to complete is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/BackupsRecovery Continues. ACTION: RGM to ask Steven Duffield to review the WFAU backup procedures. Continues. This action could not progress until RPB had written notes describing the backup and recovery procedures. ACTION (amended): RPB, ETWS, NJC Generate a Gaia DR1 neighbours table for the most recent WSA release and the VVV DR4 release. Continues. RPB had generated a neighbours table for VVV DR4. ETWS had still to regenerate the schema browser and NJC had still to update the other documentation. RPB had still to generate neighbours tables for the WSA releases (UHS was done at release). ACTION (amended): MSH (was all) set up a meeting to review tests of SQL server on Linux prior to the next telecon with John Hopkins. Continues, awaits results of testing. Actions carried forward from the SQL Server on Linux Meeting 27/1/2017: --------------------------------------------------------------------------- ACTION: NJC to run the "testall" script as a basic sanity check. Check all file paths and change to Linux style. Should run in 30 minutes. ACTION: RPB to run curation scripts on the WSA database. ACTION: MSH to check the crossmount/crossmatch/crossshare capabilities, connect two more servers, one Windows and one Linux. Candidate machines are ramses1 and ramses2. Specific points and new actions: -------------------------------- Project management: * All the instruments were working as expected. WFCAM, VISTA and VST updates: * Nothing for the minutes Networking: * All WFAU networks were working nominally. WSA/VSA/OSA/GES Operations: * UKIDSS LAS was under CU6. * VSA P98 data had all been ingested (apart from VVVX) ACTION: NJC to check if any of the new VISTA surveys warranted a P98 release. ACTION: MAR to update the release deadlines tiwki doc with P98 VSA releases. * VPHAS release still running. * VVV was having some constraints applied. * the ingest of the latest GES data was not working with the latest codebase and 3rd party libraries. It was thought that rolling back the software might be the quickest way forward. ACTION: ACD/ETWS to investigate rolling back the GES ingest code. Hardware and Systems: * Ramses 5 had suffered a dead disk, the plan was to retire it on or after Monday 11th September. Status of Backups: * RPB reported that he'd disabled incremental backups in AMANDA and was now working through initial full backups of everything. Deep stack products would also be backed up. Review of Software Release Deadlines: * The prioritised list of software items is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/PrepareSoft Other Software: ETWS and NJC had worked on the finalise catalogue code including ppErrBit flagging and tile-pawprints. NJC had also worked on the matched aperture pipeline, LAS/GCS fixes and VVV team catalogues. MAR had done a bulk GPS outgest for a user at USNO and would also be sending them VHS data when they decided on what attributes they wanted. Review of survey deadlines: * Progress towards forthcoming survey deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * Nothing to report. Non-survey Data Release: * WSERV10 was released. Astrogrid deployment, VO & Data Analysis services: Nothing for the minutes Miscellaneous: * MSH noted that he, RPB and MAR had access to C2 under the new card entry system.