From acd@roe.ac.uk Mon Jun 5 16:03:42 2017 Date: Mon, 5 Jun 2017 15:03:18 +0000 From: Clive Davenhall To: Bob Mann , Andy Lawrence , "Read, Mike" , Eckhard Sutorius , Mark S. Holliman , Nicholas Cross , Nigel C. Hambly , Rob Blake , Ross Collins , Stelios Voutsinas , Dave Morris , Clive Davenhall Cc: "dante@astro.puc.cl" , James Dunlop , "j.p.emerson@qmul.ac.uk" , ktn keithnoddle.org , "Lorenzo.Rimoldini@unige.ch" , "mcioni@aip.de" , "matt.jarvis@astro.ox.ac.uk" , "mike@ast.cam.ac.uk" , "nigel.metcalfe@durham.ac.uk" , "norman@astro.gla.ac.uk" , "pwl@star.herts.ac.uk" , "rgm@ast.cam.ac.uk" , "Simon.Dye@nottingham.ac.uk" , "s.j.warren@ic.ac.uk" , "tom.shanks@durham.ac.uk" Subject: WFAU VDFS Science Archives progress meeting minutes, 12 May 2017 WFAU VDFS Science Archives progress meeting minutes, 12 May 2017 Present: RPB, RGM, NJC, MSH, ETWS, MAR, ACD Apologies: RSC, STV, DMR, KTN, AL, NCH DoNM: 10:00 am Friday 9 June 2017, Villas Meeting Room MAR in the chair (room is booked). Actions discharged this time: ----------------------------- ACTION RSC: check the schedule for availability of iDR5 data with Clare Worley and Anna Hourihane. Done. Gerry Gilmore had contacted WFAU with details of the release schedule for iDR5. The full set of iDR5 data (spectra and all astrophysical parameters, including the WG15 recommended parameters), are scheduled to be available by the 4/7/17. WFAU ingest is scheduled to start in mid-July and take up to 6 weeks, with completion no later than the end of August. Note that, in principle, ingest of the spectra can start sooner. ETWS had sent an inquiry to Jim Lewis about the availability and location of the spectra and was awaiting a reply. ACTION: MAR to reset UKIDSS saturation flag. Done. ACTION: All to test SQL server on Linux and update twiki. Discharged. MSH and RSC ICW NJC were running some tests. Actions carried forward from 26/5/2017 meeting: ---------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan DR13 is installed but not yet cross-mounted. However, we still need to decide which SDSS releases to keep so the present action should remain. ACTION: MAR to write some H-ATLAS documentation once H-ATLAS is available. Continues. ACTION: RPB to check with CASU whether they had any WFCAM observations made during the gap in the Edinburgh tapes. It was suggested we might like to check here first! Continues. Actions carried forward from the Backup Strategy Meeting held on 27/1/2017: -------------------------------------------------------------------------- ACTION: RPB to make a full tape backup of VVV promptly after it completes and henceforth make a copy annually. Continues. ACTION: NJC, In the context of the VVV, to check which curation steps can be paused without misadventure and hence compile a list of points at which backups may be made during the ingest. The list should be added to the twiki. Continues. NJC had a list of suitable pause points in his head. He would write it down in a twiki page. ACTION: NJC (was NJC, ETWS, MSH, MAR) to create a twiki page comprising a list of all the external datasets that WFAU hosts. The information included for each dataset should include: its name, size, where and when it originates and what WFAU uses it for. Continues. ETWS had created a page which is largely complete. NJC would check and add any remaining details. Once NJC is satisfied with the list the action would be complete but henceforth ETWS would periodically check the list. ACTION: RPB to create a twiki page documenting the backup procedure. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. Tuesday 28 February 2017 was subsequently agreed as a target date for completing these notes. A stub page for RPB to complete is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/BackupsProcedures Continues. ACTION: RPB to create a twiki page documenting the procedure to restore datasets from a backup. These notes should be usable by someone not routinely involved in the backups and, for example, include the physical location of the tapes. A stub page for RPB to complete is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/BackupsRecovery Continues. ACTION: ACD to obtain an estimate of the likely cost of a fire safe. Continues. ACD now had an estimate of the volume of tapes to be stored. The next step is to investigate suitable safes. ACTION: RGM to ask Steven Duffield to review the WFAU backup procedures. Continues. This action could not progress until RPB had written notes describing the backup and recovery procedures. ACTION: RPB to create a tape backup of the Gaia commissioning data at terra:/disk1/gaia/KAKAI_OLD and hand the tape to NCH for safe-keeping. Continues. ACTION (amended): RPB Generate a Gaia DR1 neighbours table for the most recent WSA release and the VVV DR4 release. Continues. ACTION (amended): MSH, RSC, NJC (was all) run further tests of SQL server on Linux. Continues. ACTION (amended): MSH (was all) set up a meeting to review tests of SQL server on Linux prior to the next telecon with John Hopkins. Continues. Specific points and new actions: -------------------------------- Project management: * RGM would write up the comments on last winter's Annual Reviews early in the week beginning 29/5/17 and circulate them to the reviewees, who should aim to respond promptly. WFCAM, VISTA and VST updates: * All the instruments were working as expected. CASU had written a paper on their calibration of VISTA images which had been accepted for publication. ACTION: MAR to arrange the date of the next CASU/WFAU telecon with Mike Irwin. Networking: * All the ROE and WFAU networks had been upgraded to 10 Gbit/sec and this upgrade had gone fairly smoothly. All the networks were now enabled for IPv6. WSA/VSA/OSA/GES Operations: * ETWS reported that: - after ingests of the latest OSA catalogue data (to Jan. 2017) had finished he had started checking and transferring the CASU list-driven photometry data. - updating the VVV objIDs for the WFAU-produced stacks and tiles was in progress. - new browser pages for VVV DR4, including the new PSF tables, had been created. - the missing WSA version 6 catalogue data had been checked and ingest was now in progress. * RPB noted that since the last meeting he had either been on leave or working on EUCLID. * MAR reported that he had worked on the QC for VIKING P97 and UKIDSS. Hardware and Systems: * MSH reported that a new NAS box had arrived but had not yet been installed. (There was also a new server for Gaia and new kit for EUCLID but these machines were out-of-scope for the meeting.) Finally, UoE Finance had been asked to place an order for another database node. * MSH noted he planned to decommission disks RAMSES3 and RAMSES5 in the near future but still needed confirmation that all the data of these disks could be deleted. ACTION: MAR to check that any data on RAMSES3 and RAMSES5 which is still required is copied elsewhere. Review of Software Development Deadlines: * ACD led a brief review of the software items currently being worked on. The prioritised list of software items is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/PrepareSoft Other Software: * NJC reported: - He had set up an infrastructure for dual image mode which allowed the metadata and catalogues associated with master images created by the VIDEO team to be ingested (the technique is to create a fake filter identifier for the master images). - Tom Shanks had requested the following enhancements to the OSA database: - inclusion of the revised calibration, 'untercal'. - list-driven photometry for all filters, - model magnitudes Only the last of these items might require significant software effort. ACTION: NJC to contact Robert Lupton to check if he already has code for computing model magnitudes which can be adapted. * ETWS noted: - Minor updates for the transfer of the list-driven photometry data, including the deprecation of earlier data. - An update to the calendar creation tool to include the addition of the PSF tables into VVV DR4. Review of Survey Release Deadlines: * Progress towards forthcoming survey deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * None to report. Non-survey Data Release: * None to report. Astrogrid deployment, VO & Data Analysis services: * MSH noted that a new TAP service for the OSA was currently publicly available for testing. It appeared to work but additional tests were needed. Miscellaneous: * MSH noted that he would be working from home on Mon. and Tue. 29-30/5/17. * NJC noted that Kai Polsterer (Heidelberg Institute for Theoretical Studies) would be visiting during the week beginning Mon. 29/5/17 to discuss the application of machine learning techniques to sky survey data. He would be giving a seminar on Wed. 31/5/17. Anyone interested in talking to him should contact NJC. ACD, 26/5/17.