From mar@roe.ac.uk Thu Jan 26 15:56:43 2017 Date: Thu, 26 Jan 2017 15:56:28 +0000 From: Mike Read To: VDFS Science Archive Team -- Clive Davenhall , dmr@roe.ac.uk, Eckhard Sutorius , Mark Holliman , Nicholas Cross , Nigel Hambly , Rob Blake , Ross Collins , Stelios Voutsinas Cc: CCs for VDFS -- Alastair Edge , Andrew Lawrence , Bob Mann , Dante Minniti , James Dunlop , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Maria-Rosa Cioni , matt.jarvis@astro.ox.ac.uk, Mike Irwin , Nigel Metcalfe , Norman Gray , Phil Lucas , Richard McMahon at IOA , Simon Dye , Stephen Warren , Tom Kerr , Tom Shanks Subject: WFAU VDFS Science Archives progress meeting minutes, 20 January 2017 WFAU VDFS Science Archives progress meeting minutes, 20 January 2017 Present: ETWS, RSC, MSH, MAR, RGM, RPB, NJC, ACD Apologies: STV, DMR, KTN, AL, NCH DoNM: 10:00 am Friday 3rd February 2017, Villas Meeting Room ACD in the chair (room is booked). Actions discharged this time: ---------------------------- ACTION: RGM to complete and circulate the Agenda for the Quarterly Meeting. Done. ACTION: MAR to arrange the next regular CASU/WFAU operations meetings with Mike Irwin. Done, telecon held last week. ACTION: NJC to send details to RGM of suggestion of how we could handle VSA and ESO releases of UltraVISTA data. Done. ACTION: ETWS to ask for and re-transfer the missing SDSS databases. Done. All missing databases re-transferred apart from SegueDR6 which is no longer available. NEW ACTION: RPB to install and make available the re-transferred SDSS databases. Actions carried forward from 19/12/2016 meeting: --------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan DR13 is installed but not yet cross-mounted. However, we still need to decide which SDSS releases to keep so the present action should remain. ACTION: MAR to write some H-ATLAS documentation once H-ATLAS is available. Continues. ACTION: RPB to check the log files and inform NJC of the problems with the previously listed non-survey releases. Continues. ACTION: NJC to investigate and fix the problems with the previously listed non-survey releases. Continues. This action cannot progress until RPB checks the log files and passes on details of the problems (action above). ACTION: RPB to check with CASU whether they had any WFCAM observations made during the gap in the Edinburgh tapes. It was suggested we might like to check here first! Continues. ACTION: MSH to investigate the speed of disk44 Continues. MSH noted that the whole NAS is slow but shows no errors and is set-up the same as our other boxes. He would try switching over the network cable. ACTION: ACD to set-up a WIKI page on MS SQL Server on Linux and all to add their input about what and how it needs to be tested for our uses. Continues. ACTION: RGM and ACD to discuss the WFAU backup strategy once the current set of backups are complete. Specific points and new actions: -------------------------------- Project management: * RGM noted that most annual reviews had been done and the agreed notes would be sent out shortly. WFCAM, VISTA and VST updates: * Nothing to report. Networking: * MSH reported that the network upgrades to 10 Gb would take place April-June. * ETWS noted: - Transfers of the SDSS backup files via Globus were fast with speeds around 20MB/s and up to 40MB/s for the largest DB (RunsDB). WSA/VSA/OSA/GES Operations: * ETWS reported: - Transferred the SDSS backup files for: BestDR1, BestDR2, BestDR3, BestDR5, BestDR7, and Stripe82. Ani Thakar recommended the RunsDB containing imaging scans, some of which are not present in DR7, and incomplete stripes. - Transferred the latest released VISTA data (July 2016). CU3 and CU2 are in progress. Writing the multiframeIDs into the files is slow due to the slow disks on shebaka. - Started ingesting P97 catalogue data (April to June 2016) into the VSA. - Started the provenance update procedure for skys in the WSA. - Ingested the missing VPHAS catalogue data into the OSAP95 database. * RPB reported that the VVV deep-stacking had continued been running over the holidays. The VPHAS re-seaming had fallen over when it encountered missing detections in the OSAP95 database. It was not clear how much of the re-seaming ,and indeed source merging, would need re-doing. Hardware and Systems: * MSH reported that the new NAS box, nekau, with > 300TB was online. Ramses13 had been offline with a failed power distribution board. This had been fixed under maintenance. Ramses16 has issues which MSH would address during quiet times. SQL Server on Linux was installed and running on ramses7 with the SSA as a test database. Read-only cross-mounts might be an issue. ACTION: MSH/RGM arrange meeting for MS SQL on Linux. Review of Software Development Deadlines: * The list of software deadlines was briefly discussed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/PrepareSoft Other Software: It was thought that some log files had recently been over-written recently. As the cueventID should be unique it wasn't clear how this had happened but if it happened again we'd investigate further. Review of Survey Release Deadlines: * Progress towards forthcoming survey deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * UHSDR1 had been released and made available to the PI for final checks. Non-survey Data Release: * Nothing to report. Astrogrid deployment, VO & Data Analysis services: * Nothing to report. Miscellaneous: NJC had met with Phil Lucas and Mike Irwin in Cambridge last week to discuss VISTA/VVV re-processing. It was thought that the VVVX data would come with new processing (v1.5) and that in a few months time we would gradually be getting re-processed versions of all VISTA data (new tiles, maybe just header updates to paw-prints). NJC also noted that it was unlikely we'd be getting difference imaging products from the VVV team and that maybe the matched aperture pipeline could do a similar job.