From acd@roe.ac.uk Mon May 23 18:02:17 2016 Date: Mon, 23 May 2016 17:01:46 +0000 From: Clive Davenhall To: Bob Mann , Andy Lawrence , "Read, Mike" , Eckhard Sutorius , Mark S. Holliman , Nicholas Cross , Nigel C. Hambly , Rob Blake , Ross Collins , Stelios Voutsinas , Dave Morris , Clive Davenhall Cc: "dante@astro.puc.cl" , James Dunlop , "j.p.emerson@qmul.ac.uk" , ktn keithnoddle.org , "Lorenzo.Rimoldini@unige.ch" , "mcioni@aip.de" , "matt.jarvis@astro.ox.ac.uk" , "mike@ast.cam.ac.uk" , "nigel.metcalfe@durham.ac.uk" , "norman@astro.gla.ac.uk" , "pwl@star.herts.ac.uk" , "rgm@ast.cam.ac.uk" , "Simon.Dye@nottingham.ac.uk" , "s.j.warren@ic.ac.uk" , "tom.shanks@durham.ac.uk" Subject: WFAU VDFS Science Archives progress meeting minutes, 13 May 2016 WFAU VDFS Science Archives progress meeting minutes, 13 May 2016 Present: NJC, RSC, MAR, RGM, MSH, ACD Apologies: RPB, ETWS, STV, DMR, KTN, AL DoNM: 10:00 am Friday 27th May 2016 in the Villas meeting room. ACD in the chair (room is booked). Actions discharged this time: ---------------------------- ACTION: ETWS/RPB to investigate whether there is sufficient space to also host SDSS DR12. ETWS reported that DR12 is some 12TB uncompressed. Discharged. MSH reported that Ramses15 was full. However RGM noted that a new grant to support the VDFSSA work had been awarded so it was now possible to buy more hardware. ACTION (evolved from previous action): NJC register for the VMC meeting to be held at Keele University during June 2016. NJC had emailed Maria-Rosa but not yet registered. Discharged. ACTION: NJC to check with the VIDEO team when they are likely to supply data for the VIDEO P94 Team Release. Discharged. They are likely to supply data once the most recent ESO release is complete. The new team release will comprise a mixture of P95/P96 data. ACTION: NJC to check with Matt Jarvis whether work can start on the VIDEO P93 ESO Release. Discharged. Work can start on a combined P93/P94 release ACTION: RPB to experiment with ingest via partitioned tables, along the lines of the SSIS paper. Discharged uncompleted. This item is both involved and low-priority for an an action and hence has been moved to the software project deadlines page. Actions carried forward from 22/4/2016 meeting: --------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues: this action still has low priority. ACTION: NJC noted that VISTA mosaicing crashes with normal filtering, which he should fix. Continues. NJC had implemented a fix. Though it completed successfully when he ran some tests it crashed when run 'in anger' on a complete ingest dataset. NJC should revisit and correct his fix. ACTION: NJC noted that he had received a report of a problem with the ESO VVV DR1 release which he should investigate. In one pointing the merged band catalogue contained fewer sources than expected. Continues. This action was very low on NJC's priority list and unlikely to be implemented in the foreseeable future. ACTION: MSH, DMR install and configure the Nagios (https://www.nagios.org/) system monitoring package to monitor the availability of our database services. Continues. Progress is stalled until additional IP addresses are made available to WFAU. ACTION: RPB/NJC to fix the errors encountered while trying to make non-survey releases U14BUA15, U14BUA16, U14BUA18 and U14BUA19. Continues. NJC believed that he had fixed the problems. RPB should re-try the releases and establish whether the fixes had been successful. ACTION: NJC to update Twiki page on future VISTA surveys. Continues. ACTION: RPB to restore VSA replication. Continues. RPB had restored OSA replication but was holding off restoring VSA replication until the current VVV updates had completed. ACTION: ACD to liaise with DMR and STV to establish a set of deadlines for their work. Continues. Actions resulting from the VVV DR3 brainstorming meeting held on 24/2/15: ------------------------------------------------------------------------ ACTION: ETWS (with RSC) to contact Microsoft SQL Server helpdesk for advice about the outgest problem that is the principal bottleneck with the VVV processing (briefly the ingest and outgest speed is limited to approx. 100 Mbit/sec for no apparent reason). This action has arisen due to the failure of Ani Thakar and Alex Szalay to reply to queries on this issue. Continues. Specific points and new actions: -------------------------------- Project management: * RGM noted that the next WFAU Quarterly Meeting would be in the afternoon of Monday 13/6/16. He was still planning to schedule half-yearly reviews over the next couple of months. WFCAM, VISTA and VST updates: * NJC reported on the likely future VISTA surveys. See the following twiki pages: Proposed new VISTA Public Surveys: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ProposedVISTANewSurveys Future WFAU Releases and hardware requirements: http://apache.roe.ac.uk/twiki/bin/view/WFAU/FutureReleasesNotes VIKING-KiDS-GAMA meeting 9-11 May 2016: http://apache.roe.ac.uk/twiki/bin/view/WFAU/VKG20160509 Networking: * The networks appeared to be working nominally. MSH noted that he had received 36 additional IP addresses, which would allow new services to be deployed. WSA/VSA/OSA/GES Operations: * ETWS was on holiday but he continued to superintend operations and had kindly sent the following notes: - The latest batch of WSA data from 12/15 and 01/16 is now ingested. CU2 is currently running and creating JPGs. - Metadata, catalogue data and JPGs for the latest ATLAS data up to 02/16 have been processed and ingested. - The non-ATLAS OSA catalogue data are processed for P95-P96; ingests are underway. - The VVV objid update continues. Hardware and Systems: * RPB had installed the operating system on the EUCLID head node instead of Ramses17. Consequently Ramses17 is not yet operational. * Ramses15 is full. Review of Software Release Deadlines: * Progress towards forthcoming deadlines was reviewed offline. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/PrepareSoft Other Software: * RSC had updated CU19; in the event that a release completes without mishap the software now e-mails the entire operations team announcing success. * NJC had made a number of small fixes. Review of Survey Release Deadlines: * Progress towards forthcoming deadlines was reviewed offline. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * None. Non-survey Data Release: * None. Astrogrid deployment, VO & Data Analysis services: * Nothing to report. Miscellaneous: * MAR noted that the monthly teleconferences with CASU were to be resumed. ACTION MAR: Contact CASU to resume the monthly teleconferences and arrange the date of the next one. * RGM noted that Gerry Gilmore had visited during the previous week and had given an update on the remainder of the GES survey. ACD, 23/5/16.