From acd@roe.ac.uk Mon Apr 4 16:55:26 2016 Date: Mon, 4 Apr 2016 15:55:15 +0000 From: Clive Davenhall To: Bob Mann , Andy Lawrence , "Read, Mike" , Eckhard Sutorius , Mark S. Holliman , Nicholas Cross , Nigel C. Hambly , Rob Blake , Ross Collins , Stelios Voutsinas , Dave Morris , Clive Davenhall Cc: "dante@astro.puc.cl" , James Dunlop , "j.p.emerson@qmul.ac.uk" , ktn keithnoddle.org , "Lorenzo.Rimoldini@unige.ch" , "mcioni@aip.de" , "matt.jarvis@astro.ox.ac.uk" , "mike@ast.cam.ac.uk" , "nigel.metcalfe@durham.ac.uk" , "norman@astro.gla.ac.uk" , "pwl@star.herts.ac.uk" , "rgm@ast.cam.ac.uk" , "Simon.Dye@nottingham.ac.uk" , "s.j.warren@ic.ac.uk" , "tom.shanks@durham.ac.uk" Subject: WFAU VDFS Science Archives progress meeting minutes, 1 April 2016 WFAU VDFS Science Archives progress meeting minutes, 1 April 2016 Present: MSH, MAR, RPB, RGM, NJC, ACD Apologies: RSC, STV, DMR, KTN, AL, ETWS DoNM: 10:00 am Friday 15th April 2016 in the Villas meeting room. MAR in the chair (room is booked). Actions discharged this time: ---------------------------- ACTION: RPB to backup VVV. Done. ACTION: RGM (was RSC, RGM): (i) RSC to write a succinct summary of the ingest and outgest problem that is the principal bottleneck with the VVV processing (briefly the ingest and outgest speed is limited to approx. 100 Mbit/sec for no apparent reason) and (ii) RGM to compose an e-mail message around this summary and send it to Ani Thakar. Discharged. RSC had composed a succinct summary and sent it to RGM. RGM had made several attempts to contact both Ani Thakar and Alex Szalay over an extended period, but to no avail. The action was discharged uncompleted as further progress was unlikely to be made. Actions carried forward from 4/3/2016 meeting: --------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan NEW ACTION arising: ETWS to investigate whether there is sufficient space to also host SDSS DR12. ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues: this action still has low priority. ACTION: NJC noted that VISTA mosaicing crashes with normal filtering, which he should fix. Continues. NJC had implemented a fix. Though it completed successfully when he ran some tests it crashed when run 'in anger' on a complete ingest dataset. NJC should revisit and correct his fix. ACTION: NJC noted that he had received a report of a problem with the ESO VVV DR1 release which he should investigate. In one pointing the merged band catalogue contained fewer sources than expected. Continues. This action was very low on NJC's priority list and unlikely to be implemented in the foreseeable future. ACTION (evolved from previous action): NJC register for the VMC meeting to be held at Keele University during June 2016. NJC had emailed Maria-Rosa but not yet registered. Continues. ACTION: DMR, NJC, RPB ensure that the new-style OSA release has access to all the database tables that it needs; specifically resolve the problems encountered with neighbour tables. Continues, RPB was fixing up VPHAS ACTION: MSH, DMR install and configure the Nagios (https://www.nagios.org/) system monitoring package to monitor the availability of our database services. Continues. Progress is stalled until additional IP addresses are made available to WFAU. ACTION: RPB (was NJC) to fix the errors encountered while trying to make non-survey releases U14BUA15, U14BUA16, U14BUA18 and U14BUA19. Continues, NJC thought the issues had been fixed. RPB to try the releases again. ACTION: NJC to update Twiki page on future VISTA surveys. Continues. ACTION: RPB to restore VSA replication and check OSA replication. Continues. RPB had restored the VSA replication but it crashed three days later. He was also unsure whether OSA priority replication was running or note. Further investigation was required in both cases. ACTION: MAR to contact Phil/Dante about VVV QC. Continues. Actions resulting from the VVV DR3 brainstorming meeting held on 24/2/15: ------------------------------------------------------------------------ ACTION: RPB to experiment with ingest via partitioned tables, along the lines of the SSIS paper. Deferred (changed back from 'continues'). This action has now been overtaken by events: the VVV DR4 (was DR3) processing would now be done differently. However, the technique was still potentially useful for future work and should be investigated before the next VVV release, which is time away. NEW ACTION arising: ETWS to contact Microsoft SQL Server helpdesk for advice about the outgest problem that is the principal bottleneck with the VVV processing (briefly the ingest and outgest speed is limited to approx. 100 Mbit/sec for no apparent reason). This action has arisen due to the failure of Ani Thakar and Alex Szalay to reply to queries on this issue. Specific points and new actions: -------------------------------- Project management: * RGM noted that the Quarterly Meeting that was to be held during April would be deferred until June because of staff absences. During April RGM would be holding half-yearly reviews for all staff which would allow him to discuss issues that would otherwise have been covered during the Quarterly Meeting with individual members of staff. WFCAM, VISTA and VST updates: * Nothing to report. Networking: * A meeting with UoE IS staff to discuss future network arrangements on site was planned but the ATC still needed to select a delegate to attend. Also MSH was waiting to hear from Sam Wilson about the activation of IPv6 on our network and hence the availability of additional addresses. WSA/VSA/OSA/GES Operations: * The VVV ingest was progressing more slowly than anticipated. Some of the scripts had failed, leaving the data ingests incomplete. Other bits of the VVV curation software are broken. The pawprints could not be generated because of the failed ingests. The return of ETWS from annual leave to investigate these problems was eagerly awaited. * Other VSA operations were going well; a VIKING Team release was in preparation. * VIKING and VVV ESO releases were also going well. * Some GES inquiries had been received and successfully resolved. Hardware and Systems: * MSH noted that a substantial amount of new hardware had arrived, including a cluster for EUCLID, a cluster for the VDFSS work, the machine bought on behalf of Sadegh Khochfar and the new KVM switch. The last should allow some operations which hitherto had to be carried out from the machine room to be done remotely. All this equipment was racked-up and most was connected. There were still decisions to be made about how the machine for Sadegh Khochfar was to be deployed. A beta version of SLQ Server for Linux is now available and MSH was inclined install it on Sadegh's machine by way of an experiment. Software: * NJC noted the following progress: - He had completed basic working list driven processing for five pointings of VIKING data with J-band measurements and ZY band frames. He was now adding improvements to also compute luptitudes, and to take input from a file as well as from an SQL selection. - He had also processed some of the mosaics from DXS and had modified the mosaic X-talk to work on a range of mosaics and rather than just UDS, although this has yet to be fully tested. - He had written a script to modify Chilean VVV deep stacks by adding the header values that are needed by subsequent WFAU processing steps and checked that these stacks ingest. He had also made tiles from these stacks and checked that subsequent processing completes satisfactory and that they ingest. A few changes may still be needed for catalogues. - Processing for the VIKING ESO release is complete, although some header changes may be needed to accommodate recent changes required by ESO, and some headers apparently do not include one or two values. This issue needs investigating. - Processing of the VVV ESO release would complete later in the day of the meeting, although again some header changes may be needed to accommodate recent changes required by ESO. Review of Deadlines: * Progress towards forthcoming deadlines was reviewed offline. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines ACTION: NJC to check with the VIDEO team when they are likely to supply data for the VIDEO P94 Team Release. ACTION: NJC to remind Maria-Rosa about specifying what data are to be included in the VMC P93 ESO Release. ACTION: NJC to check with Matt Jarvis whether work can start on the VIDEO P93 ESO Release. ACTION: RPB to check whether the error bit calculation has completed successfully for the UHS Team Release. * It was decided to henceforth include software development deadlines in the review of deadlines. ACTION: NJC to produce a list of software deadlines for the VDFSS work. ACTION: ACD to liaise with DMR and STV to establish a set of deadlines for their work. ACTION: ACD to incorporate software deadlines in the Review of Deadlines twiki pages. Survey Data Release: * The VVV DR4 Public Release was made. Non-survey Data Release: * One non-survey release had been made: VISTA P95 D0771. Astrogrid deployment, VO & Data Analysis services: * Nothing to report. Miscellaneous: * MSH noted that he would only be in the Observatory on Tuesday 5/4/16 and Wednesday 6/4/16 next week and after that would be on leave until Monday 25/4/16. He would have only intermittent e-mail access during his absence. * RGM added that he would be on leave Monday 4/4/16 to Friday 8/4/16 inclusive. ACD, original: 1/4/16, revised: 4/4/16.