From acd@roe.ac.uk Wed Jun 1 22:03:57 2016 Date: Wed, 1 Jun 2016 21:03:24 +0000 From: Clive Davenhall To: Bob Mann , Andy Lawrence , "Read, Mike" , Eckhard Sutorius , Mark S. Holliman , Nicholas Cross , Nigel C. Hambly , Rob Blake , Ross Collins , Stelios Voutsinas , Dave Morris , Clive Davenhall Cc: "dante@astro.puc.cl" , James Dunlop , "j.p.emerson@qmul.ac.uk" , ktn keithnoddle.org , "Lorenzo.Rimoldini@unige.ch" , "mcioni@aip.de" , "matt.jarvis@astro.ox.ac.uk" , "mike@ast.cam.ac.uk" , "nigel.metcalfe@durham.ac.uk" , "norman@astro.gla.ac.uk" , "pwl@star.herts.ac.uk" , "rgm@ast.cam.ac.uk" , "Simon.Dye@nottingham.ac.uk" , "s.j.warren@ic.ac.uk" , "tom.shanks@durham.ac.uk" Subject: WFAU VDFS Science Archives progress meeting minutes, 27 May 2016 WFAU VDFS Science Archives progress meeting minutes, 27 May 2016 Present: NJC, ETWS, RPB, MSH, ACD Apologies: RGM, RSC, MAR, STV, DMR, KTN, AL DoNM: 10:00 am Friday 10th June 2016 in the Villas meeting room. MAR in the chair (room is booked). Actions discharged this time: ---------------------------- ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Discharged uncompleted: unlikely to be completed in the near future and therefore discharged as an action and moved to the software deadlines page. ACTION: NJC noted that VISTA mosaicing crashes with normal filtering, which he should fix. Discharged uncompleted: unlikely to be completed in the near future and therefore discharged as an action and moved to the software deadlines page. ACTION: NJC noted that he had received a report of a problem with the ESO VVV DR1 release which he should investigate. In one pointing the merged band catalogue contained fewer sources than expected. Discharged uncompleted: unlikely to be completed in the near future and therefore discharged as an action and moved to the software deadlines page. ACTION: NJC to update Twiki page on future VISTA surveys. Discharged. Actions carried forward from 13/5/2016 meeting: ---------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan ACTION: MSH, DMR install and configure the Nagios (https://www.nagios.org/) system monitoring package to monitor the availability of our database services. Continues. NEW ACTION: ACD to speak to DMR about the scope and schedule for installing and configuring Nagios and its intended subsequent use. ACTION: RPB/NJC to fix the errors encountered while trying to make non-survey releases U/14B/UA15, U/14B/UA16, U/14B/UA18 and U/14/BUA19. Continues. NJC believed that he had fixed the problems. RPB re-ran the releases, two of which worked, but the other two failed and require further investigation by NJC. ACTION: RPB to restore VSA replication. Continues. RPB had restored OSA replication but was holding off restoring VSA replication until the current VVV updates had completed, which is likely to take approximately another week. ACTION: ACD to liaise with DMR and STV to establish a set of deadlines for their work. Continues. ACTION MAR: Contact CASU to resume the monthly teleconferences and arrange the date of the next one. Continues. Actions resulting from the VVV DR3 brainstorming meeting held on 24/2/15: ------------------------------------------------------------------------ ACTION: ETWS (with RSC) to contact Microsoft SQL Server helpdesk for advice about the outgest problem that is the principal bottleneck with the VVV processing (briefly the ingest and outgest speed is limited to approx. 100 Mbit/sec for no apparent reason). This action has arisen due to the failure of Ani Thakar and Alex Szalay to reply to queries on this issue. Continues. ETWS did not feel that he was the best person to progress this issue. ACD should discuss how to proceed with RSC. NEW ACTION: ACD to discuss with RSC the best way to progress investigations of the outgest speed bottleneck. Specific points and new actions: -------------------------------- Project management: * Nothing to report. WFCAM, VISTA and VST updates: * Nothing to report. Networking: * The performance of the networks had been considerably less than nominal. Problems had been encountered logging in to the WFAU servers (from both on- and off-site), starting on Wed. 25/5/16. The problem was caused by a new firewall on the University network which, unbeknownst to the University sysadmins, was randomising TCP/IP packet sequence numbers as a security measure. This reconfiguration would not, of itself, have caused problems, but in addition JANET had failed to implement some requested changes, resulting in 'asymmetric routing' (that is, packets going out via one route and returning via another). The problem is now apparently fixed, although connections to the servers still seem more fragile than before. Work continues. WSA/VSA/OSA/GES Operations: * ETWS reported the following progress: - OSA: catalogue data are now completely ingested up to 03/2016. The next step is updating the illumination correction entries for ATLAS. - WFCAM: data are JPGed, metadata and catalogue data are ingested up to 01/2016; that is, semester 15A is complete. - VSA: the VVV objID update is still running and will take approximately another week to complete. * NJC reported on the preparation of ESO releases as follows: 'I have been sorting out data for ESO releases for VVV, VIKING, VIDEO and VMC. The data are ready for all of them apart from VMC. There has been some back and forth with ESO, and various updates to the ESO release software. VMC is not quite ready because we are given some team catalogues that we ingest and then run the ESO release code. The ingest took a bit longer: each of the catalogues we received was in a slightly different format. The PSF catalogues were similar to last time, so no major issues there, except that I found an oddity in the colours which was subsequently fixed. The Cepheid data were presented in a completely new, and possibly better, format. There were new RRLyrae catalogues. Over the next few weeks I will try to put a guide together for VMC and other surveys who produce team data, to speed up this process, now that all the issues are fresh in my mind.' Hardware and Systems: * RPB was configuring Ramses17 (having finally successfully distinguished it from the EUCLID head node) but this was proving more difficult than anticipated (the version of Windows being installed has obsolete network drivers, but in order to locate and retrieve more recent ones a working network is required, which requires installed network drivers). Review of Software Release Deadlines: * ACD and NJC had reviewed the software deadlines a couple of days ago, so they were only briefly discussed at the meeting. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/PrepareSoft Other Software: * NJC reported as follows: 'Steve Warren reported two issues with the first matched aperture pipeline release: (1) there is a small offset in magnitudes which has an RA dependence, but no magnitude dependence; (2) he would prefer a linear calibrated flux to Luptitudes. I am not 100% sure what the first problem is: I think it is probably that the seeing measurement used by the grouting is not the same in the remeasured catalogues and may be easily fixed by setting this after extraction before grouting. I am currently trying to test this, but network problems have caused delays. In addition, this part of the code has been a bit slower because I have used a larger than necessary input catalogue around each image (based on a circle) and filtered out the objects beyond the images' edges later than necessary. I am addressing these issues at the same time. I think I will solve the second issue by converting to AB system and from there to the flux in Janskys. This approach is straightforward, and the main consideration will be whether I just remove the luptitudes and have calibrated fluxes or include both. 'I have added more documentation in the VSA about which magnitudes to use (in the data overview), as suggested during the VIKING-KiDS-GAMA meeting, although I could do with someone to have a check through to see if I missed anything. I should also complete the VVV guide.' ACTION MAR: Check the revised VSA documentation and give feedback to NJC. Review of Survey Release Deadlines: * Progress towards forthcoming survey deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * None. The VHS P95 was stalled due to insufficient disk space. It would proceed once RPB's incantations have quickened new life into Ramses17. Non-survey Data Release: * U/14B/UA06 had been released successfully. U/14B/UA15, U/14B/UA16, U/14B/UA18 and U/14/BUA19 had all failed. NJC had fixed two of these releases and they could be retried. Astrogrid deployment, VO & Data Analysis services: * Nothing to report. Miscellaneous: * RPB, RSC, MSH (and KTN) would be at a EUCLID meeting 30/5/16 to 3/6/16. RPB would then be on holiday 6-10/6/16. NJC will be attending a VMC meeting in Keele 27-29/6/16. ACD, 1/6/16.