From acd@roe.ac.uk Mon Jul 4 15:25:26 2016 Date: Mon, 4 Jul 2016 14:24:25 +0000 From: Clive Davenhall To: Bob Mann , Andy Lawrence , "Read, Mike" , Eckhard Sutorius , Mark S. Holliman , Nicholas Cross , Nigel C. Hambly , Rob Blake , Ross Collins , Stelios Voutsinas , Dave Morris , Clive Davenhall Cc: "dante@astro.puc.cl" , James Dunlop , "j.p.emerson@qmul.ac.uk" , ktn keithnoddle.org , "Lorenzo.Rimoldini@unige.ch" , "mcioni@aip.de" , "matt.jarvis@astro.ox.ac.uk" , "mike@ast.cam.ac.uk" , "nigel.metcalfe@durham.ac.uk" , "norman@astro.gla.ac.uk" , "pwl@star.herts.ac.uk" , "rgm@ast.cam.ac.uk" , "Simon.Dye@nottingham.ac.uk" , "s.j.warren@ic.ac.uk" , "tom.shanks@durham.ac.uk" Subject: WFAU VDFS Science Archives progress meeting minutes, 1 July 2016 WFAU VDFS Science Archives progress meeting minutes, 1 July 2016 Present: NJC, ETWS, ACD Apologies: STV, DMR, KTN, AL, RPB, RGM, MAR, MSH, RSC DoNM: 10:00 am Friday 15th July 2016 in the Villas Meeting Room MAR in the chair (room is booked). Note: the meeting was not quorate so only brief discussions were held. Actions discharged this time: ---------------------------- None. Actions carried forward from 27/5/2016 meeting: ---------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan We currently have space so it was decided to start transferring DR12 ACTION: ETWS to transfer SDSS DR12 Continues. ETWS had had contacted Ani Thakar, who had advised that SDSS DR13 would be released at the end of July (and that SDSS DR12 had recently been removed from the SDSS servers in preparation). RGM was happy to wait for DR13 but did not want further significant delays as one of his students planned to work with the date. ACTION: MSH, DMR install and configure the Nagios (https://www.nagios.org/) system monitoring package to monitor the availability of our database services. Continues. ACTION: ACD to speak to DMR about the scope and schedule for installing and configuring Nagios and its intended subsequent use. Continues. ACTION: RPB/NJC to fix the errors encountered while trying to make non-survey releases U/14B/UA15, U/14B/UA16, U/14B/UA18 and U/14/BUA19. Continues. NJC believed that he had fixed the problems. RPB re-ran the releases, two of which worked, but the other two failed and require further investigation by NJC. Check status with RPB. ACTION: RPB to restore VSA replication. Continues. RPB had restored OSA replication but was holding off restoring VSA replication until the current VVV updates had completed, which is likely to take approximately another week. ACTION: ACD to liaise with DMR and STV to establish a set of deadlines for their work. Continues. ACTION: NJC to set-up WISExSCOS neighbour tables. Continues. ACTION: MAR to look at H-ATLAS documentation. Continues. Actions resulting from the VVV DR3 brainstorming meeting held on 24/2/15: ------------------------------------------------------------------------ ACTION: ETWS (with RSC) to contact Microsoft SQL Server helpdesk for advice about the outgest problem that is the principal bottleneck with the VVV processing (briefly the ingest and outgest speed is limited to approx. 100 Mbit/sec for no apparent reason). This action has arisen due to the failure of Ani Thakar and Alex Szalay to reply to queries on this issue. Continues. ETWS did not feel that he was the best person to progress this issue. ACD should discuss how to proceed with RSC. ACTION: ACD to discuss with RSC the best way to progress investigations of the outgest speed bottleneck. Leave as an action for now until ACD present. Specific points and new actions: -------------------------------- Project management: * Interviews for a new Chancellor's Fellow to work in the WFAU area had been held but no appointment had yet been announced. WFCAM, VISTA and VST updates: * CASU had mentioned that VISTA had recently suffered a helium leak and consequently some some downtime in the near future is anticipated. Networking: * Nothing to report; the networks were operating nominally. WSA/VSA/OSA/GES Operations: * ETWS reported the following progress: - Transfer of the latest WSA data, up to May 2016. - Ingested Metadata and produced JPGs for the Feb. and Mar. 2016 VISTA data. - Updated all illumination correction values for non-ATLAS VST data. (ATLAS had been already updated for the latest release.) * NJC noted that he had worked on the following: - Setting up the WISExSCOS Photometric Redshift Catalogue and updating the 2MPZ (2MASS Photometric Redshift Catalogue) along with MAR and NCH. These catalogues are all-sky photometric redshift catalogues, and so are useful for many of our surveys and keep up collaborations with Maciej Bilicki and John Peacock. - Starting to set up HATLAS database. He had got some examples of all the data types, set up schemas, and made a test database. He still has to check the headers of the GAMA images and write the code to populate the tables. - Working to produce a UKIDSS-UDS release, which is now complete. This release was made on a very short timescale, which ended up being a little too short, but the database is now set up and everything looks fine. Hardware and Systems: * Neither MSH nor RPB was not present to report, but all the servers were believed to be operating nominally. * ETWS reported that his ATC-managed desktop machine had once again spontaneously and unexpectedly rebooted, losing some work and returning with some previously- installed software missing. One solution to this problem would be for ETWS to self-manage this machine. However, MHE has now allowed ETWS the previously- denied privilege to mount USB sticks into fstab. This concession should allow him to carry out most of the necessary tasks, so he will not proceed with self-management in the immediate future. Review of Software Release Deadlines: * Progress towards the software deadlines was reviewed See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/PrepareSoft Other Software: * NJC reported that he had worked on the following: - Testing the imcore_list code to try to work out the field-to-field offsets. He had had no luck so far, but had ruled out a few possibilities, and may have found some other issues. He will email Mike Irwin soon. - Started testing the GAaP software provided by Konrad Kuiken for gaussianising images, for use in improved matched aperture photometry. He has successfully run the code, but has established that images have to be a) uncompressed and b) broken out into individual detectors. - More ESO Release work. The VMC release is almost ready. He has been waiting for RPB to rerun the release, after he spotted a mistake in the list he had sent him. Good progress is now being made. * ETWS noted that he had: - Written schemas and code to parse OGLE data provided by the VMC team for inclusion with the next VMC release. - Installed the GAaP code ('Gaussian Aperture and PSF'). Review of Survey Release Deadlines: * Progress towards forthcoming survey deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * UKIDSSDR11PLUSUDSONLY (yes, really) was released on 29/6/16. Non-survey Data Release: * None Astrogrid deployment, VO & Data Analysis services: * Nothing to report. Miscellaneous: * NJC had attended a VMC team meeting at Keele. The meeting had a good atmosphere and NJC's talk was well-received. NJC's main concern arising from the meeting is that CASU are reprocessing all VISTA data to improve the Galactic extinction correction. At the level of individual paw-prints only the zero points in headers are affected. However, at the tile level the catalogues have to be re-generated. For WFAU, handling the reprocessed data would require significant additional transfer and processing time and storage capacity. The main concern is the VVV Survey (which is both the largest survey and the one most affected by extinction). ACD, original version: 1/7/16, final version: 4/7/16.