From acd@roe.ac.uk Wed Mar 16 12:15:08 2016 Date: Wed, 16 Mar 2016 12:15:00 +0000 From: Clive Davenhall To: Bob Mann , Andy Lawrence , "Read, Mike" , Eckhard Sutorius , Mark S. Holliman , Nicholas Cross , Nigel C. Hambly , Rob Blake , Ross Collins , Stelios Voutsinas , Dave Morris , Clive Davenhall Cc: "dante@astro.puc.cl" , James Dunlop , "j.p.emerson@qmul.ac.uk" , ktn keithnoddle.org , "Lorenzo.Rimoldini@unige.ch" , "mcioni@aip.de" , "matt.jarvis@astro.ox.ac.uk" , "mike@ast.cam.ac.uk" , "nigel.metcalfe@durham.ac.uk" , "norman@astro.gla.ac.uk" , "pwl@star.herts.ac.uk" , "rgm@ast.cam.ac.uk" , "Simon.Dye@nottingham.ac.uk" , "s.j.warren@ic.ac.uk" , "tom.shanks@durham.ac.uk" Subject: WFAU VDFS Science Archives progress meeting minutes, 4 March 2016 WFAU VDFS Science Archives progress meeting minutes, 4 March 2016 Present: RSC, MAR, ETWS, MSH, ACD Apologies: RGM, RPB, NJC, STV, DMR, KTN, AL DoNM: 10:00 am Friday 18th March 2016 in the Crawford Meeting Room 1 MAR in the chair (room is booked; note the change of room again). Actions discharged this time: ---------------------------- ACTION: NJC to look at improving neighbour table documentation. ACTION: MSH to write a report on the C2 works. Actions carried forward from 19/2/2016 meeting: --------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues: this action still has low priority. ACTION: NJC noted that VISTA mosaicing crashes with normal filtering, which he should fix. Continues. NJC had implemented a fix. Though it completed successfully when he ran some tests it crashed when run 'in anger' on a complete ingest dataset. NJC should revisit and correct his fix. ACTION: NJC noted that he had received a report of a problem with the ESO VVV DR1 release which he should investigate. In one pointing the merged band catalogue contained fewer sources than expected. Continues. This action was very low on NJC's priority list and unlikely to be implemented in the foreseeable future. ACTION (evolved from previous action): NJC register for the VMC meeting to be held at Keele University during June 2016. NJC had emailed Maria-Rosa but not yet registered. Continues. ACTION: DMR, NJC, RPB ensure that the new-style OSA release has access to all the database tables that it needs; specifically resolve the problems encountered with neighbour tables. Continues. ACTION: MSH, DMR install and configure the Nagios[1] system monitoring package to monitor the availability of our database services. Continues. Progress is stalled until additional IP addresses are made available to WFAU. ACTION: NJC to fix the errors encountered while trying to make non-survey releases U14BUA15, U14BUA16, U14BUA18 and U14BUA19. Continues. ACTION: NJC to update Twiki page on future VISTA surveys. Continues. Actions resulting from the VVV DR3 brainstorming meeting held on 24/2/15: ------------------------------------------------------------------------ ACTION: RPB to experiment with ingest via partitioned tables, along the lines of the SSIS paper. Deferred (changed back from 'continues'). This action has now been overtaken by events: the VVV DR4 (was DR3) processing would now be done differently. However, the technique was still potentially useful for future work and should be investigated before the next VVV release, which is time away. ACTION: RGM (was RSC, RGM): (i) RSC to write a succinct summary of the ingest and outgest problem that is the principal bottleneck with the VVV processing (briefly the ingest and outgest speed is limited to approx. 100 Mbit/sec for no apparent reason) and (ii) RGM to compose an e-mail message around this summary and send it to Ani Thakar. Continues. RSC had composed a succinct summary and sent it to RGM. RGM had emailed Ani Thakar but no reply to date. RGM had sent a reminder but had still received no response. RGM had also contacted Alex Szalay and not received a reply from him either. He would consider how to proceed. Specific points and new actions: -------------------------------- Project management: * Nothing to report. WFCAM, VISTA and VST updates: * Nothing to report. Networking: * Nothing to report; everything appeared to be working nominally. WSA/VSA/OSA/GES Operations: * ETWS reported: - WSA: The UHS release is underway. - VSA: All the catalogues have been ingested up to Oct 2015 (the start of P96) apart from some broken files which are pending a response from CASU (Carlos). - VVV: The catalogue ingests have finished up to Oct 2015 (beginning of P96). The objID update has started. - OSA: All ATLAS and Chilean non-survey data have been ingested apart from some broken list-driven files. Also all the illumination corrections have been calculated and updated for the ATLAS and the Chilean non-survey data. Hardware and Systems: * MSH noted that the system ordered jointly with Sadegh Khochfar had been delivered but was not yet installed; that should happen next week. Various other hardware was also on order. Software: * ETWS reported: - OSA: The code had been modified to be run only on the Chilean non-surveys and a list of these had been included in SystemConstants. Also a rewrite of the illumination correction update using CTEs (Common Table Expressions) has sped up the processing by a factor of approx. sixty. - WSA: The UHS had been included in the Browser Parser. * ETWS also mentioned a weird, intermittent error that had occurred several times while running the VVV ingests on ramses16. The (unhelpful) error message was: 'there is insufficient memory available in the buffer pool'; it was unclear whether the problem was software or hardware. If the job was simply restarted it would run successfully to conclusion. ACTION: MSH, ETWS, RPB to investigate intermittent error running VVV on ramses16: 'there is insufficient memory available in the buffer pool'. Review of Deadlines: * Progress towards forthcoming deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * The UHS release was still running. It was currently re-seaming and it was anticipated that completion would be during the week beginning Mon. 7/3/16. * The full GES iDR4 release had been made on 28/2/16. To date one problem had been reported, which was solved by regenerating the views on table AstroAnalysis. Non-survey Data Release: * None. * It was hoped to include a non-survey release for Nial Tanvir between the forthcoming VMC and VIKING releases. This release is a small survey of approx. fifty tiles. Astrogrid deployment, VO & Data Analysis services: * MSH noted that: - The ASTERICS DADI Second Technology Forum would be held in Edinburgh on 7-8/3/16. For details see: https://www.asterics2020.eu/dokuwiki/doku.php?id=open:wp4:wp4techforum2 - RGM's student had been given the code to create coverage maps and was apparently using it successfully. Miscellaneous: * ACD mentioned the 28th International Conference on Scientific and Statistical Database Management (SSDBM 2016) that would be held in Budapest during 18-20/7/16. Many of the topics to be discussed seemed relevant to WFAU work and astronomy was specifically mentioned as an application area of interest. For details see: http://ssdbm2016.org/ and the e-mail circulated by Nuria Lorente to the ADASS announcement list on 3/3/16. ACD, 4/3/16.