From acd@roe.ac.uk Thu Oct 15 17:05:01 2015 Date: Thu, 15 Oct 2015 16:04:53 +0000 From: Clive Davenhall To: Bob Mann , Andy Lawrence , "Read, Mike" , Eckhard Sutorius , Mark S. Holliman , Nicholas Cross , Nigel C. Hambly , Rob Blake , Ross Collins , Stelios Voutsinas , Dave Morris , Clive Davenhall Cc: "dante@astro.puc.cl" , James Dunlop , "j.p.emerson@qmul.ac.uk" , ktn keithnoddle.org , "Lorenzo.Rimoldini@unige.ch" , "mcioni@aip.de" , "matt.jarvis@astro.ox.ac.uk" , "mike@ast.cam.ac.uk" , "nigel.metcalfe@durham.ac.uk" , "norman@astro.gla.ac.uk" , "pwl@star.herts.ac.uk" , "rgm@ast.cam.ac.uk" , "Simon.Dye@nottingham.ac.uk" , "s.j.warren@ic.ac.uk" , "tkerr@jach.hawaii.edu" , "tom.shanks@durham.ac.uk" Subject: WFAU VDFS Science Archives progress meeting minutes, 25 September 2015 WFAU VDFS Science Archives progress meeting minutes, 25 September 2015 Present: RGM, RPB, NJC, MAR, ETWS, ACD Apologies: RSC, MSH, STV, KTN, AL DoNM: 10am Friday 9 October 2015 in the Crawford meeting room 1, MAR in the chair (room is booked). Actions discharged this time: ---------------------------- ACTION: NJC and ETWS to troubleshoot software/path issues on recent installations. Done. Adequate but not entirely satisfactory solutions had been found for the various issues. ACTION: ACD to obtain the details of obtaining an mxODBC licence from ETWS and progress obtaining a sufficient number of such licences. Done. New licences would be purchased when a new hardware grant was available. Actions carried forward from 4/9/2015 meeting: --------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues: this action still has low priority. ACTION: NJC noted that VISTA mosaicing crashes with normal filtering, which he should fix. Continues. NJC had implemented a fix and checked it into SVN. However, when it was run 'in anger' it still crashed. NJC to investigate. ACTION: NJC noted that he had received a report of a problem with the ESO VVV DR1 release which he should investigate. In one pointing the merged band catalogue contained fewer sources than expected. Continues. This action was low on NJC's priority list. Actions resulting from the VVV DR3 brainstorming meeting held on 24/2/15: ------------------------------------------------------------------------ ACTION: RPB to experiment with ingest via partitioned tables, along the lines of the SSIS paper. Deferred (changed back from 'continues'). This action has now been overtaken by events: the VVV DR3 processing would now be done differently. However, the technique was still potentially useful for future work and should be investigated, but not as a matter of urgency. ACTION: RGM (was RSC, RGM): (i) RSC to write a succinct summary of the ingest and outgest problem that is the principal bottleneck with the VVV processing (briefly the ingest and outgest speed is limited to approx. 100 Mbit/sec for no apparent reason) and (ii) RGM to compose an e-mail message around this summary and send it to Ani Thakar. Continues. RSC had composed a succinct summary and sent it to RGM. RGM had emailed Ani Thakar but no reply to date. RGM had sent a reminder but had still received no response. Specific points and new actions: -------------------------------- Project management: * RGM summarised progress and plans for the forthcoming VDFSSA grant application. RGM also mentioned that on 1/10/15 George Beckett would start at as the half-time LSST Project Manager and on 1/11/15 Marcus Ebert as the half-time software developer. They would be based in R5 and R13 respectively. WFCAM, VISTA and VST updates: * WFCAM was new believed to be back on the telescope following one of the other instruments developing a leak. Networking: * Nothing to report; the networks were performing nominally. WSA/VSA/OSA/GES Operations: * VVV processing was currently paused while additional disk space was freed-up. * Ingest of VISTA metadata for February and March 2015 (the last two months of P94) is finished. Ingest of all non-VVV catalogue data is well underway. * Ingest of WFCAM metadata up to February 2015 is also finished. * Ingest of uncorrected VST catalogues (v1.1) is complete to September 2014 (P93). The illumination correction is currently being calculated and updated. * Some checks had been performed on the incomplete GES parameter files received and a couple of minor issues reported to Cambridge. Table AstroAnalysis was being revised to accommodate the changed columns in the input parameter tables. Hardware and Systems: * Most systems were operating nominally, but a few points were mentioned, which are noted below. * A disk had failed on shepseskaf which translated into a failure in user disk40. The failed disk still needs replacing. * A disk had failed on RAMSES2 at the same time as the OSA processing crashed, compounding the problems on that machine. Software: * NJC reported that he had: - updated the main pipeline and some other frequently executed bits of code to work with the newly updated version of PyFITS, so that all servers can be updated to the latest version, - changed the QualityBitFlagging to operate in chunks when the number of new rows to flag is extremely large, - modified the to infrastructure to allow the improvements to the variability code to go through and also to allow us to create or ingest medium deep stacks/mosaics from very deep surveys, - finally he had made various small bug fixes. * ETWS reported that: - Development of the WSALM (Lockheed Martin) branch had progressed well. A test run with all 14A/B data was successful and a new release database was created in time for the most recent visit by Lockheed-Martin. - Problems previously encountered with updating the third-party software seem to have been fixed by reverting to earlier versions of scipy and numpy. * RSC was not present at the meeting but had reported the following enhancements from a Cypriot beach: - Merged the parallel CU19 code into the main CU19 script, generalised the code so that it was no longer specific to vvvDetection table bulk copies and applied it to all chunked outgests now by default for all surveys. Some further improvements still to be done: e.g. not apply to cases where there is a single chunk and re-enable support for automatic resumption of chunked copy after interruptions (the parallelised version currently requires the chunk number to be manually specified on resumption). - Also, the current delay in releasing the VVV DR3 database due to time consuming file group shrinking has highlighted the importance of ensuring the data are copied in primary key order irrespective of the peculiarities of the monthly-split detection table data (this would also have sped up ingests and avoided some of the earlier interruption in the CU19 run): so this is also high on my list of necessary improvements to the process (it would also help reduce the [temporary] disk space requirements for a release on the public server). * Finally noted that he had speeded up cross-identification. Review of Deadlines: The review of deadlines was skipped (to popular acclaim) due to the meeting running late. The current list of deadlines is available at: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * None Non-survey Data Release: * A WSA release had been made in connection with the Lockheed-Martin work. Astrogrid deployment, VO & Data Analysis services: * MSH was absent but there were no known issues to discuss. Miscellaneous: * Nothing to report. ACD, Original: 25/9/15.