From acd@roe.ac.uk Tue Aug 11 11:39:49 2015 Date: Tue, 11 Aug 2015 10:39:37 +0000 From: Clive Davenhall To: Bob Mann , Andy Lawrence , "Read, Mike" , Eckhard Sutorius , Mark S. Holliman , Nicholas Cross , Nigel C. Hambly , Rob Blake , Ross Collins , Stelios Voutsinas , Dave Morris , Clive Davenhall Cc: "dante@astro.puc.cl" , James Dunlop , "j.p.emerson@qmul.ac.uk" , ktn keithnoddle.org , "Lorenzo.Rimoldini@unige.ch" , "mcioni@aip.de" , "matt.jarvis@astro.ox.ac.uk" , "mike@ast.cam.ac.uk" , "nigel.metcalfe@durham.ac.uk" , "norman@astro.gla.ac.uk" , "pwl@star.herts.ac.uk" , "rgm@ast.cam.ac.uk" , "Simon.Dye@nottingham.ac.uk" , "s.j.warren@ic.ac.uk" , "tkerr@jach.hawaii.edu" , "tom.shanks@durham.ac.uk" Subject: WFAU VDFS Science Archives progress meeting minutes, 7 August 2015 WFAU VDFS Science Archives progress meeting minutes, 7 August 2015 Present: RSC, NJC, ETWS, RPB, MSH, MAR, ACD Apologies: RGM, STV, KTN, AL DoNM: 10am Friday 21 August 2015 in the Villas meeting room, MAR in the chair (room is booked). Actions discharged this time: ---------------------------- ACTION: RPB to compile a Web page listing all the completed releases. Done. ACTION: RPB to investigate whether the way in which VVV extractions are allocated to machines could be rejigged to improve performance. Discharged as incomprehensible. RPB did not understand the action and hence could not attempt to carry it out. ACTION: MAR to contact AJC to see if he was able to query our databases remotely via SSH tunnel. Done. MAR had e-mailed AJC but he had not replied. However, both MAR and RSC were able to successfully submit database queries via an SSH tunnel so no further action was believed to be needed. Actions carried forward from 17/7/2015 meeting: ---------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues: this action still has low priority. ACTION: ACD to obtain the details of obtaining an mxODBC licence from ETWS and progress obtaining a sufficient number of such licences. Continues. There are currently no issues in using the version of mxODBC we have but it is not desirable to do so in the long- or medium-term. ACD should liaise with MSH to ensure that cost of the mxODBC licences is included in the plans for spending the forthcoming hardware grant. ACTION: NJC noted that VISTA mosaicing crashes with normal filtering, which he should fix. Continues. NJC had implemented a fix and checked it into SVN. However, when it was run 'in anger' it still crashed. NJC to investigate. ACTION: NJC noted that he had received a report of a problem with the ESO VVV DR1 release which he should investigate. In one pointing the merged band catalogue contained fewer sources than expected. Continues. This action was low on NJC's priority list. ACTION NJC: check that CU19 is outgesting a specific list from TilePawprints' tables. Continues. Actions resulting from the VVV DR3 brainstorming meeting held on 24/2/15: ------------------------------------------------------------------------ ACTION: RPB to experiment with ingest via partitioned tables, along the lines of the SSIS paper. Continues (changed from 'deferred'). RPB had investigated various other options and found them wanting. He would now proceed to work on the present action. ACTION: RGM (was RSC, RGM): (i) RSC to write a succinct summary of the ingest and outgest problem that is the principal bottleneck with the VVV processing (briefly the ingest and outgest speed is limited to approx. 100 Mbit/sec for no apparent reason) and (ii) RGM to compose an e-mail message around this summary and send it to Ani Thakar. Continues. RSC had composed a succinct summary and sent it to RGM. RGM had emailed Ani Thakar but no reply to date. RGM had sent a reminder but had still received no response. ACTION: NJC to remove dependencies on old libraries. Done. NJC had modified the code so that it now accessed FITS files using astropy rather than pyfits. Consequently VOTable tables were accessed correctly. NEW ACTION: NJC had noticed a new problem when some of the code was run on Teti (which has the more recent libraries installed). He should isolate and characterise this problem so that it could be investigated. Specific points and new actions: -------------------------------- Project management: * RGM was away and there were no known items to report. * The information being assembled for inclusion in the forthcoming VDFSS grant application was reviewed. WFCAM, VISTA and VST updates: * Nothing to report; all the telescopes were believed to be operating nominally. Networking: * Nothing to report; the networks were performing nominally. WSA/VSA/OSA/GES Operations: * ETWS reported that ingest of missing or new OSA list-driven catalogues is well underway. * ACD noted that he had successfully completed the modifications necessary to support the ingest of GES UVES singleSpec spectra. Following a telecon earlier in the week the iDR4 parameter files were now expected in the next few days, but would contain astrophysical parameters only; versions of the files also containing element abundances would follow later. Once the parameter files were received ACD would begin work on the modifications necessary to ingest them. Hardware and Systems: * MSH reported that a new RAID card had been installed in Djer. Otherwise all the systems were operating nominally. Software: * NJC reported that he had mostly been working on VPHAS. He also mentioned that additional work was needed on automating the multi-epoch surveys, specifically the infrastructure required a few days of work. * RSC noted that the parallelised version of CU19 had been tested in anger, and consequently he had made some improvements to the multi-threaded exception handling (an exercise not dissimilar to nailing jelly to a wall). In part these changes are a consequence of having to make updates to CU19 to deal with some new issues encountered when creating 15+ TByte tables in MS SQL Server. In the end the 2x performance improvement this change has yielded has been lost to a general (albeit variable) average 3x decline in overall performance for unexplained reasons (though the decline has been confirmed not to be directly related to the parallel running of CU19 itself). * ETWS reported that he had: - made some small code updates for the ingest of the OSA catalogues, - finished installing the third-party software on thoth, menkaure and unas. The other servers would be updated during next week. * MAR mentioned that the WSA and SSA had moved to their new server/domains (ssa.roe.ac.uk and wsa.roe.ac.uk) and appeared to be working correctly. * ACD reiterated that the changes to the GES ingest software necessary for iDR4 were continuing. Review of Deadlines: * The review of deadlines was deferred; ACD would check them after the meeting, liaising with individuals as necessary. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * MAR noted that a few tile based, band-merged ATLAS catalogues had been uploaded to ESO for checking. Non-survey Data Release: * None since the last meeting. Astrogrid deployment, VO & Data Analysis services: * Nothing to report; all systems operating nominally. Miscellaneous: * MSH noted that he would be on leave on Thursday and Friday next week (13-14/8/15). RPB added that he would away all next week (10-14/8/15), but would be at home, checking his e-mail and could perform system management tasks if required. ACD, Original: 7/8/15, Revised: 7/8/15.