From acd@roe.ac.uk Fri Jul 17 12:07:00 2015 Date: Wed, 15 Jul 2015 15:43:40 +0000 From: Clive Davenhall To: Bob Mann , Andy Lawrence , "Read, Mike" , Eckhard Sutorius , Mark S. Holliman , Nicholas Cross , Nigel C. Hambly , Rob Blake , Ross Collins , Stelios Voutsinas , Dave Morris , Clive Davenhall Cc: "dante@astro.puc.cl" , James Dunlop , "j.p.emerson@qmul.ac.uk" , ktn keithnoddle.org , "Lorenzo.Rimoldini@unige.ch" , "mcioni@aip.de" , "matt.jarvis@astro.ox.ac.uk" , "mike@ast.cam.ac.uk" , "nigel.metcalfe@durham.ac.uk" , "norman@astro.gla.ac.uk" , "pwl@star.herts.ac.uk" , "rgm@ast.cam.ac.uk" , "Simon.Dye@nottingham.ac.uk" , "s.j.warren@ic.ac.uk" , "tkerr@jach.hawaii.edu" , "tom.shanks@durham.ac.uk" Subject: WFAU VDFS Science Archives progress meeting minutes, 3 July 2015 WFAU VDFS Science Archives progress meeting minutes, 3 July 2015 Present: RGM, ETWS, MSH, NJC, RPB, RSC, ACD Apologies: MAR, STV, KTN, AL, DoNM: 10am Friday 17 July 2015 in the Villas meeting room, MAR in the chair (room is booked). Actions discharged this time: ----------------------------- ACTION: RSC/ETWS to see if we can get another mxODBC trial period set-up as the current one had expired (was RSC to test the new version of mxODBC). Done. ETWS now had the requisite details to hand. ACTION: NJC to produce a roadmap for future VSA releases in order to plan for hardware requirements. Done. The roadmap included all anticipated releases. ACTION: NJC to speak to the VVV PIs, notably Dante, about the schedule for and expected number of remaining VVV releases. Done. ACTION: RPB to backup VSAVVV and copy to ramses16. Done. Actions carried forward from 24/4/2015 meeting: ----------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues: this action still has low priority. NEW ACTION: ACD to obtain the details of obtaining an mxODBC licence from ETWS and progress obtaining a sufficient number of such licences. ACTION: NJC noted that VISTA mosaicing crashes with normal filtering, which he should fix. Continues. NJC had implemented a fix and checked it into SVN. However, when it was run 'in anger' it still crashed. NJC to investigate. ACTION: RPB to compile a Web page listing all the completed releases. Continues. RPB had made a start compiling this list. However, the exercise revealed that a number of databases had been misplaced and he was investigating their whereabouts. ACTION: NJC noted that he had received a report of a problem with the ESO VVV DR1 release which he should investigate. In one pointing the merged band catalogue contained fewer sources than expected. Continues. This action was low on NJC's priority list. Actions resulting from the VVV DR3 brainstorming meeting held on 24/2/15: ------------------------------------------------------------------------ ACTION: RPB to experiment with ingest via partitioned tables, along the lines of the SSIS paper. Deferred. RPB was investigating various options and commencement of work on the present action would depend on the results of these investigations. ACTION: RGM (was RSC, RGM): (i) RSC to write a succinct summary of the ingest and outgest problem that is the principal bottleneck with the VVV processing (briefly the ingest and outgest speed is limited to approx. 100 Mbit/sec for no apparent reason) and (ii) RGM to compose an e-mail message around this summary and send it to Ani Thakar. Continues. RSC had composed a succinct summary and sent it to RGM. RGM had emailed Ani Thakar but no reply to date. RGM had sent a reminder but had still received no response. ACTION: NJC to remove dependencies on old libraries. Continues. This was mainly done but NJC to check on what if anything was still needed for VOTable handling. Specific points and new actions: -------------------------------- Project management: * Nothing to report. WFCAM, VISTA and VST updates: * Nothing to report; all the telescopes were believed to be operating nominally. Networking: * Nothing to report; the networks were performing nominally. WSA/VSA/OSA/GES Operations: * ETWS reported that he had: - Processed and ingested the metadata for the reprocessed science and new list-driven files for the OSA. The catalogue data ingests are nearly finished, although quite a lot of the catalogues had missing or wrong zero-point information due to a hiccup at CASU and need to be re-transferred. - Fully processed and ingested the Jan. 2015 data for VISTA. - Finished the objID update on the VVV which is now also up-to-date to Jan. 2015. ACTION RPB: for VVV investigate whether the way in which extractions are allocated to machines could be rejigged to improve performance. ACTION NJC: check that CU19 is outgesting a specific list from TilePawprints' tables. Hardware and Systems: * MSH noted that: - the University was planning to put a firewall on the SRIF network, - he had started negotiations with the Lockheed Martin sys admins, - the Redmine repository was down following a disk failure. * The feasibility of WFAU locating machines at the University's Easter Bush campus was discussed: it was possible but would require network upgrades in JCMB. Software: * NJC reported: - recent changes would nearly bring the older surveys into line with the soon to be implemented ones in terms of deep stacking, - he had started designing schemas to hold the tables needed by Carlos Ferreira Lopes' technique to automatically detect variability. * RSC noted that he had released a parallelised version of CU19. He had not tested it and was reluctant to do so as he feared that once started it would be difficult to stop (which is a feature of multi-threaded python code). * ETWS reported that he had: - Fixed some minor issues with the deprecation and ingest of the re-processed list-driven data (this code had not been used in anger before). - Started pruning the WSA-LM software branch to only include the relevant code for satellite data processing. - Installed the latest third party software on niuserre to help MAR to run the ColourJpegger code. The new software needed only very small tweaks to be successfully adapted to a new server after being frozen on horus for seven years. Review of Deadlines: * Progress towards forthcoming deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * None since the last meeting. Non-survey Data Release: * None since the last meeting. Astrogrid deployment, VO & Data Analysis services: * Nothing to report; all systems operating nominally. Miscellaneous: * RPB noted that he had recently attended a VMC meeting in Naples. * NJC was preparing a document about ingesting VVV PSF photometry and difference imaging and integrating it into the existing VSA data. * NJC would be giving a talk at NAM. * RGM would shortly be e-mailing a draft schedule for the half-yearly reviews. ACD, Original: 6/7/15, Revised: 15/7/15.