From mar@roe.ac.uk Mon Feb 22 16:31:04 2016 Date: Mon, 22 Feb 2016 16:30:56 +0000 From: Mike Read To: VDFS Science Archive Team -- Clive Davenhall , dmr@roe.ac.uk, Eckhard Sutorius , Mark Holliman , Nicholas Cross , Nigel Hambly , Rob Blake , Ross Collins , Stelios Voutsinas Cc: CCs for VDFS -- Alastair Edge , Andrew Lawrence , Bob Mann , Dante Minniti , James Dunlop , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Maria-Rosa Cioni , matt.jarvis@astro.ox.ac.uk, Mike Irwin , Nigel Metcalfe , Norman Gray , Phil Lucas , Richard McMahon at IOA , Simon Dye , Stephen Warren , Tom Kerr , Tom Shanks Subject: WFAU VDFS Science Archives progress meeting minutes, 19 February 2016 WFAU VDFS Science Archives progress meeting minutes, 19 February 2016 Present: MAR, RPB, MSH, NJC, ETWS, RSC, RGM, ACD Apologies: STV, DMR, KTN, AL DoNM: 10am Friday 4th March 2016 in the Crawfoed meeting room 1, ACD in the chair (room is booked). Actions discharged this time: ---------------------------- ACTION: DMR, STV MSH to liaise over setting up a test registry TAP service. Done. Actions carried forward from 19/2/2016 meeting: --------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues: this action still has low priority. ACTION: NJC noted that VISTA mosaicing crashes with normal filtering, which he should fix. Continues. NJC had implemented a fix. Though it completed successfully when he ran some tests it crashed when run 'in anger' on a complete ingest dataset. NJC should revisit and correct his fix. ACTION: NJC noted that he had received a report of a problem with the ESO VVV DR1 release which he should investigate. In one pointing the merged band catalogue contained fewer sources than expected. Continues. This action was very low on NJC's priority list and unlikely to be implemented in the foreseeable future. ACTION (evolved from previous action): NJC register for the VMC meeting to be held at Keele University during June 2016. NJC had emailed Maria-Rosa but not yet registered. Continues. ACTION: DMR, NJC, RPB ensure that the new-style OSA release has access to all the database tables that it needs; specifically resolve the problems encountered with neighbour tables. Continues. ACTION: MSH, DMR install and configure the Nagios[1] system monitoring package to monitor the availability of our database services. Continues. ACTION: NJC to fix the errors encountered while trying to make non-survey releases U14BUA15, U14BUA16, U14BUA18 and U14BUA19. Continues. Actions resulting from the VVV DR3 brainstorming meeting held on 24/2/15: ------------------------------------------------------------------------ ACTION: RPB to experiment with ingest via partitioned tables, along the lines of the SSIS paper. Deferred (changed back from 'continues'). This action has now been overtaken by events: the VVV DR4 (was DR3) processing would now be done differently. However, the technique was still potentially useful for future work and should be investigated before the next VVV release, which is time away. ACTION: RGM (was RSC, RGM): (i) RSC to write a succinct summary of the ingest and outgest problem that is the principal bottleneck with the VVV processing (briefly the ingest and outgest speed is limited to approx. 100 Mbit/sec for no apparent reason) and (ii) RGM to compose an e-mail message around this summary and send it to Ani Thakar. Continues. RSC had composed a succinct summary and sent it to RGM. RGM had emailed Ani Thakar but no reply to date. RGM had sent a reminder but had still received no response. RGM had also contacted Alex Szalay and not received a reply from him either. He would consider how to proceed. Specific points and new actions: -------------------------------- Project management: * Nothing for the minutes. WFCAM, VISTA and VST updates: * Nothing to report. Networking: * Nothing to report; everything appeared to be working nominally. WSA/VSA/OSA/GES Operations: * ETWS reported: - OSA: ingest of the uncorrected data for P93, P94 and P95 is complete, illumination corrections are being applied. - VVV: ingesting (CU4) P94 data - VSA (other): ingested up to end of October 2015. - WSA: ingested up to end of August 2015. * ACD reported the GES iDR4 parameter ingest was complete but work was still needed on the nightly files. RSC mentioned an issue around the documentation of neighbour schemas and querying neighbour table whereby it was not readily obvious which data release / database should be used. ACTION: NJC to look at improving neighbour table documentation. Hardware and Systems: It wasn't clear if the C2 offical handover had taken place, certainly MSH and RPB had not been involved or received any instruction. There were some H&S concerns over one of the doors only opening from the inside. ACTION: MSH to right a report on the C2 works. * MSH reported: A dead battery on ramses16 had been replaced but was still having issues, the power supply or RAID controller itself were thought to be the underlying cause. Ramses9 had also suffered from a dead battery and dead disk. The new KVM switch had been ordered. Several problems had arisen with the cross-mounting of databases. This could be being caused by the patch panels now employed in C2. Further investigation was needed. Software: * NJC had worked on VMC frameset association and ordering of passbands as per requests from Maria-Rosa. He'd also been looking into problem with the latest VVV synoptic source table as well as working on the DXS mosaicing. * ETWS had worked on fixing up headers of WSA files on a particular day when they came through with some missing attributes. Review of Deadlines: * Progress towards forthcoming deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * The VMC P95 release was running and would take about two weeks to complete. Non-survey Data Release: * None. Astrogrid deployment, VO & Data Analysis services: * Nothing to report. Miscellaneous: * RGM asked NJC if he would update the Twiki page on future VISTA surveys. ACTION: NJC to update Twiki page on future VISTA surveys. * ACD reminded those present of the local meetings he'd recently circulated details of (eg data management).