From mar@roe.ac.uk Thu May 29 18:27:30 2014 Date: Thu, 29 May 2014 17:54:52 +0100 From: Mike Read To: VDFS Science Archive Team -- Clive Davenhall , Dave Morris , Eckhard Sutorius , Mark Holliman , Nicholas Cross , Nigel Hambly , Rob Blake , Ross Collins , Stelios Voutsinas Cc: CCs for VDFS -- Alastair Edge , Andrew Lawrence , Bob Mann , Dante Minniti , James Dunlop , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Maria-Rosa Cioni , matt.jarvis@astro.ox.ac.uk, Mike Irwin , Nigel Metcalfe , Norman Gray , Phil Lucas , Richard McMahon at IOA , Simon Dye , Stephen Warren , Tom Kerr , Tom Shanks Subject: WFAU VDFS Science Archives progress meeting minutes, 23 May 2014 WFAU VDFS Science Archives progress meeting minutes, 23 May 2014 Present: MAR, RPB, RSC, ETWS, NJC, RGM Apologies: NCH, KTN, AL, MSH, ACD DoNM: 10am Friday 6 June 2014 in the Villas meeting room, ACD in the chair (room is booked). Actions discharged this time: ---------------------------- ACTION: MAR to look at the requirements and timescales for the next batch of ESO releases. Done, VHS and VIKING P91 releases are underway and these would form the basis of their ESO submissions. VMC wanted a release based on 7 pointings with some new deprecations added in. This would be run as soon as the VHS/VIKING databases had been generated. The deadline is 15 June, this is ambitious especially given that various folk are on leave during the first half of June. It was still not known exactly what VVV would be supplying, lists of v1.3 deprecated files had been sent to Phil Lucas so that he could let CASU know what pipeline files to upload, but as we are still a long way off generating a database release we are not really in a position to supply additional data products (eg band merged cats). Actions partly discharged but continuing: ---------------------------------------- None. Actions carried forward from 9/5/2014 meeting: ---------------------------------------------- ACTION: ACD, RPB to formulate a policy for backups and for checking that data could be recovered from the backups. Continues. Notes on a proposed policy are available on the Twiki. No decision had yet been made on backing up the SVN repository. ACTION: RPB to put all prices of additional items associated with the backup policy on the Twiki pages. Continues. ACTION: RGM to investigate the size and content (how does it differ from DR9?) of DR10 to see if it's worth us hosting and whether it is necessary to keep all of the other releases. Continues. RGM had started a wiki page for WFAU SDSS strategy (http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan). ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues: this one still has low priority. ACTION: MAR to document issue with ATLAS unpaired sources and give examples of recovering matches. Continues. ACTION: ETWS to investigate modifying post CU4 ops to calculate and store illumination correction values. Continues. ACTION: ACD to arrange follow-up VVV brainstorming meeting. Continues Specific points and new actions: ------------------------------- Project management: RGM noted that the previous week's meeting with Mike Irwin to discuss VST ATLAS and WFCAM and the subsequent telecon with Durham (VST ATLAS) had been productive. Notes were on the Wiki. WFCAM, VISTA and VST updates: Attempts to reduce scattered light in VST continue, with further modifications to the baffling scheduled for June/July. Comments and issues arising from CASU minutes: No minutes had been received since the previous meeting. Review of Deadlines: Those present went over the relevant entries on http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines RSC noted that he has been updating the GES release deadline as the ingest throws up new issues. Networking: Working. The current ATLAS list driven files had been transferred. WSA/VSA/OSA/GES Operations: RPB reported that following the VMC meeting, a neighbour table had been created with 3XMMDR4. The VVV objID update had stalled and become unresponsive. An SQL server re-start had been required which had led to a rollback of the update. Unfortunately the rollback was estimated to take around 60 days. RPB suggested that there must be a better way of maintaining unique objID, eg creating unique IDs at the time of ingest. ACTION: RPB to create wiki page to discuss way of generating unique objIDs. Hardware and Systems: RSC reported: A repaired Nytro server was returned to us, so we could test it out as a catalogue server until its Nytro RAID array lone expired on Wednesday 14th. A mock, sub-set VVV ingest database was copied to Nytro and test release performance runs, mostly testing outgest performance, were eventually started after a lot of team work to get it all set up. VVV performance testing on with the database hosted on Nytro disappointingly didn't reveal faster than average outgest performance (though this is consistent with what others have found), but none of the previously predictable outlier slow outgests occurred, which may just be an effect of using a fresh mock db for testing. More tests on the mock db on a benchmark server would be required. To benefit from the 10GigE connection on Nytro we prepared the test release database on ramses12, which surprisingly offered us 10% faster ingest performance over ramses13 - still not sure why. The mock sub-set DB was then transferred off Nytro for safe keeping. Software: ETWS noted he'd been working on curating the ATLAS list driven catalogues and ingesting the night of WFCAM satellite data. He also mentioned that STARLINK required an upgrade and CFITSIO a downgrade to enable some parts of the curation codebase to run. ACTION: RPB to upgrade STARLINK on curation servers. RSC reported: Prior to the return of Nytro, we completed the initial set of simple CU19 modifications to test VVV performance configuration options, with a test of ingests to a simple heap (a table without primary key, as we do for CU16). This revealed a 30% performance improvement, but more than the total time saved on ingest was lost when attaching the primary key after all ingests were completed, and it's also a higher risk strategy, so it's clearly unsuitable in this case. GES ingest script was updated for the expected changes to the data model for the new iDR2, but testing as always resulted in 99% of the work, as the iDR2 data had many new surprises in-store for us as always (changing data types, use of position to identify stars with high proper motion, new undefined nodes, child analyses without parents: nodes analysing spectra not considered by their parent working group). testing. More tests on the mock db on a benchmark server would be required. Clive completed work on a script to automatically generate a new copy of the GES working group FITS files that can be more easily described by our schema to allow for simple schema-driven ingest. He's also provided some additional user friendly views of AstroAnalysis and SpectrumAndFrame as well as updates to the schema requested by Cambridge. Survey Data Release: P91 release database had been created for ATLAS and VHS but space needed to be found/created on the public servers to host them. ACTION: RPB/ACD/MSH to see if a new database node could be ordered. RSC noted that VVV DR3 will not happen this year due to the delay in the objID updates, but should still be on course for early next year Non-survey Data Release: Currently in the process of trying a non-survey combined release of U/12B/H20B and U/13A/H28B, the latest attempt crashing with a segmentation fault. Astrogrid deployment, VO & Data Analysis services: Nothing to report. Miscellaneous: Nothing to report -- Scanned by iCritical.