From mar@roe.ac.uk Fri Apr 19 15:50:07 2013 Date: Thu, 18 Apr 2013 23:14:41 +0100 From: Mike Read To: VDFS Science Archive Team -- Clive Davenhall , Eckhard Sutorius , Mark Holliman , Nicholas Cross , Nigel Hambly , Rob Blake , Ross Collins , Stelios Voutsinas Cc: CCs for VDFS -- Alastair Edge , Andrew Lawrence , Bob Mann , Dante Minniti , James Dunlop , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Maria-Rosa Cioni , Matt Jarvis , Mike Irwin , Nigel Metcalfe , Norman Gray , Phil Lucas , Richard McMahon at IOA , Simon Dye , Stephen Warren , Tom Kerr , Tom Shanks Subject: WFAU VDFS Science Archives project meeting, 12/4/2013 Present: MAR, RPB, ETWS, RGM, NJC, NCH, RSC, MSH Apologies: JPE, AL, STV, NG, KTN DoNM: 10am Friday April 26th 2013 in the Villas meeting room NCH in the chair (room is booked). Actions discharged this time: ----------------------------- ACTION: MAR to supply a list of non-survey projects to be released. Discharged, 4 WFCAM projects from semesters 11 A/B made it through the release procedure. ACTION: ETWS to contact EGS to get details of the versioning differences V1.0-1.3 in Vista processed products Discharged, the plan is to do a wholesale re-transfer/re-ingest, see later. Actions partly discharged but continuing: ----------------------------------------- None this time. Actions carried forward from 15/03/2013 meeting ----------------------------------------------- ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues; this one still has low priority. Specific points and new actions: -------------------------------- Project management: Nothing for the minutes. WFCAM, VISTA and VST updates: Nothing to report. Comments and issues arising from CASU minutes: No new minutes this time. Networking: Working and transfers up-to-date (apart from VISTA v1.3!) WSA/VSA/OSA/GES Operations: The small matter of moving to v1.3 for VISTA data was discussed. Various scenarios were considered (header updates / header transfers for some files etc). ETWS estimated some 100Tb was required to hold the full v1.3 products. RGM noted that there was more money available for hardware than previously thought, so in the end it was decided to just transfer the whole lot up again and ingest into a new load database. On the VSA (non-VVV) load server existing operations could run in parallel with the new ingests etc. However space restrictions on the VVV load server would require a freeze at some suitable point (DR2 release?) followed by a backup and delete before ingesting VVV v1.3. ETWS reported: -- To release some disk space for the VSA v1.3 we will delete v0/v0.6/v0.8/v1.0 data, which will give us ~49TB. NCH asked about the SDSS DR9 installation, RPB replied that it was now available and agreed to retrospectively add its neighbour tables in to UKIDSS DR10. ACTION: RPB to create and publish neighbours tables between UKIDSS DR10 and SDSS DR9. Hardware and Systems: MSH reported that he'd been working on the wiki documentation for our hardware systems and services. He also reminded everyone of the planned server room downtime for the weekends 19th-21st and 26th-28th April. A new NAS box was being ordered (0.2 Pb raw storage). The Ramses9 test DB server was now running SQL Server 2012. Its tempdb filesystem was housed on SSDs (900 Gb). Survey Data Release: The VVV P87 release is on hold whilst investigations are carried out on speeding up detection table outgests. VMC - an ad hoc deep stack, excluding a Ks tile with poor ellipticity, was provided to Maria-Rosa. VHS - rumours of the P89 release in the last minutes were somewhat premature but it was now nearly ready, honest. VIKING - the P89 release was running. ATLAS - QC up to 15th September 2012 (date matches required ESO release) had been applied and the release process could be started. VPHAS - NJC noted that, in looking at the observations a bit more, incorporating VPHAS might not be as straight forward as first thought. The data mainly consist of normal frames and catalogues i.e. no stacks. NJC would investigate further to see what we could readily provide given limited resources. GES - a beta release had been made available to a subset of the consortium. There was still some confusion over what GES data products we could get and when. Specifically the spectra would be very useful in shaking down various processes in the archive and understanding the metadata and WG supplied content and how they relate to each other. ACTION: NCH to contact Gerry et al to explain our reasoning for wanting access to GES spectra. Software: NJC noted continued work on the archive-end list-driven photometry process and ESO release code. RSC noted a modification to the code so as not to apply saturation corrections to tiles. ETWS reported: -- Minor fixes to the parser for the OSA browser pages. -- Updated the browser parser for the GES browser pages and created new pages for ges201303beta. It was agreed that we should move to Python 2.74 but should hold off upgrading pyFITS. Non-survey Data Release: Databases had been released for 4 WFCAM non-survey projects. Iain McDonald (Manchester) had been in touch re concerns over tile patterns in VISTA mag vs 2MASS mag plots and whether he could have v1.3 files. With Mike Irwin's help we managed to confirm that he essentially had v1.3 data, the differences were as expected given the various errors present and to do better he should use pawprints. Attempts to run the automated non-survey QC on the VSA had resulted in unexpected deprecations for survey data, these had been reset. Project 089.C-0102 would be manually QC'ed to expedite its release. Astrogrid deployment, VO & Data Analysis services: Nothing of note. Miscellaneous: NJC noted ongoing discussions on archiving the Herschel Redshift Survey (HeRS). -- Scanned by iCritical.