From acd@roe.ac.uk Wed Mar 26 16:02:45 2014 Date: Wed, 26 Mar 2014 16:01:24 +0000 From: Clive Davenhall To: rgm@roe.ac.uk, al@roe.ac.uk, mar@roe.ac.uk, etws@roe.ac.uk, msh@roe.ac.uk, njc@roe.ac.uk, nch@roe.ac.uk, rpb@roe.ac.uk, rsc@roe.ac.uk, stv@roe.ac.uk, dmr@roe.ac.uk, acd@roe.ac.uk Cc: dante@astro.puc.cl, jsd@roe.ac.uk, j.p.emerson@qmul.ac.uk, keith@keithnoddle.org, Lorenzo.Rimoldini@unige.ch, mcioni@aip.de, matt.jarvis@astro.ox.ac.uk, mike@ast.cam.ac.uk, nigel.metcalfe@durham.ac.uk, norman@astro.gla.ac.uk, pwl@star.herts.ac.uk, rgm@ast.cam.ac.uk, Simon.Dye@nottingham.ac.uk, s.j.warren@ic.ac.uk, tkerr@jach.hawaii.edu, tom.shanks@durham.ac.uk Subject: WFAU VDFS Science Archives progress meeting minutes, 14 March 2014 WFAU VDFS Science Archives progress meeting minutes, 14 March 2014 Present: RGM, NJC, MAR, RSC, MSH, ETWS, RPB, ACD Apologies: NCH, KTN, AL DoNM: 10am Friday 28 March 2014 in the Villas meeting room, MAR in the chair (room is booked). Note: in a change of venue the meeting was held in Crawford MR2. Actions discharged this time: ---------------------------- ACTION: NJC to perform a quick check on the VMC public release. Done. ACTION: RSC to set a date for completing the actions from the brainstorming meeting held shortly before Christmas. (This action is in preparation for the next brainstorming meeting, which will be arranged once the VVV release is complete.) Discharged. The target date for completion is around 30 April, so that significant progress may be reported at the next WFAU Quarterly Meeting, and in time for the optimisations to be in place for the CU6 run for VVV DR3 (which will happen about 1-2 months after the completion of DR3 CU4 ingests). ACTION: NJC to remind Omar Almaini about the UKIDSS DR11 final data release. Deleted: no longer relevant. Actions partly discharged but continuing: ---------------------------------------- None. Actions carried forward from 28/2/2014 meeting: ---------------------------------------------- ACTION: ACD, RPB to formulate a policy for backups and for checking that data could be recovered from the backups. Continues. Notes on a proposed policy are available on the Twiki and discussion is still in progress. NEW ACTION: RPB to put all prices of additional items associated with the policy on the Twiki pages. NEW ACTION: All to read the Twiki pages: http://apache.roe.ac.uk/twiki/bin/view/WFAU/BackupReview ACTION: RSC to investigate modifying CU7 to recover ATLAS VST unmatched sources around frameset boundaries. Continues. Investigations had begun and a strategy had been worked out. However, this action now has lower priority. Nonetheless a fix should be put in place before the ATLAS meeting on 14-16 April and it should be followed by a consortium release shortly after the meeting. ACTION: ALL to add/maintain entries on http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Continues. ACTION: ACD to discuss with RGM the available budget for hardware purchases. Continues. ACTION: RGM to investigate the size and content (how does it differ from DR9?) of DR10 to see if it's worth us hosting and whether it is necessary to keep all of the other releases. Continues. RGM had started a wiki page for WFAU SDSS strategy (http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan). However, it needs more work and in particular RGM still had to determine which WFAU archives had cross-matches with which SDSS releases. ACTION: NJC to sort out the magnitude limits for V1.3 data. Continues. The software was largely written, but NJC had not had the time to finish and test it. ACTION: ALL: make sure that all forthcoming absences during the next couple of months are entered into the 'staff absences' Wiki page. Continues: this information is needed for planning next WFAU quarterly, which is likely to be in late April. ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues: this one still has low priority. Specific points and new actions: ------------------------------- Project management: * RGM reported on progress with the one-year extension to the WFAU Rolling Grant and UKIRT work. ACTION: RGM to check with AL on progress of negotiations with the new owners of UKIRT. WFCAM, VISTA and VST updates: * Nothing to report. Comments and issues arising from CASU minutes: * Nothing to report. Networking: * The networks continued to work optimally. ETWS reported that he had transferred data as follows: - VISTA: Nov. 2013 - Jan. 2014, - WFCAM: Oct. - Dec. 2013, - VST: Oct. 2013 - Jan. 2014. WSA/VSA/OSA/GES Operations: * ETWS reported that metadata had been ingested and JPGs created for: - VISTA: Nov. 2013 - Jan. 2014, - WFCAM: Oct. - Dec. 2013, - VST: Oct. 2013 - Jan. 2014. * NJC reported he had been preparing a VISTA release for ESO. The VMC and VIDEO releases had been prepared but not yet uploaded. The VVV release was not quite complete; some additional data were required from CASU. * ACD noted that the GES iDR2 astrophysical analysis results were expected imminently, though some of the data would be dummy. Hardware and Systems: * MSH reported that there had been an air conditioning meltdown in C2 (the machine room), but luckily all our equipment survived. Discussions will be opened with the IT group to be sure the problem is a one-off. ACTION: MSH to e-mail RGM suggested improvements to C2. * Ini hung rather unexpectedly in the middle of VVV operations. A hard reboot fixed the problem. Since then its OS has been upgraded to Debian Jessie like all the other servers. * A new NAS box and processing server have been ordered and are expected soonish. * A new test system from Eclipse had arrived. It contains an LSI Nytro RAID card (a fancy piece of kit meant to boost disk I/O). It will be tested as both a processing platform (Linux) and then as a database server (Windows) to see if the added performance is worth the cost in either role. * There will be a short visit from Viglen on Thr. 20/3/14. Survey Data Release: * RPB reported that the VHS P91 release is having CU19 run to create the release database. The detection table is very large and took a long time to outgest. CU19 is now creating indices, which are taking roughly 2 days each to complete. * VMC P91 release has just started. It is currently applying quality bit flags, before progessing to the tile/paw print stuff. Software: * NJC reported that he had mainly been working on the ESO release and that he had made substantial changes, particularly to tracking the provenance of multi-epech tables. There were a few other changes, for example to calculating zero points. * ETWS noted that for the VVV ingests he had plotted the number of ingested detections against ingest time (and showed examples of the plots, which caused considerable interest and discussion). These plots showed two distinct branches; one linear comprising quick ingests independent of the number of detections and the other consisting of slow ingests above a threshhold of 4*10^6 detections. Consequently he has decided to rewrite CU4 to split up the large ingest files into smaller ones with a maximum size of 4*10^6 rows. Non-survey Data Release: * No non-survey releases had been prepared. One user had expressed an interest in non-sky subtracted files, but this was an issue for CASU rather than WFAU and they were looking into it. * The WFCAM open time database has been updated. Astrogrid deployment, VO & Data Analysis services: * MSH noted that TAP services had been deployed for all the latest VISTA releases (VMC DR2, VHS DR2, VIKING DR3, VIDEO DR3), and the VISTA SIAP service has been updated to include these as well. * Coverage maps for all these releases have been generated and uploaded to the WSA, VSA, and OSA Web sites. Miscellaneous: * Nothing to report. ACD, draft: 17/3/14, completed: 26/3/14. -- Scanned by iCritical.