From acd@roe.ac.uk Fri Aug 16 11:49:06 2013 Date: Fri, 16 Aug 2013 10:17:17 +0000 From: Clive Davenhall To: Nigel C. Hambly , Eckhard Sutorius , Nicholas Cross , Rob Blake , Ross Collins , "Read, Mike" , Mark S. Holliman Cc: Lorenzo Rimoldini , Bob Mann , Jim Emerson , ktn keithnoddle.org , Norman Gray , Stephen Warren , Tom Shanks , Simon Dye , Stelios Voutsinas , Tom Kerr , Mike Irwin , Andy Lawrence , Nigel Metcalfe , Maria-Rosa Cioni , Alastair Edge , James Dunlop , Matt Jarvis , Richard McMahon at IOA , Dante Minniti , Phil Lucas Subject: WFAU VDFS Science Archives project team meeting minutes, 9/8/2013 WFAU VDFS Science Archives progress meeting minutes, 9 August 2013 Present: RPB, ETWS, MAR, NJC, RSC, ACD Apologies: RGM, MSH, NCH, STV, KTN, AL, JPE DoNM: 10am Friday 23 August 2013 in the Villas meeting room MAR in the chair (room is booked). Actions discharged this time: ---------------------------- ACTION: RPB to investigate maintaining consistency between WFAU and ESO releases. RPB reported:- I found the following: VVVDR1 - both match (all data to the end of P85) VHSDR1 - both match (all data to the end of P85) VMCDR1 - slight discrepancy. WFAU release is to the end of P86 (end of March 2011) while ESO release is to the end of November 2010. However, both are of only two fields. NJC suggested that, as both fields were complete, the difference in date range won't actually affect the data. VIKING and VIDEO are not comparable. WFAU has released a DR2 release for both surveys including data up to the end of P86 (end of March 2011) in both cases. ESO, on the other hand, have not yet released DR2 for either survey. Instead they have VIDEODR1 which only contains data to the end of the Dry Run phase (up to 08/02/2010) while VIKINGDR1 contains most of P86 (up to 13/02/2011) So the ones that are comparable are broadly in agreement, while the ones that aren't comparable aren't comparable. (NOTE ADDED TO MINUTES - RSC subsequently queried: Is there any VIKING data between 13/02/2011 and 31/03/2011 that we actually released in the fields we selected for DR2? If not, then our releases match each other - just there's the naming difference. Also, does this mean we should be making world releases of VVVDR1, VHSDR1, VMCDR1 and VIKINGDR2 now? MAR replied to the second point that: They already are and have been for some time.) Actions partly discharged but continuing: ---------------------------------------- ACTION: MAR to have a go at releasing other VISTA PI DBs (e.g. for JPE) Continues: MAR did not know if the release of 089.D-0113 had yet been successful as it had been causing problems, the release of JPE's project would follow. Actions carried forward from 26/7/2013 meeting: ---------------------------------------------- ACTION: All to think about items that could be included in the WFAU poster. ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues: this one still has low priority. Specific points and new actions: ------------------------------- Project management: * Nothing to report. WFCAM, VISTA and VST updates: * Some good news was reported concerning UKIRT: UK-funded operations have been officially extended until 31/12/2013 pending transfer of the telescope to a new operator. Comments and issues arising from CASU minutes: * Nothing to report. Networking: * Nothing to report. WSA/VSA/OSA/GES Operations: * Most systems were operating nominally. * ETWS reported: -- Catalogue ingest of VISTA v1.3 data has finished and done so much quicker than expected. Now broken ingests need to be checked. -- Transferred and ingested the latest OSA data and the reprocessed fixed catalogues. -- Created new sets of browser pages for the latest GES and VSA releases. -- Created XML metadocs files for DMR for the OSA for TAP/DSA services. * A copy of GALEX[1] GR7 was now installed on the WFUA servers. [1] - http://www.nasa.gov/mission_pages/galex/index.html * NJC noted that the SDSS APOGEE Survey[2] was available and wondered whether we should have a copy. [2] - http://www.sdss3.org/surveys/apogee.php ACTION: NJC to investigate the utility of the APOGEE Survey. Hardware and Systems: * RPB reported: The new database cluster node (ramses14) has been delivered and set up with an experimental RAID10 array to see if we can achieve better throughput for the new VVV v1.3 database. Userkaf (the test server for MonetDB experiments) is having problems with one of its disk arrays and is currently not able to experiment as much as we'd like. * NOTE ADDED TO MINUTES (about an issue discovered shortly after the meeting) RPB circulated the following message: We have a major problem with one of our NAS boxes. Pami, the server that hosts disk27, disk28, disk29, disk30, disk31, disk32, disk33, and disk34, has lost its system volume and is currently offline. So all of those disks will be unavailable until further notice. The good news is that the problem isn't affecting the data disks, so all of our data should be OK once the system disk is fixed. I've has emailed eclipse to get them to come round. Mike will ensure that they get access in the absence of me and Mark. Note that as well as the pixel data hosted on these disks being unavailable the temporary loss of disk34 will hamper a whole bunch of curation and development activities as it hosts our codes' common work space and reference files. Survey Data Release: * ACD reported that an initial GES iDR1 release had occurred. This release had been made somewhat prematurely to (almost) meet the scheduled deadline. The atomic line list was not present and the documentation had not been updated. These issues will be addressed shortly. * The UKIDSS DR9 release became public on 8 August 2013. Software: * NJC reported that he had continued to tidy the list-driven photometry software. It was now largely working. However, further tests of dual image mode and using it with multiple surveys were needed. * ETWS reported: -- Wrote helper scripts to check ingests and to be able to re-ingest new fixed catalogue data into the OSA where unfixed data already exists. * RSC reported that the GES software had been updated to support ingest of iDR2 spectra, final homogenised iDR1 parameters & abundances, and the latest complete line list data (molecular and atomic) all to a new schema for the new data. Non-survey Data Release: * There was an outstanding problem with a VISTA non-survey release, which had failed. Investigation of this problem had been upstaged by the VMC release, which was more urgent. However, the problem still needs to be addressed. ACTION RPB and NJC: RPB to check the logs, identify the error and forward to NJC, who would investigate and fix it. Astrogrid deployment, VO & Data Analysis services: * RSC reported that the MonetDB project came to its scheduled end last week and has some promising results despite suffering numerous delays through hardware failures. Only the small table tests were completed but they revealed that MonetDB was around 10x faster, both for ingests and SQL-driven row-by-row updates. Since the amount of data was less than the system memory these results should be taken with a pinch of salt, but it was a fair test and the system monitoring tools revealed exactly how MonetDB was optimised to produce such good performance. Maria will be continuing to work on this project in her spare time as she intends to prepare a paper on these results for a forthcoming conference this November. However, the latest hardware failure has not yet been fixed. Miscellaneous: * Nothing to report. ACD, draft: 9/8/13, completed: 16/8/13. -- Scanned by iCritical.