From nch@roe.ac.uk Fri Apr 26 17:34:05 2013 Date: Fri, 26 Apr 2013 12:51:29 +0100 From: Nigel Hambly To: Eckhard Sutorius , Nicholas Cross , Rob Blake , Ross Collins , Mike Read , Clive Davenhall , Mark Holliman Cc: Lorenzo Rimoldini , Bob Mann , Jim Emerson , Keith Noddle , Norman Gray , Tom Shanks , Stephen Warren , Simon Dye , Stelios Voutsinas , Tom Kerr , Mike Irwin , Andy Lawrence at ROE , Nigel Metcalfe , Maria-Rosa Cioni , Matt Jarvis , James Dunlop , Richard McMahon at IOA , Alastair Edge , Dante Minniti , Phil Lucas Subject: WFAU VDFS Science Archives project meeting minutes, 26/4/13 Present: MAR, ETWS, RGM, NJC, NCH, RSC Apologies: JPE, AL, STV, NG, KTN, MSH, RPB DoNM: 10am Friday May 17th 2013 in the Villas meeting room MAR in the chair (room is booked). Actions discharged this time: ----------------------------- ACTION: NCH to contact Gerry et al to explain our reasoning for wanting access to GES spectra. Discharged; an understanding has been reached and the data copied up. Actions partly discharged but continuing: ----------------------------------------- ACTION: RPB to create and publish neighbours tables between UKIDSS DR10 and SDSS DR9. Continues; ETWS has created the table in the ingest DB, but this needs to be copied into the release DB by taking the latter offline briefly. MAR and ETWS agreed that this was better left to the expert (RPB). Actions carried forward from 15/03/2013 meeting ----------------------------------------------- ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues; this one still has low priority. Specific points and new actions: -------------------------------- Project management: Nothing for the minutes. WFCAM, VISTA and VST updates: Nothing to report. Comments and issues arising from CASU minutes: The team noted the minutes of the meeting of 3rd April; there were no comments Networking: ETWS reported: -- Transfers for VISTA v1.3 data are going well, the latest P90 data and the SV data are transferred and metadata is ingested. -- Transferred the stacked and the nightly final spectra for GES. ETWS noted that the network transfer speeds are holding up well. He also pointed out that a further 3 months of newly process WFCAM data are ready to transfer, but that last week's power outage had taken out the DB ingest server shares, so transfer and ingest is on hold until these are sorted out (besides which, there's another power outage this weekend - see below). WSA/VSA/OSA/GES Operations: ETWS -- Created SDSSDR9 neighbour tables in the WSA. NCH asked that the relevant people get together for a GES conflab at midday today to discuss the way forward now we have all the spectra (intermediate and final stacks). The ATLAS consortium release DB (P89 minus 15 days; including SDSS DR9 crossmatching for what it's worth) should finish later today. Hardware and Systems: MSH reported: "There is another power outage in C2 from tonight through tomorrow afternoon. I'll be shutting down all the servers remotely around 6pm, and Rob will be powering them back on tomorrow afternoon. Also, ramses7 is having an issue with a power supply that should be fixed next week (it had its motherboard replaced this week). And finally, the new NAS should be on order now (160TB formatted storage space)." Regarding the power outage, NCH asked if announcements have been made to the wsa- and vsa-announce lists. ACTION: MAR to announce archive downtime via the wsa- and vsa-announce lists. ETWS noted that venerable curation serve khafre has a degraded RAID array, so all hosted data is being migrated to new storage prior to installation of a fresh disk and reformatting to ease the current flat file congestion. Survey Data Release: ETWS reported: -- The ESO release for VIKING has finished and needs to be checked. -- The ATLASv20130426 release containg data up to 2012-09-15 will finish today. MAR noted that he will push the VIKING ESO release files over once their servers are able to receive the data; NJC noted that VIDEO is next in the queue. Issues raised over file provenance keywords have been sorted out in these latest releases. It was also noted that VIKING and VHS P89 consortium releases have been completed. Software: ETWS noted: -- Updated CU1 for the transfer of the GES spectra. Also enhanced CU1 for a separate multiframeID assignment in FlatFileLookUp to be able to transfer and ingest the VISTA v1.3 data in a more flexible way. RSC noted that he has run some tests of the curation processes on SQL 2012 installed on ramses9; so far, so good. Interestingly, this latest edition of MS's RDBMS contains a column-index store option that may be useful in speeding up queries. RSC noted also that he has been working on an enhanced VSA quality bit to flag saturated sources in VISTA tiles. NJC reported further work on the ESO release helper scripts, and the matched aperture pipeline; MAR noted tweaking the User Interface for a few fixes to enable all the data access functions to work for VISTA private PI programme databases. RGM noted that the new all-signing, all-dancing OmegaCam Science Archive User Interface (to be used for the ATLAS consortium release) is nearly ready. Non-survey Data Release: VISTA project 089.C-0102 has been released to it's PI, following some QC mods. It was agreed that automated QC for all such programmes should be modified to deprecate "C", "D" and "R" ESO QC such that these do not figure in forward curation (but are still released to PIs). Standard crossmatch tables were also discussed. ACTION: NJC to update VISTA auto-QC to deprecate C/D/R flagged frames ACTION: NJC to check the standard crossmatching of non-survey PI programmes for inclusion of WISE (all) and SDSS-DR9 (WFCAM) ACTION: MAR to have a go at releasing other VISTA PI DBs (e.g. for JPE) Astrogrid deployment, VO & Data Analysis services: RGM and NJC asked if it is now possible to quantify the overlap between different surveys (e.g. VST-ATLAS and SDSS DR9) given MSH's recent coverage map enhancements: http://horus.roe.ac.uk/vsa/coverage-maps.html http://surveys.roe.ac.uk/wsa/coverage-maps.html which provide visualisations of survey footprints. It was suggested that maybe Aladin will be enhanced to quantify overlaps. ACTION: All to have a think about how to employ coverage maps to provide quantitative overlap information Miscellaneous: RGM noted that there is some help in the offing for looking into use of column-oriented storage engine MonetDB via Informatic's participation in an NSF scheme on e-Science and large DBs. An astronomy PDRA will be here for 6 weeks from mid-June looking into use of MonetDB for astro apps, and is has been suggested that our large survey DB curation activities, especially those coping with VISTA-VVV, would be a good area to work in. RSC noted that his Wiki description of benchmarking potential bottleneck areas contains a lot of info, and that work on optimal use of the existing engine, MS SQL Server, should be finishing next week. RGM noted that there will be a kick-off conflab via Skype in mid-May. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ============================================================= -- Scanned by iCritical.