From nch@roe.ac.uk Sat Jan 26 14:20:17 2013 Date: Fri, 25 Jan 2013 16:35:28 +0000 From: Nigel Hambly To: Nicholas Cross , Eckhard Sutorius , Rob Blake , Ross Collins , Mike Read , Clive Davenhall , Mark Holliman Cc: Lorenzo Rimoldini , Bob Mann , Jim Emerson , Keith Noddle , Norman Gray , Tom Shanks , Stephen Warren , Simon Dye , Stelios Voutsinas , Tom Kerr , Mike Irwin , Andy Lawrence at ROE , Nigel Metcalfe , Richard McMahon at IOA , Matt Jarvis , James Dunlop , Alastair Edge , Maria-Rosa Cioni , Dante Minniti , Phil Lucas Subject: WFAU VDFS Science Archives project team meeting minutes, 25 Jan 2013 Present: MAR, RSC, RPB, ETWS, NJC, MSH, NCH, RGM Apologies: JPE, AL, STV, NG, KTN DoNM: 10am Friday February 11th 2013 in the Villas meeting room MAR in the chair (room is booked). Actions discharged this time: ----------------------------- ACTION: MSH, with input from ETWS, to estimate hardware requirements, timescales and costs up to end March 2014. Discharged; it was noted that this includes a new infiniband switch. Actions partly discharged but continuing: ----------------------------------------- ACTION: ETWS (with help from NJC as necessary) to add FeII 1.644 micron filter details into the WSA. Continues; all done apart from Vega-to-AB and zeropoint attributes which can now be done by NJC as sufficient data have been ingested Actions carried forward from 07/01/2013 meeting ----------------------------------------------- ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues; this one still has low priority. Specific points and new actions: -------------------------------- Project management: NCH noted that JPE has asked that more frequent and detailed communication be established with the VISTA Public Survey PIs, so the suggestion is that these (approximately) fortnightly minutes get circulated to them. In this way it is hoped that the PIs will be more informed about progress on survey releases. (PIs: please feel free to ask to be excluded from these mail shots if you find them unhelpful). RGM noted that, in agreement with STFC, the funding WFAU for staff resources will continue via roll--out of the current grant supplemented by a bridging grant for hardware (the current grant proposal review is on hold pending the outcome of STFC's Programmatic Review). This secures funding until the end of March 2014. Let the good times roll. WFCAM, VISTA and VST updates: UHS news: the proposal to observe UHS at K band was not awarded time by the Board; however RGM noted that a mini-survey of the Herschel-NGP field (an area not covered by VIKING) is to go ahead and the PI is keen to take advantage of VDFS processing/archiving. Regarding the future of UKIRT, no-one present was aware of any updates on cessation of operations. NJC noted that closer integration of VIKING-KIDS is under discussion amongst the various PIs and survey groups, and this may impact on what is required in the VSA/OSA. Comments and issues arising from CASU minutes: No new minutes this time. Networking: MAR noted that the possible network outage downstream in Edinburgh University has been flagged on the various archive web pages. ETWS noted that since all transfers from CASU are presently up-to-date (see below) this will not affect any archive transfers. WSA/VSA/OSA/GES Operations: ETWS reported: "-- Finally managed to get the backlog due to the disk38 failure reduced. VST metadata is ingested up to December 2012, catalogues are ingested up to August 2012. VISTA metadata is ingested up to October 2012, non-VVV catalogues up to August 2012. This means P89 ingests will be finished next week. -- WFCAM metadata as well as catalogues are ingested up to October 2012. -- I've asked MJI about the latest and planned data updates. This is his reply: VISTA - new zero-point for pawprints and tiles the rest is just catalogue updates to put the whole system on v1.3 this includes regrouting all tile catalogues up to end of 2011 VST - its going to be zero-point updates only at this stage WFCAM - we did a big update about a year ago for all zero-points and catalogue aperture corrections to end of 2011. Unfortunately a similar minor update to aperture corrections and zero-points is needed for most of 2012 data Since the directories at CASU still contain the old version numbers, I guess it might be some time until we need to apply fixes to the VISTA data. " It was noted that when the reprocessing fixes are done at the archive end, it will consist of header updates to FITS files and then propagation of the photometric changes using curation script CU8 which has been used previously for similar WFCAM updates. RPB noted that catalogue ingests for stacked products generated for the VVV is currently running (in preparation for the P87 consortium release). NCH noted that CASU has indicated that a bulk GES data release will occur within the next week or two for ingest into the Gaia-ESO Science Archive in preparation for a consortium beta-testing release. Hardware and Systems: MSH noted that our oldest NAS box (12TB) is on the fritz with degraded RAID arrays and an apparent inability to sort itself out. Data holdings are currently being copied to a newer store and then the errant device will be hammered back into submission. ETWS noted running further tests on the data copied back into disk38 after the NRAID (that's non-redundant array ...) snafu last year. Survey Data Release: UKIDSS DR10 was released and announced to the world (well, the usual email distribution lists) on Jan 14th. RPB noted that SDSS DR9 is currently still in the process of unpacking/installing on the archive servers, so users will have to content themselves with SDSS DR8 crossmatches for the time being. The timescale for application of the reprocessed photometry within the VSA is a little uncertain, so in the meantime we continue with the existing version of the products in preparation of consortium releases as follows: VMC - preparations will start next week after ingests of P89 data are finalised; ditto VIKING and VHS. As noted above, the VVV P87 preparations are well underway. VIDEO: new mosaics are anticipated sometime in April at which point a new release DB will be created. A P88 VST-ATLAS database is in preparation: quality bit flagging is ready to go, then source merging followed by crossmatching to external surveys (ESO deliveries are required March-April time). Finally, MAR noted that a request for ~1 million cut-outs from the archive(s) has been requested by GalaxyZoo. Software: RSC reported: "Lots of fruitful discussions about maintaining the VSA<->VSAVVV database synchronisation have resulted in a better understanding of how it should all work and a clear plan that should help make this vital infrastructure more reliable and easier to recover from failure (including the implementation of new algorithm in the curation software for distributing unique cuEventIDs between the two databases) as described in a new TWiki document. Squeezed into some spare time between operations, the CU6 profiling on the VVV continues. A few bugs were identified and squashed, most importantly we noticed the objID index that CU6 uses can't be applied to the new combined view of the monthly-split detection tables, so we'll need to investigate how helpful it is to just apply it to the individual tables. An initial performance improvement came from identifying that one week of the 6.5 week DR1 run was spent just calculating the number of parts the data needed to be split into, which on our new curation servers with lots of memory is invariably one. This originally was querying the detection table, hence its slowness, but NJC has found a way around that problem." Since the VISTA reprocessing looks like it will be a photometric recalibration of the existing data in the database, we will need to update CU8 to support updates to the monthly-split VVV detection tables sooner than originally anticipated. Development work for the initial ATLAS release in the OSA, including a new detection quality bit-flag - bit 21 - that flags detections from catalogues that have not been fixed for the illumination correction, and some further tweaking of the quality flags for OSA data, the results of this investigation are detailed on the TWiki. The OSA development work identified a bug in the assignment of frames for source merging in shallow programmes that were set-up by ProgrammeBuilder, which potentially has affected some VISTA and WFCAM non-survey releases." NJC reported: "Lots of development on matched-aperture photometry pipeline. It works from end-to-end, but lots of details need to be filled in. I have also been changing the ProgrammeBuilder and CU13 to work with VST data. This seems to almost be complete now." ETWS noted some small software fixes for OSA and enhancements to general operations code. Non-survey Data Release: MAR suggested that now that UKIDSS DR10 is out the door we could use the quiet (!) time to do some WFCAM non-survey projects. NCH noted that once VSA ingests for P89 are finalised next week, there are a few VISTA PI programmes that could do with DBs being set up and released to their respective owners. Astrogrid deployment, VO & Data Analysis services: MSH noted that a new CoSADIE Data Centre questionnaire needs filling out for WFAU VO data holdings and services. ACTION: RGM to review the CoSADIE DC census returns for WFAU-VO MSH noted also that, thanks to MAR, a new UKIDSS DR8 (DR7 for GPS) SIAP service has been set up so that the images now appear as part of SkyView facilities in the US for example. MSH further reported work on setting up Aladin footprint metadata for UKIDSS which provides coverage facilities within that application. RGM noted that STV has been implementing something similar via Web-SAMP. Miscellaneous: NJC and RGM attended the VISTA celebrations last week; NJC noted many useful discussions with various survey PIs and stakeholders, and described his recent work on the list-driven photometry pipeline. RGM noted that everyone seems generally happy with the VSA. Forthcoming meetings: VMC in Potsdam (April) and VVV in Chile (March). RSC will attend the GES meeting in Nice (April). RGM noted that some limited funds from the roll-out of the current grant are available for working on the Monet column-oriented DBMS. RSC added that the current profiling work is assembling useful data concerning log-jams in database IO-limited curation activities, so we should be in a good position soon to provide specific use cases as benchmarks in any such investigations. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ============================================================= -- Scanned by iCritical.