From mar@roe.ac.uk Sat Nov 19 17:27:16 2011 Date: Sat, 19 Nov 2011 17:27:10 +0000 From: Mike Read To: VDFS Science Archive Team -- Clive Davenhall , Eckhard Sutorius , Mark Holliman , Nicholas Cross , Nigel Hambly , Rob Blake , Ross Collins , Stelios Voutsinas Cc: CCs for VDFS -- Andrew Lawrence , Bob Mann , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Stephen Warren , Tom Kerr Subject: WFAU VDFS Science Archive project meeting mins, 18/11/11 Minutes of WFAU VDFS Science Archive meeting: November 18th 2011 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: ETWS, RPB, MSH, NJC, ACD, MAR, RSC, RGM, AL, DJCL Apologies: NCH, JPE, KTN, STV, C37, 89X DoNM: 10am Friday December 2nd 2011 in the VH NB: meetings back to fortnightly Actions discharged this time: ----------------------------- None this time Actions partly discharged but continuing: ----------------------------------------- ACTION: RSC to test temporary ingest file writing to high bandwidth shares between curation clients and DB servers. Continues ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues ACTION: RPB (or whoever communicates with the PIs) to keep up the pressure on world public release DB preparation. Continues, discussed later. ACTION: RPB to try monthly detection table partitions for VVV. CU4 ingest of VVV data into separate partitions implemented, but further tests required Continues Actions carried forward from 28/10/11 meeting --------------------------------------------- ACTION: RPB to set up WSA mirror/sync for flat file DBs using MS SQL replication. Specific points and new actions: -------------------------------- Project management: The meeting welcomed Dylan Lawrence (yes a relation) here for "work" experience. Dylan had spent the week generating colour images from WFCAM data for inclusion in the gallery. MAR thanked STV for giving up his desk in the office to accommodate Dylan. WFCAM & VISTA updates: Nothing of note. Comments and issues arising from CASU minutes: No new minutes this time. Going back to the previous minutes, RSC raised the issue of the deblend column being replaced by average confidence values. ACTION: NJC to check how we are handling this and update schema as required. Networking: All OK, at least it was up until the power was switched off last night. WFCAM data up to end Sept 2011 and VISTA data up to end Aug 2011 have been transferred. ETWS noted that reprocessed UDS H-band data are available for transfer and that this would be done shortly. WSA/VSA Operations: RPB reported that the missing VVV catalogue data from P85 were now going in. ETWS reported: -- Working on fixing the segmentation fault in the C++ code on the two new servers. This prevents us in the moment to run CU4 on the VISTA v1.2 data on these machines. -- Ingested most of the GPS reprocessed data apart from a few broken ones. -- Started work on the OSA (OmegaCam Science Archive) schemas with ACD. ETWS noted that the appearance of unexpected project names for non-survey data in P87 had caused a few issues but this had now been handled. Hardware and Systems: MSH noted that apart from two of the Astrogrid servers and various ongoing disk checks, our machines had recovered well from the power downtime. MSH hoped to order some SSD based storage in the near future and test the performance in various scenarios (e.g. hosting the indices on SSD). MSH also noted that following the floods in Thailand disks prices in general had increased by 25%. It was thought that we currently have 6-7 months of pixel storage available. Software: NJC reported: "I have been working on trying to understand differences between the zeropoints that the VIDEO team measure for their mosaics and the ones we do. I have reduced the problem and eliminated other differences (e.g. the seeing). I have also been helping users with queries - particularly the VVV and found an issue with the last VMC release concerning the data which went into deep stacks and tiles." Operations are now routinely running things from trunk rather than release branches. RSC also noted that there should be a way of checking a table column to see if it allows nulls thus avoiding the problem we had with the GALEX neighbour table. ACTION: MAR to look into the SQL needed. Survey Data Release: None since last meeting. NJC reported that following a query from Maria-Rosa it was discovered that some deep stacks in the latest VMC release were not as deep as they could be. The issues had been identified and it would be fixed in the next release. There then followed a run through of where we stand with regard to the various VISTA survey releases. The next proprietary releases of VHS, VMC and VIKING are awaiting availability/completion of P87 data. NJC noted that once he'd finished looking at the issues with the VIDEO data a release could follow very shortly. For VVV the missing P85 data are currently being ingested. The next VVV release would then include both P85 and P86 data and deep stacks. It was thought that the deep stacking could begin now as all the necessary metadata are in. In terms of public releases it was agreed to push the PIs for December releases. A complication with VVV is that they are in the process of publishing a DR1 paper and it was agreed that a public release would make more sense if it matched that. ACTION: NJC to look at VVV DR1 contents. ACTION: RGM, with input from NCH, to contact VISTA PIs with suggested timescale for public releases. Non-survey Data Release: A query had come in from Eric Gaidos trying to track down his data. We passed the buck to CASU who sorted it out tout de suite, merci. Astrogrid deployment & Data Analysis services: Nothing of note. Miscellaneous: ACD reported that he held a meeting with ETWS, NCH and MAR to discuss the entity relationship model for the Gaia-ESO spectroscopic survey. ETWS noted that he'd transferred an initial sample of VST data from CASU. There is a VST Atlas meeting in Durham on the 5th December, RGM, ETWS, ACD and MAR to attend. AL noted, as he headed out the door for doughnuts, that the UKIRT Hemisphere Survey (UHS) had been approved. Initially for one year to do the J band. -- Scanned by iCritical.