From nch@roe.ac.uk Fri Oct 30 17:21:06 2009 Date: Fri, 30 Oct 2009 16:57:56 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes 30/10/09 Minutes of WFAU VDFS Science Archive meeting: October 30th 2009 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: PMW, RPB, NCH, MAR, RSC, MSH, ETWS, NJC Apologies: JPE, JMS, RGM, AL DoNM: 10am, Friday November 6th 2009 in the VISTA Hut Actions discharged this week: ----------------------------- ACTION: ALL to review the VSA metadata schema in trunk/sql SQL script VSA_MultiframeSchema.sql ACTION: ETWS to check for corresponding essential FITS keys for VSA ingest against those defined for the WSA ACTION: NCH & RSC to update the VSA schema and install ramses2.VSA ACTION: RSC to check out metadata data mirror production for the VSA ACTION: MAR to continue to shake down VSA user interface for serving SV programmes Actions partly discharged but continuing: ----------------------------------------- ACTION: RPB: backup all DB-driven products after UDS DR5. Awaiting delivery of SCSI card ACTION: NJC to email Terapix to ask about support for RICE compression in S-Extractor Continues Actions carried forward from 23/10/09 meeting: ---------------------------------------------- None this week Specific points and new actions: -------------------------------- Project management: Nothing new to report this week. WFCAM & VISTA updates: Thanks to CASU for the new batch of processed VISTA data. The team have been working away at the metadata schema for the VSA, incorporating the changes to the FITS keywords. NCH requested that this continue to be made top priority over the next few days. NCH noted that there is potential for a major metadata issue in assigning data to survey programmes in that it's not clear if all necessary information will be propagated through keyword HIERARCH ESO OBS PROG ID for the pre-SV, SV and then PSPs. We need to keep a close eye on this one. ETWS and NJC noted also that the higher order radial distortion terms PV2_4 and 5 may require us to make some SW tweaks. NCH noted that the usual request for a WSA update for the next UKIRT Board report has come in. ACTION: NCH to write a WSA update for the current UKIRT Board report and send to JAC by 30th Nov. Comments and issues arising from CASU minutes: The team noted the minutes of the meeting of 21st Oct, specifically the likely timescale for the SV data and the meeting at ESO towards the end of November (we'd like to send a representative if that's OK). PMW and NCH also noted the latest low-level funnies found in the WFCAM data since the September engineering work. Networking: Nothing of note this week. WSA Operations: RPB proposed to migrate WSA operations across to the cluster over the coming weekend, and noted that public-facing DBs will be ready later today. Hardware and Systems: RPB and MSH noted a bug in W2008 that results in horrendous performance when copying large (>20GB) files around. The hope is that this gets fixed in an SP (sometime next year?!) but in the meantime RPB is exploring various options; the impact shouldn't be too bad generally since a lot of large-scale data shuffling is done via tape backups anyway. MSH noted that new webserver seshat is available, so thoth can be security-upgraded and then kept as a fail-over backup. Software: RSC noted: - Following final tests of metadata mirroring on the cluster the completed software changes necessary for cluster operation has been committed to trunk, in expectation of the imminent move of the WSA to the cluster. At which stage a new release branch will be made to allow WSA operations to continue on the cluster. - After some further discussions with RPB & MAR regarding release database requirements on the cluster, some additional software enhancements were made and tested to help prepare username setups etc. To handle updates of read-only cross-mounted survey release databases, these will now be initially prepared on the load server and manually released by the operator. - The metadata mirror script has now been made archive-agnostic allowing for dual VSA/WSA operation by making use of a redesign of the SystemConstants module. Instead of all constants being statically declared, those that differ between VSA & WSA operations are dynamically set at run-time, depending on the supplied command-line arguments. This is implemented using a class structure that employs polymorphism to enable the software to have only a single if-statement to switch between VSA & WSA operations. Initialisation is automatically made by all scripts that use the DbSession object, but others will need to include this new step. All the software will need to be migrated, eventually, to this new design, avoiding static access of runtime-dependent values, but we will run both the old and the new designs in parallel to ease the transition. - To reduce the amount of effort required for the software migration I've asked everyone's help to check to see if there are any old scripts that are no longer required, so that they can be moved out of the src/curation directory and no longer maintained. The team discussed the options concerning the final changes needed to ensure the curation software is agnostic over WSA/VSA operations; RSC and ETWS agreed on the solution as being workable from the development and operational points of view. Survey Data Release: Nothing new to note this week. Non-survey Data Release: Nothing new to note this week. Astrogrid deployment & Data Analysis services: MSH and ETWS have sorted out the VO metadoc for UKIDSSDR6PLUS and MSH noted that it will be published to the VO later today. Miscellaneous: Nothing else this week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================