From nch@roe.ac.uk Mon Aug 10 16:49:35 2009 Date: Mon, 3 Aug 2009 10:43:34 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 31/07/09 Minutes of WFAU VDFS Science Archive meeting: 31st July 2009 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, NJC, ETWS, PMW, RPB, RSC, RGM, MSH Apologies: JPE, JMS, AL, MAR DoNM: 10am, Friday August 7th 2009 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: RPB to update the metadata mirrors on the main public server. Discharged ACTION: NCH to tidy up unwanted bulk outgest files and tempdb etc Discharged; tempdb could only be shrunk by stopping and starting SQL Server (a known quirk with SQL 2005; possibly the same for 2008?) Actions partly discharged but continuing: ----------------------------------------- ACTION: ETWS to parse that latest keyword info into the VSA metadata schema files. Continues; NCH again emphasised that this is now top priority after UDS DR5 (see below). ACTION: NCH to draft an interface control document for VSA to ESO-SAF On hold until end August by which time the grant is supposed to be sorted out. ACTION: RPB to fettle /mnt/hatshepsut mounts from linux curation servers Re-actioned; much activity on ironing out SQL 2005 differences in preparation for VSA deployment by RSC, but decision now made to use a cluster node of ramses to host the VSA ingest DB, so the same testing and set-up now needs to be done for that system running SQL 2008. Actions carried forward from 17/07/09 meeting: ---------------------------------------------- None this week. Specific points and new actions: -------------------------------- Project management: The team filled in the progress chart for July. Things are finally moving on the grant, with the separate equipment part having been announced. NCH noted that a request has come from the research council for a summary report on outputs from the e-Science round #2 "VEGA" grant award. ACTION: NCH to fill in the e-Science research outputs form for the VEGA grant. NCH wondered if a similar thing for round #1 might also emerge at some point... WFCAM & VISTA updates: UKIDSS observing on WFCAM is gliding seamlessly into Semester 09B, with most MSBs having been uploaded. Comments and issues arising from CASU fortnightly minutes: The team noted the minutes of the meeting of 15th July. With regard to the extended, deblended photometric attributes bug fix in advance of UKIDSS DR6, MJI has sent an f77 example code and NCH and NJC had a close look; this will now be run on all 05A-08A flat FITS file catalogues. NJC noted that he's started to code up a Python equivalent to do the same corrections to the SQL DBs. Networking: Nothing of note this week. WSA Operations: ETWS and RPB noted that the UDS is now ready for the final catalogue production run and curation stages. NJC has been testing the various codes and tidying up, and volunteered to coordinate the production runs. RSC asked if the unfiltered GPS merged sources in the WSA DB had been saved for comparison with the new filtered versions and if not then there may be an opportunity this weekend as things may be otherwise quite on the load server. ACTION: NCH to liaise with ops over running the GPS DR5 unfiltered source save. RSC also suggested that after UDS DR5 DB-driven products are installed in their final resting place, it would be a good idea to back up all such items (and for previous releases) using the new linux-side backup facility. ACTION: RPB (defered): backup all DB-driven products after UDS DR5. Hardware: MSH and RPB noted that a copy of 2MASS is now residing on the cluster for crossmatch tests between nodes using multiply-attached read-only DBs. NCH noted that the cross-neighbour table would need to be copied over from hatshepsut to facilitate the tests. Software: RSC noted: Finished investigation of SQL Server 2005 issues on hatshepsut - remaining mysteries are documented in TWiki under SqlServer2005, and it's working sufficiently well for full curation now. This means I've been able to finish testing of, and have now committed, recent improvements to the autocuration process: CU8 is now only run if necessary, CU7 can now automatically tell if it only needs to do reseaming, and CU19 is more robust against bcp errors. Updated full curation test script ready for SQL Server 2008 and VSA tests and used it to test menkaure fully for the first time, now that we have forced deep curation onto 64-bit machines, and all is working well there now. During testing I spotted a bug in running external processes for a certain rare combination of options that has now been resolved. NCH and NJC noted discussions earlier in the week with Simon Driver (GAMA PI) concerning galaxy photometry catalogue attributes, and NCH suggested we should have a chat with MJI about this to get his thoughts. ACTION: NCH to bug MJI about standard catalogue photometric attributes, especially wrt extended source photometry. Survey Data Release: UDS DR5: Nottingham delivered the final deep stacks at the end of last week, and NJC is coordinating the curation tasks for the production run for DR5. Should all be done within a week or so. Regarding GPS DR5, NCH and RPB tracked down a problem with a couple of broken FITS catalogue files that were preventing a couple of nights of the reprocessed filtered data from ingesting; CASU have now fixed these. NCH asked that the team now prioritise VSA roll-out tasks before doing any more fiddling with the GPS, however. Non-survey Data Release: Flat-file access is up to date to end of June, and users are accessing non-survey data up to that date. Astrogrid deployment & Data Analysis services: MSH noted that the UKIDSS world (i.e. un-authenticated) access to the recent DR3 release is now switched on; and also that he has put the finishing touches to a monster speedy trawl Java client based on cluster partitioned tables - e.g. trawls the 2 billion row SSA in less than 5mins. The idea is to deploy as a separate service for single large tables in the various archive interfaces for the infrared surveys; MSH plans to work with MAR on this when the latter returns from hols. Miscellaneous: Nothing else this week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================