From pmw@roe.ac.uk Fri Aug 28 20:22:16 2009 Date: Fri, 28 Aug 2009 16:37:55 +0100 From: Peredur Williams To: Nigel Hambly , Eckhard Sutorius , Mike Read , Mark Holliman , Nicholas Cross , Rob Blake , Ross Collins Cc: Andy Lawrence , Andy Adamson , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Bob Mann , Stephen Warren , Peredur Williams Subject: RE: WFAU VDFS Science Archive weekly project meeting minutes 21/08/09 Minutes of WFAU VDFS Science Archive meeting: August 28th 2009 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: ETWS, MAR, RPB, RSC, MSH, PMW Apologies: JPE, JMS, RGM, NCH, AL, NJC DoNM: 10am, Friday September 4 2009 in the Vista Hut; PMW in the chair Actions discharged this week: ----------------------------- ACTION: RPB to fettle /mnt/ramses mounts from linux curation servers Continues; RPB and MSH noted that Microsoft have messed everything up by removing support for Unix services in Windows 2008. Suggestion is now to either use Samba, or anonymous NFS. NCH suggested whichever is the easiest to administer, modulo any significant speed gain. Discharged: Working with NFS but may switch to Samba at a later date. ACTION: RSC to convene a cluster deployment meeting asap. Discharged: Meeting held Monday 24th. Minutes are on the twiki. Actions partly discharged but continuing: ----------------------------------------- ACTION: NCH to draft an interface control document for VSA to ESO-SAF On hold until end August by which time the grant is supposed to be sorted out. ACTION: ALL to review the VSA metadata schema in trunk/sql SQL script VSA_MultiframeSchema.sql Continues; ETWS, RSC, NJC and NCH have been examining/editing the schema. ACTION: NJC to test his Python DB deblend fixing code Continues; NJC is in the process of testing. ACTION: RPB (defered): backup all DB-driven products after UDS DR5. Continues; this can now be done; the team agreed that all science image products (diffs, mosaics, stacks) should be backed up from /disk*/wsa/products/ and RSC asked that /disk05 is included for various bits and pieces. Continues - imminent. ACTION: ETWS to check for corresponding essential FITS keys for VSA ingest against those defined for the WSA Continues: ACTION: RPB to put a note on the operations TWiki concerning exactly how to ensure there is sufficient disk space for a CU19 run. Continues: Specific points and new actions: -------------------------------- Project management: PMW gave a brief update on progress of the various grants replacing the WFAU roller. WFCAM & VISTA updates: Comments and issues arising from CASU minutes: (August 17 meeting): Comment on screwed-up 09A files noted; ETWS still trying to sort one strange night of July data. The starting date (July 27th) of Semester 09B was noted. Networking: Nothing new to report this week. WSA Operations: RPB reported: DR5 is finally ready for release. The GPS processing has finished at last, just about the same time as the UDS re-release has completed, so we're ready to go. New databases are in position on both public servers and ready for the big red switch to be thrown. Cluster preparation continues. Copying all of the databases from our existing public servers. All of the external catalogues (with the exception of SDSS) have copied over, along with some of the earlier UKIDSS data releases. What's left are the big, chunky databases which will copy over during the course of the next week or so. ETWS reported: - Transferred the July data for 09A and 09B from CASU. - Created a new set of browser pages for DR5. - Ingested the 2 missing nights for 09A after MJI fixed the data where broken WCS info was stopping ingests. Hardware: Following Cluster discussion on Monday, MSH, with input from RSC and others, developed various plan for distribution of WSA, VSA and external databases on the Cluster nodes. MSH reported the trawls of network-mounted databases on the Cluster were slower than expected and would continue tests on the network to speed them up ACTION: MSH to continue tests to speed up queries of network-mounted databases MSH planned to install the new network switch on Monday 4th, next at-risk period, and MAR would notify users. Software: RSC reported: Tested all of our curation software under SQL Server 2008 on the cluster in anticipation of the WSA moving there and the VSA being created there. As long as the cluster is set-up the same as the existing load/public servers it will work, as there were no noticeable changes since SQL Server 2005. However, I would like to test some new functionality that makes full use of the cluster's unique features to improve performance and most importantly reliability, but cannot until hardware tests and configuration are complete. Also, minor modifications to the automatic curation process were required to ensure it can cope with re-releases, for the UDS DR5 re- release. Survey Data Release: DR5 phase 2 with GPS and UDS is on Amenhotep and Thutmose and ready for release this afternoon. MAR is revising the plots. Non-survey Data Release: Nothing to report this week. Astrogrid deployment & Data Analysis services: Nothing of note this week. Miscellaneous: ------------------------------------------------------------------- Peredur Williams, Institute for Astronomy, University of Edinburgh Royal Observatory, e-mail: pmw@roe.ac.uk Blackford Hill, or: peredur.williams@ed.ac.uk Edinburgh fax: (+ 44) (0) 131 668 8416 EH9 3HJ www: http://www.roe.ac.uk/~pmw phone: (+44) 131 668 8300 (University exchange: 75 300) "The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336." -------------------------------------------------------------------