From nch@roe.ac.uk Mon Aug 17 08:31:58 2009 Date: Fri, 14 Aug 2009 15:44:00 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 14/08/09 Minutes of WFAU VDFS Science Archive meeting: August 14th 2009 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, NJC, ETWS, AL, MAR, RPB, RSC, MSH Apologies: JPE, JMS, RGM, PMW DoNM: 10am, Friday August 21st 2009 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: NCH to construct test VSA on ramses including new metadata schema Discharged; bug report sent to ETWS. Actions partly discharged but continuing: ----------------------------------------- ACTION: NCH to draft an interface control document for VSA to ESO-SAF On hold until end August by which time the grant is supposed to be sorted out. ACTION: RPB to fettle /mnt/ramses mounts from linux curation servers Continues ACTION: ALL to review the VSA metadata schema in trunk/sql SQL script VSA_MultiframeSchema.sql Continues; the team discussed various detailed points, and agreed to switch to manual editing following one last parse incorporating the comments received so far. Main changes are to make VDFS attributes in common between WSA and VSA have the same names, and to split off the interminable ESO hierarchical keywords into separate tables to keep metadata searches of the more useful stuff as quick as possible. Actions carried forward from 07/14/09 meeting: ---------------------------------------------- ACTION: RPB (defered): backup all DB-driven products after UDS DR5. Continues; this can now be done; the team agreed that all science image products (diffs, mosaics, stacks) should be backed up from /disk*/wsa/products/ and RSC asked that /disk05 is included for various bits and pieces. Specific points and new actions: -------------------------------- Project management: PMW and RGM continue to negotiate with STFC on behalf of all of us... WFCAM & VISTA updates: No updates this week. Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: Nothing of note this week. WSA Operations: RPB and ETWS noted that GPS source merging incorporating the filtered catalogues is now finished, and that we will proceed with crossmatching (CU16) and release copy-out to the public DB over the weekend. The plan is now to synchronize the announcements and various documentation fettles for both UDS and GPS. Hardware: RPB noted that tests over remotely attached databases on the cluster via the infiniband network have been successful, with only a very minor drop off in trawl speed (less than 5%) with respect to locally mounted storage. The last thing to check is that individually hosted databases can be multiply attached to different instances of SQL running on different cluster nodes. MSH noted sorting out the Windows networking problem on the new cluster, which means that super fast infiniband and bog-standard 1Gbs ethernet can coexist peacefully and efficiently. Software: RSC noted: - Fix for CU19's ability to insert previously excluded programme data following first live test, UDS DR5 release, of the new method, which supports re-release of metadata due to reprocessing / schema changes (previously only insertion of new detection/source products was supported), also took the opportunity of having a set-up test case to improve CU19 by supporting the insertion of multiple programmes on the same run. - Further fixes to CU7 for the support of the multi-epoch frame set association requirements of the GPS DR5 release. This includes a new restriction on the append mode that the merge log remains static save for the addition of new data and removal of deprecated data. No changes to existing frame set associations are allowed due to algorithmic changes, in this case the existing affected frame sets must be deprecated or else we start from scratch again. - Created a new helper script to see if we can have a faster metadata only rollback method by temporarily removing foreign key constraints on the detection tables. This requires a live test on the WSA to see if it is indeed faster than keeping the constraints in place. Survey Data Release: Concerning DR5, the Nottingham folks are doing final checks on the production DR5 deep UDS stacks in the WSA prior to a release announcement. RPB noted that the GPS is progressing well, and that we should check the available public server disk space to make sure there is enough room for CU19 to append the GPS to the existing DR5 DB. ACTION: RPB to check disk space and tempdbs etc on the load and public servers in advance of the CU19 GPS release run. Once DR5 is fully out of the door, the priority will be the deblend fix for 05A-08A, flat FITS catalogue files and DBs in advance of DR6. ACTION: NCH to forward MJI's f77 fix-up code to ETWS, and the latter to write a Python wrapper ACTION: NJC to test his Python DB deblend fixing code MAR noted SJW's recent email concerning DR6 and multiple-epoch data in UKIDSS. The team discussed the various issues on handling stacks of J1/J2 in the LAS for example, and came up with a few options that will be discussed with the survey heads at next week's meeting. Non-survey Data Release: Nothing to report this week. Astrogrid deployment & Data Analysis services: MSH noted that some VO service fixes have been necessary this week for interoperability as certain key metadata items were incompatible with existing standards; also that two DBs (UKIDSSDR3PLUS and SDSS BestDR7) are now published via ogsa-dai for some experimental data mining work. AL noted that plans are afoot to presevre Astrogrid as an "open source" project with some minimal staff resources from a new grant to the research council. Miscellaneous: AL noted that he is on the look-out for suggestions for trumpeting the VDFS contributing to the scientific success of UKIRT in recent years for his talk at the UKIRT@30 workshop in Edinburgh. NJC noted that his contributed talk will show the science archive philosophical design to good advantage. UKIDSS Survey Heads meeting in Edinburgh, Wed 19th Aug. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================