From nch@roe.ac.uk Mon Aug 10 16:53:09 2009 Date: Fri, 7 Aug 2009 17:24:39 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archives weekly project meeting mins, 7/08/09 Minutes of WFAU VDFS Science Archive meeting: August 7th 2009 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, NJC, ETWS, PMW, RPB, RSC, MSH Apologies: JPE, JMS, AL, MAR, RGM DoNM: 10am, Friday August 14th 2009 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: ETWS to parse that latest keyword info into the VSA metadata schema files. Discharged; see software below ACTION: NCH to fill in the e-Science research outputs form for the VEGA grant. Discharged ACTION: NCH to liaise with ops over running the GPS DR5 unfiltered source save. Discharged ACTION: NCH to bug MJI about standard catalogue photometric attributes, especially wrt extended source photometry. Discharged; MJI noted that it's difficult to be certain until the RGs are sorted out, but the baseline plan is still (in this priority order) a) replace Hall with half-light radius attributes, b) move to elliptical apertures for extended source measures, and c) do further model analysis photometry (psf, sersic). Actions partly discharged but continuing: ----------------------------------------- ACTION: NCH to draft an interface control document for VSA to ESO-SAF On hold until end August by which time the grant is supposed to be sorted out. ACTION: RPB to fettle /mnt/hatshepsut mounts from linux curation servers Continues Actions carried forward from 31/07/09 meeting: ---------------------------------------------- ACTION: RPB (defered): backup all DB-driven products after UDS DR5. Continues; this can now be done; the team agreed that all science image products (diffs, mosaics, stacks) should be backed up from /disk*/wsa/products/ Specific points and new actions: -------------------------------- Project management: PMW noted that the split of effort and costs between the rolling grant, trimmed to fit the ground-based budget, and the new VISTA exploitation grant, was agreed with Colin Vincent and the details are now being worked on, together with the JeS for the VISTA grant. WFCAM & VISTA updates: No updates this week. Comments and issues arising from CASU fortnightly minutes: No new minutes this week. With regard to the deblend bug fix discussed last week, MJI, NJC and NCH have all been communicating and are satisfied that the proposed fix is doing what it should. Both ends will run the same f77 code to fix the FITS catalogue files, while NJC is creating a Python script to do the same fix in WFAU's SQL DB catalogues. Networking: ETWS and RPB noted that some 09A retransfers of broken catalogue files had to be done via Janet - apparently someone put a "spade" through UKLight somewhere near Leeds earlier in the week. WSA Operations: RPB noted: "UDS now fully processed. Ready to go ahead with CU19 to release the public facing database for DR5. Begun setting up test dbs on cluster. Need to sort out mounting shared directories on linux servers. Plan to run a full backup of the WSA over the weekend." Ops noted that GPS DR5 will now continue with CU7 once the full WSA backup is secured. NCH thanked NJC, ETWS, RSC and RPB for sorting out the 64-bit curation codes for efficient handling of large UDS (and Vista) tiles. Hardware: RPB noted: "Tests begun with cross-matching the SSA and TWOMASS on the cluster. After some strange behaviour, managed to streamline the bloated SSA Source table to see how much speed that produces. Early results show a 50% speed up. Further tests to compare dbs mounted over infiniband to follow." MSH noted that a new router, and memory expansion modules for the 64-bit servers, are being ordered. Software: ETWS noted: - Created the VSA Multiframe and MultiframeDetector schemas by parsing the available VISTA files. There are ~400 attributes in Multiframe and ~800 in MultiframeDetector. - Changed the GPS detection ingestion procedure to only check constraints with the last ingested binary data file. This will save us ~2h per ingested data file. ACTION: ALL to review the VSA metadata schema in trunk/sql SQL script VSA_MultiframeSchema.sql ACTION: NCH to construct test VSA on ramses including new metadata schema NJC has spent much time tidying up and sorting out the mosaic/tile curation procedure using UDS DR5 as the roll-out test and with a view to efficient handling of Vista tiles. RSC noted: - Aiding NJC with regression testing of deep/synoptic changes for the UDS DR5 release. - Diagnosed and supplied a fix for a bug in CU8 that lead to zero-point metadata not being updated in the database. - Prepared a release_11 branch ready for the UDS DR5 release, making sure the code base passed all epydoc and pyflakes tests first and moving aside some old code that no longer requires maintenance, as a proactive attempt to aid future software changes for dual WSA/VSA curation. More work can be done in this area. Survey Data Release: UDS DR5 release is being done today, with an announcement to follow early next week just to give a few folks the chance to test. GPS DR5 is now the ops priority, with CU7 (source merging) expected to take a couple of days (as estimated by RSC). As far as DR6 is concerned, the deblend fix is of course the top priority, so no schedule just yet, but NCH noted that cross-semester QC must be done with care in the LAS now as deliberate J repeats for 2nd epoch are included in semester 08B (thanks to SJW for the heads-up). Non-survey Data Release: RSC noted that further survey-like DB releases are now on hold until after the deblended, extended bug fix. Flat-file access continues to be routinely quickly available; NCH noted that a Korean PI has been in touch earlier in the week concerning their 09A data. Astrogrid deployment & Data Analysis services: MSH noted that he's been experimenting with some new data mining infrastructure software ("admire") which sits on top of "ogsa-dai" (DSA-like web service software; made in Wales, isn't it?!). The idea is to investigate qso discovery from a learning set of known objects. Miscellaneous: Nothing else this week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================