From nch@roe.ac.uk Fri Oct 17 15:11:01 2008 Date: Fri, 17 Oct 2008 15:04:10 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 17/10/08 Minutes of WFAU VDFS Science Archive meeting: 17th October 2008 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RPB, RSC, NJC, LGR Apologies: JPE, JMS, AL, BCW, RGM, MSH, ETWS, MAR, PMW DoNM: 10am, Friday 24th October 2008 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: NCH to assign a new deprecation code for rejected MSBs Discharged ACTION: NCH to update the generic non-survey QC script to set deprecation on the basis of the above Discharged ACTION: MAR to include reject deprecation in the survey QC script(s) Cancelled due to UKIDSS consensus that it's not necessary. Actions partly discharged but continuing: ----------------------------------------- ACTION: RSC to update the WSA Multiframe schema with a new varchar(128) attribute which defaults to NONE ACTION: NJC to update the WSA schema with the new filter details Both done but not checked in/applied due to ongoing CU19/GPS issues Actions carried forward from 10/10/08 meeting: ---------------------------------------------- ACTION: NCH to break everything by deprecating the loadServer DbConstant so that curations codes can be generalised to work for both VSA and WSA... but not until the WSA non-survey SW has been rolled out. - continues ACTION: All to comment on the development of the VSA metadata schema as described on the relevant TWiki page - continues Specific points and new actions: -------------------------------- Project management: Nothing of note this week. WFCAM & VISTA updates: Nothing new to report this week. Comments and issues arising from CASU fortnightly minutes: NCH noted that there has been some communication regarding file naming, and both teams have agreed that keeping things as close to the WFCAM model is a good idea. NJC wondered what the filenaming for intra-OB, nightly pipeline processed tiles would be for VIRcam, as there isn't really a WFCAM analogue as such... maybe "ti" for tile? Networking: Nothing new this week. WSA Operations: RPB reported: "CU19 problems continue. Ross had problems with a certain MFID over the weekend. I ran dbcc checktable over the gps detection tables and no errors were reported. Then I ran the CU19 code by hand in SQL Query analyzer, and it showed no problems at all. Given that, I started up CU19 again. It ran for several days before seemingly grinding to a halt. Given this problem, and the fact that Ross had made some improvements to CU19 in the interim, I decided to kill off the CU19 process, reboot ahmose (just to make sure), and start it off again. RSC noted: "Following a recurrence of the outgest failure on Friday evening, I restarted CU19 using the new, more fault tolerant, version that breaks large outgests into chunks. It failed again over the weekend approximately half-way through (on the fifth chunk of ten). I restarted from this position and it failed again, apparently at the same point, indicative of a table problem rather than spurious error or resource problem." Status us at today: the final 10% of the GPS is being copied into the DR4 release database... fingers crossed... NCH noted that a consensus of opinion has been reached over summit-reject MSB status, and that we will *not* be applying this to survey data as existing QC procedures are implemented to get the best out of the observations (the status will be propagated into deprecations for non-survey data however). Hardware RPB noted: "After the disappointment of the Sun SAN box, Jonathan set up a couple of volumes on his own SAN box for me to test the fibre channel connection with. Using these volumes, I managed to get consistent write speeds of 244MB/s. This is about twice the speed of our existing SAS, and a lot faster than the Sun. Trawl speeds, however, didn't seem greatly improved over our current DAS. May be worth investigating this further before buying our own SAN solution." Scottish Power have decided not to fettle the site power this coming weekend, so ACTION: NCH to remove from the WSA website the archive downtime warning for this weekend Software: RSC reported: "Completed work on the new, more fault tolerant, version of CU19 that breaks outgests of large detection tables into chunks, and provided support for restarting a partially complete copy of a copy. Also gave CU19 the capability to just copy the data from a previously excluded programme into an existing release database, which will save time and disk space when we come to re-release UKIDSS DR4 with the new UDS mosaic data. I've also been helping ETWS update CU3 for integration with the new non-survey automated curation regime and have further developed some helper scripts to help testing procedures, e.g. the capability to copy detections into a test database whilst access to the WSA is restricted. I've now returned to the testing of the new non-survey automated curation and release procedure, which will require a lot of software merges and tests over the next week to bring in new CU3/4 features and CU6 developments." RSC and NJC have both been working hard on merging CU6 changes between the various SVN branches; NJC has enhanced some synoptic recalibration procedures. Survey Data Release: RSC noted that when (if!) the GPS DR4 ever gets out of the door, the schema browser should be updated to reflect a few modifications since the phase-1 release. ACTION: NCH to remember to remind ETWS/MAR to reparse the DR4 schema browser. Non-survey Data Release: Final testing of new regime on hold until GPS DR4 is out of the door... Astrogrid deployment & Data Analysis services LGR reported that he's been working on application of the photometric calibration illumination correction for the intra-WFCAM list-driven photometric analysis service, and that he expects to have the service up and running by the end of the month. NCH noted that there have been several requests in from users concerning areal-coverage usage modes, and it would be good to start working on some new applications to cover these requests with a view to presenting prototypes at the "Science from UKIDSS II" meeting in December. ACTION: LGR to register for "Science from UKIDSS II" meeting, and all to make sure they book accomodation early. Miscellaneous: Nothing else this week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================