From nch@roe.ac.uk Mon Nov 27 09:29:25 2006 Date: Fri, 24 Nov 2006 12:05:14 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , John Taylor , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU Science Archive weekly project meeting minutes, 24/11/06 Minutes of WFAU VDFS Science Archive meeting: 24th November 2006 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, MAR, JB, PMW, NJC, ETWS, RSC, LGR, BCW Apologies: JPE, RGM, JMS, AL, MSH, JDT DONM: 10am, Friday 1st December 2006 in the Plate Library Actions discharged this week: ----------------------------- ACTION: JB to send yet another email nudge to EUCS/UKLight. ACTION: NCH to email CASU concerning these. Both discharged; see Networking below. ACTION: JB to update WSA.Provenance as far as possible as soon as possible. Discharged ACTION: NCH and MAR to start QC1 for 05A(v3) Mon pm. Discharged; QC1 well underway for 05A (anticipate finishing automated part early next week at the latest). Actions partly discharged but continuing: ----------------------------------------- The following from last week partly done but continue: ACTION: JB to rearrange external catalogues onto batch catalogue server Continues; smaller catalogues have been shifted. ACTION: NCH to circulate a release schedule for DR2 to all concerned. Continues; see Survey Release below ACTION: RSC to profile CU4 and investigate possible optimisation Continues; on hold until parallelisation and other refactoring is complete. ACTION: MAR to communicate with registered non-survey PIs where 06A data sets are complete. Continues (nearly done) ACTION: MAR to build ra/dec indexes on all external DBs offered for access through DSA where there are none. Continues; DR3 RA done, Dec next; 2MASS already done. Actions carried forward from 10/11/06 meeting: ---------------------------------------------- The following from last week continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Specific points and new actions: -------------------------------- Project management: Nothing to report this week WFCAM update: AA reported: "Those of you who have had data taken over the past couple of weeks will have seen from the shift comments that we've been having some major problems with "zero countdown" hangups. This is a hangover from the installation of new PC hardware. Current status is that we've now finally fixed this with assistance from the UK ATC, and things are working well. The per-frame overhead has been reduced to almost exactly 2 seconds (compared to 3.5 seconds when WFCAM first went onto the telescope) and I'm pretty sure last night's data taking was a record - more than 2650 frames taken in a mix of 5 and 10 second modes. This improvement is at present entirely down to hardware speed; but this may also allow further software adjustments so there may be more improvements to come. With all the problems over the past few weeks, the first tranche of data sent to CASU will have a few header issues (in DR group keywords) but we have kept them informed and there should be no problems in reducing it all; the problems should not be detectable by the time the data reaches the archive." Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: JB noted some movement on the UKLight front thanks to EUCS: "Following correspondence with Sam at EUCS we set up a test of the diplexers with some trepidation as they had not worked in a previous test of something similar. The installation and initial testing proved simple enough and confirmed that they do work in our situation though testing is ongoing to confirm that they are able to adequetely handle heavy traffic. Sam has also tried contacting UKLight re: the missing equipment they are supposed to provide, no joy as yet however." JB and NCH also noted that PSB is pushing UKLight from the southern end to try to get some action at our end. PMW noted that provided the cost of any new hardware modules is low we should push ahead and procure without waiting for the bits from UKLight. PSB replied to NCH's networking message earlier in the week. Rather than optimising for periodic, high rate transfer bursts, a lower bandwidth, more continuous transfer process is advocated and NCH noted that ETWS is sorting out the WFAU transfer software (CU1) to facilitate this (see also Software below). WSA Operations: JB gave the following update: "Ingest of 05A version 3 has been "signed off" in that all files have now been ingested or have been removed as test data and the final scripts run on the data to allow QC1 to start. Ingest of 06A version 1 is going smoothly with all files uploaded, all but four days having been CU3'd and CU4 having approximately 3 weeks of data to go. CU2 (jpeging) has about two weeks of data left to do (thanks Eckhard :) . Initial work on 06A indicates that the number of broken pipeline files coming through is significantly reduced even compared to 05B (it appears that bad data coming from WFCAM is the main issue affecting the pipeline as the only broken image file to date in 06A I believe was caused by this - thanks for much info from MJI). The databases on Thutmose have been shuffled to provide greater performance from the UKIDSS DBs and to give BestDR3 more space for new indices. IRAS, FIRST and ROSAT have also been copied over to Thutmose as part of the ongoing reshuffle of databases, however the WSA interface does not yet used these versions, this awaits completion of the reshuffle. BestDR3 and UKIDSSDR1 have also been shrunk which, again, should improve performance. Ahmose's software issues have interrupted tape system backups however disk backups are still being taken in the mean time, with the WSA also having been backed up to disk on Wednesday night." Hardware: JB noted: "As well as the installation of diplexers for the SRIF network Amenhotep had a disk fail (Eclipse have been contacted). Ahmose's software installation is causing some concern and IT support is working on it, this may however lead to the need to reinstall the machine from the ground up, this would delay DR2 by a week or so however it may be unavoidable (this is the one of the worse case scenarios however). Some routine RAID maintenaince has been necessary on the public server amenhotep this week, and rebuild/initialisation has impacted the server's performance a little. Software: ETWS reported continuing CU1-4 refactoring for optimisation, and fixing bugs along the way. NCH asked, with reference to the preferred transfer mode advocated by CASU, if any potential problems were foreseen over continual, low rate transfers; it was noted by the team that provided CASU stick rigidly to the policy of no fiddling in directories once OK_TO_COPY is set, and let us know immediately when new disk filesystems are added at their end, there is no reason why things shouldn't work. JB reported working with ETWS to iron out a few more bugs in early CUs. RSC reported: "Implemented and tested the new helper script to set detection quality flags for just the two simplest cases, bad pixels, and deblended sources. Saturation flag awaits confirmation of the threshold level for saturation, and whether we should split this flag into two: a severe flag of high confidence of saturation, and a warning of possible saturation. Performance is of the order of 1 minute for the UDS detections. I'm documenting the design and my progress on the TWiki, in the WSA->SoftwareNotes->QualityBitFlags article. Fixed a variety of minor software bugs that are of no immediate concern, but would possibly cause problems in the future, most notably in FitsUtils.makeFileList(). Made a few improvements here and there, specifically to aid Eckhard's refactoring of CUs1-4. Tidied up Python modules, and released a new version of the on-line software documentation. Also, I've created a program based on our Python interface to perform interactive SQL queries (in the same manner as iSQL). You can find it in src/testers/pySQL.py. The helper script SyncTestDb.py, is now able to synchronise test databases with the contents of the WSA, though with relatively limited functionality at this stage." ACTION: NCH to email SJW, STH and MJI to ask for their opinions on archive end saturation error-bit flagging. JB asked about the status of photometric recalibration software; NJC reported that some further testing and performance checks will be done next week. NCH suggested we nudge SJW to ask about how things stand with finalising the DR2 recalibration. ACTION: NJC to email SJW about finalisation of the DR2 recalibration with STH/PCH Survey Data Release: Reiterating the proto-schedule from last week, minus another week of transfer/ingests: Complete transfer of 06A : Done " ingest " " : +2 (CU4 for GPS is the bottleneck) Photometric recal : Still not finalised by the Calibration WG, but following discussions with SJW it has been agreed that QC can proceed without the the final calibration, so this is no longer on the critical path (but will be if not delivered by mid-December at the latest) Quality bit flagging : +1 (can design & test in parallel with previous) QC1 : +1 (05A to be done now; 06A in December following ingest) CU7 (source merging) : +3 (again totally dominated by GPS, and has to be done from scratch because of q-bits and recal) DXS CUs : not on the critical path, since much can be done in parallel with previous; later CUs run fast on the relatively small amount of data) CU16 : +1 QC2 : +1 Final CUs : +3 ... plus a couple of weeks for Xmas. Non-survey Data Release: JB reported that we are currently preparing to release more flat file data to non-surveys and potentially release databases for two Non-Surveys (one update, one new). MAR reported creation/checking of entries/accounts for 06A non-survey prgrammes so that they can get flat-file access. Emails should go out today or on Monday. Astrogrid deployment: Nothing new to report this week. Miscellaneous: Nothing else this week.