From nch@roe.ac.uk Fri Nov 17 17:06:45 2006 Date: Fri, 17 Nov 2006 12:46:01 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , John Taylor , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU Science Archive weekly project meeting: 17/11/06 Minutes of WFAU VDFS Science Archive meeting: 17th November 2006 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, MAR, JB, PMW, NJC, ETWS, RSC, LGR, BCW, MSH, JDT, AL Apologies: JPE, RGM, JMS DONM: 10am, Friday 24th November 2006 in the Plate Library Actions discharged this week: ----------------------------- ACTION: RSC to refactor the DB connection module to default to read-only on the main DB but with a run time option for rw. Discharged; default behaviour is now read-only for WSA and read/write for TestWSA DBs with command line options only for explicit overrides. ACTION: NCH to communicate with UKIDSS survey heads over jpeg eyeballing. Discharged; positive response from the UKIDSS survey heads and we anticipate having 05A(v3) and 06A(v1) eyeballing done at end Nov and mid-Dec respectively by SJW (LAS), Phil Lucas (GPS), NCH (GCS) and Alastair Edge (DXS). Actions partly discharged but continuing: ----------------------------------------- The following from last week partly done but continue: ACTION: JB to rearrange external catalogues onto batch catalogue server ACTION: NCH to circulate a release schedule for DR2 to all concerned. Continues; see Survey Release below ACTION: RSC to profile CU4 and investigate possible optimisation Continues; on hold until parallelisation and other refactoring is complete (see Software below) Actions carried forward from 10/11/06 meeting: ---------------------------------------------- The following from last week continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Specific points and new actions: -------------------------------- Project management: Nothing to report this week WFCAM update: Nothing new to report this week Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: JB noted that CU1 appears to have now finished downloading 06Av1. The latest download speed over Thursday night was 4.8MB/s, a welcome improvement. Things have gone all quiet on the UKLight front again. ACTION: JB to send yet another email nudge to EUCS/UKLight. ACTION: NCH to email CASU concerning these. WSA Operations: JB gave the following curation update: Backups went as normal. CU1 and CU4 have been run with CU1 finishing 06Av1. CU4 has now finished all but two days of the data from 20060428 to 20060626, these two days contain broken files and are now being investigated. A number of broken files have been identified and a note of which files these are and what is wrong with them has been sent to CASU. NCH asked that the archive provenance table be updated asap so that QC1 can progress. ACTION: JB to update WSA.Provenance as far as possible as soon as possible. Hardware: JB noted: "I got another reconditioned desktop machine midweek but had been working in the labs until then. A disk fell off once of ahmose's arrays, it was rebuilt successfully but is being watched in case it does indeed need to be replaced. Ahmose and Thutmose have both been suffering SQL problems, these seems to have been mostly fixed though not quite entirely yet - some of this is related to AstroGRID developments. One of Djoser's disks reported a corrected error, it is being watched and may be replaced should it have too many more." Software: NJC reported: "Finished and tested updateFITS - this takes information from the update tables that CASU supply and updates image and catalogue FITS headers with new values and HISTORY lines." NJC noted everyone's comments on some of the details of the FITS headers, and will be finalising and testing over the next week. MAR reported: "Refactored some of the interface code in light of user requests for better reporting and added functionality (eg upload file of SQL query)." ETWS reported continuing CU1-4 refactoring for optimisation, and fixing bugs along the way. JB reported: "Sorted out DB permissions so that things are a little more secure along with the help of Ross sorting the software side of things. Also identified a number of software bugs and got these fixed with the help of Eckhard and Ross." NCH noted that he and RSC had modified and tested the CU7 SourceMerger for enhanced seaming. Survey Data Release: Reiterating the proto-schedule from last week, minus another week of transfer/ingests: Complete transfer of 06A : Done " ingest " " : +3 (CU4 for GPS is the bottleneck) Photometric recal : Still not finalised by the Calibration WG, but following discussions with SJW it has been agreed that QC can proceed without the the final calibration, so this is no longer on the critical path (but will be if not delivered by mid-December at the latest) Quality bit flagging : +1 (can design & test in parallel with previous) QC1 : +1 (05A to be done now; 06A in December following ingest) CU7 (source merging) : +3 (again totally dominated by GPS, and has to be done from scratch because of q-bits and recal) DXS CUs : not on the critical path, since much can be done in parallel with previous; later CUs run fast on the relatively small amount of data) CU16 : +1 QC2 : +1 Final CUs : +3 ... plus a couple of weeks for Xmas. NCH noted that it is now time to tackle QC1 in earnest for 05A(v3). MAR volunteered to shadow to learn more about the QC procedures. ACTION: NCH and MAR to start QC1 for 05A(v3) Mon pm. Non-survey Data Release: MAR noted that flat-file access is now available for many registered non-survey PIs who will be nudged to make sure they are aware of this. ACTION: MAR to communicate with registered non-survey PIs where 06A data sets are complete. Astrogrid deployment: MSH noted that he has removed old DSA set-ups for WFAU datasets and rationalised, replacing where necessary with new DSA entries. MAR noted: "Looked into why the the AstroGrid cone search wasn't giving expected results for small radius searches on the WSA (and presumably other archives). Kona has tracked down the root of the problem to a mangling of the HaverSine function in some XML template. (As part of this noticed that SDSS DR3 doesn't have simple indexes on Dec (or RA)!)" ACTION: MAR to build ra/dec indexes on all external DBs offered for access through DSA where there are none. Miscellaneous: Nothing else this week.