From nch@roe.ac.uk Mon Nov 13 13:23:48 2006 Date: Fri, 10 Nov 2006 12:38:47 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , John Taylor , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU Science Archive weekly project meeting minutes, 10/11/06 Minutes of WFAU VDFS Science Archive meeting: 10th November 2006 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, MAR, JB, PMW, NJC, ETWS, RSC, LGR, BCW Apologies: JPE, RGM, JDT, AL, JMS, MSH DONM: 10am, Friday 17th November 2006 in the Plate Library Actions discharged this week: ----------------------------- ACTION: PMW to order an 8TB NAS storage brick. Discharged; 16 TB NAS brick ordered. i) Enhance seaming to use quality bit information - ACTION: RSC - on hold until quality bit flagging is implemented Discharged (by NCH) following discussion of requirements with SJW. Code modified and verified but yet to be tested in anger. ACTION: ETWS & MAR to include NVSS into the WSA external catalogue suite, but as a very low priority and without bugging JB. Discharged; the team discussed whether neighbour tables should be inserted into already released DBs, and it was decided that this should not be done to adhere to the philosophy of keeping those released DBs static. This (and other) external catalogues will become available in the next data release immediately following their incorporation into the archive (e.g. UKIDSS DR2 for NVSS and SDSS-DR5 for example) Actions partly discharged but continuing: ----------------------------------------- The following from last week partly done but continue: ACTION: JB to rearrange external catalogues onto batch catalogue server ACTION: NCH to circulate a release schedule for DR2 to all concerned. Continues; see Survey Release below ACTION: RSC to profile CU4 and investigate possible optimisation Continues; on hold until parallelisation and other refactoring is complete (see Software below) ACTION: RSC to refactor the DB connection module to default to read-only on the main DB but with a run time option for rw. Continues; JB has tweaked permissions/logins server-side to protect the WSA for the time being; RSC intends to formalise things on the curation client side early next week. Actions carried forward from 03/11/06 meeting: ---------------------------------------------- The following from last week continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Specific points and new actions: -------------------------------- Project management: NCH noted that the VDMT scheduled for Tuesday PM was postponed at the request of CASU; rescheduled date is TBD. SJW visited WFAU mid-week to give an astronomy seminar, and team members NCH, NJC and RSC took the opportunity to have a highly constructive meeting concerning DR2 (see Survey Release below). WFCAM update: WFCAM continues to push out survey data. Comments and issues arising from CASU fortnightly minutes: The team noted the minutes of the meeting of 1st Nov 2006; there were no comments. Networking: NCH noted with thanks the email communications with CASU following last week's minuted request for thoughts/advice etc. concerning network transfer performance; with reference to the same, NCH would like to take this opportunity to apologise for any offence caused by suggesting there is a bottleneck at the CASU end. We welcome CASU's input into the discussions on optimisation, and note that we are taking immediate action to optimise software and procedures at our end to accomodate the suggestions. We trust that CASU will take a similarly constructive attitude and meet us half-way on these issues (e.g. somewhere around Doncaster...?) ETWS noted that UIC anticipate having SDSS DR5 in it's higher performance, easier-to-deal-with 3 database file split version within the next few weeks, so we should have a copy of this onto the batch public server thutmose well before the end of the year in plenty of time for UKIDSS DR2. The team agreed to scrub the existing unwieldy single DB file copy as it's essentially useless to us. WSA Operations: JB gave the following curation update: CU1 : 06Av1 done excepting 17th, 18th and 25th July CU2 : 28th April - 20th May and 1st June - 11th June complete (21st May - 31st May and 12th June - 31st July left to do) CU3 : 28th April - 26th June complete (27th June - 31st July and major 05Av3 break left to complete) CU4 : 28th April - 4th May and 16th May - 5th June complete (6th June - 31st July and major break in May 06Av1 left to do) Provenance has yet to be run on any 06Av1 data. JB noted that the 05Av3 major screw-up is with a non-survey project; NCH suggested treating this as a very low priority considering the current pressures. Zero entries in some catalogue files have caused a few headaches; the team agreed that since we are now very much in an optimisation situation, and such problems affect a tiny proportion of files, then we should simply ingest what we can or reject completely any screwy files. Finally, NCH noted with great sympathy JB's recent troubles over provision of a reliable work station to enable working in this whacky organisation. - the operator's job is hard enough without his work station blowing up every 5 minutes. If JB and PMW do not receive immediate satisfaction on this from IT Support, NCH offered to jump in boots-first. Hardware: PMW reported ordering the next storage brick (16 TB NAS). Software: MAR reported: "Looked into why a long running web query hadn't been automatically cancelled. Looks like it was such a convoluted query that the initial parsing (working out the excution plan) took several minutes and the cancel statement was redundantly issued before the query was started. Put a check in to stop this happening in future. Worked on incorporating the NVSS catalogue as another external database." ETWS reported: "Still refactoring CU1 to 4. CU2 has created jpegs for 41 days of 06A (~half of the data). BESTDR5 will be available for download from Chicago at the end of November." Survey Data Release: NJC, RSC and NCH had a fruitful meeting with SJW concerning the enhancements required for UKIDSS DR2: 1) photometric calibration: SJW agreed to progress this asap with the UKIDSS Calibration Working Group and to impress on all concerned the need to get the information to WFAU by mid-December, NJC will finalise and test as far as possible the necessary software to apply the CASU-supplied updates; 2) quality control: NCH and SJW agreed to push on with the 05Av3 QC now (i.e. prior to photometric recalibration). NCH agreed to ask UKIDSS Survey Heads to muck-in with eyeballing JPEGs for 05A (with a deadline of the end of November) and 06A (with a deadline of mid-December); 3) quality error bits: SJW outlined his algorithm for flagging images potentially affected by cross-talk; this and four other error conditions were discussed, ranked and assigned quality error bits in *Detection attribute ppErrBits. RSC will push forward on coding up the required software. 4) seaming: NCH described the current algorithm and SJW suggested several enhancements to be coded up, folding in newly available error bits from 3) and a better treatment of edge-proximity, NCH undertook to code up these enhancements for DR2. ACTION: NCH to communicate with UKIDSS survey heads over jpeg eyeballing. Reiterating the proto-schedule from last week, minus another week of transfer/ingests: Complete transfer of 06A : Done (barring 3 nights) " ingest " " : +4 (CU4 for GPS is the bottleneck) Photometric recal : Still not finalised by the Calibration WG, but following discussions with SJW it has been agreed that QC can proceed without the the final calibration, so this is no longer on the critical path (but will be if not delivered by mid-December at the latest) Quality bit flagging : +1 (can design & test in parallel with previous) QC1 : +1 (05A to be done now; 06A in December following ingest) CU7 (source merging) : +3 (again totally dominated by GPS, and has to be done from scratch because of q-bits and recal) DXS CUs : not on the critical path, since much can be done in parallel with previous; later CUs run fast on the relatively small amount of data) CU16 : +1 QC2 : +1 Final CUs : +3 ... plus a couple of weeks for Xmas. Non-survey Data Release: Nothing new to report this week. Astrogrid deployment: Apparently the DSA demo for secure UKIDSS access went off without a hitch earlier this week. Miscellaneous: RSC noted the deadline for ADASS paper contributions is Monday next week: papers from NJC and RSC are being submitted.