From nch@roe.ac.uk Fri Dec 1 12:55:57 2006 Date: Fri, 1 Dec 2006 12:14:33 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , John Taylor , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes: 1/12/06 Minutes of WFAU VDFS Science Archive meeting: 1st December 2006 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, MAR, JB, PMW, NJC, ETWS, RSC, LGR, BCW, AL, MSH Apologies: JPE, RGM, JMS, JDT DONM: 10am, Friday 8th December 2006 in the Plate Library (PMW in the chair as NCH is at UKIRT until the 12th and then on hols until 23rd). Actions discharged this week: ----------------------------- ACTION: NCH to circulate a release schedule for DR2 to all concerned. Discharged since Survey Release in these minutes seems to be serving the same purpose ACTION: MAR to communicate with registered non-survey PIs where 06A data sets are complete. Discharged ACTION: MAR to build ra/dec indexes on all external DBs offered for access through DSA where there are none. Discharged (by MAR and JB) ACTION: NCH to email SJW, STH and MJI to ask for their opinions on archive end saturation error-bit flagging. Discharged; advice received from MJI & SJW. ACTION: NJC to email SJW about finalisation of the DR2 recalibration with STH/PCH Discharged; all quiet on the recalibration front. Actions partly discharged but continuing: ----------------------------------------- The following from last week partly done but continue: ACTION: JB to rearrange external catalogues onto batch catalogue server Continues; more jiggery-pokery likely as SDSS-DR5 arrives and SSA is rebuilt. ACTION: RSC to profile CU4 and investigate possible optimisation Continues; on hold until parallelisation and other refactoring is complete. Actions carried forward from 24/11/06 meeting: ---------------------------------------------- The following from last week continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Specific points and new actions: -------------------------------- Project management: PMW noted that we have received no feedback concerning the VDFS FDR in October, and the date of the next VDMT has yet to be confirmed. ACTION: PMW to contact JPE regarding VDFS FDR feedback and the date of the next VDMT. NCH noted that the ESO Public Surveys Panel have now ratified the resubmitted Vista PSPs, subject to a viable Survey Management Plan being provided by the respective PIs in the New Year. WFCAM update: Apparently another earthquake last weekend that shook UKIRT off it's shear pins, but WFCAM continues to pump out data... Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: ETWS noted that transfer of BestDR5 from Johns Hopkins is going well with a rate of ~2.7MB/s, 24 out of 64 files are downloaded. WSA Operations: JB gave the following update: "CU3 finished 06Av1, no outstanding broken data. CU4 finish 06Av1, two outstanding days of data caused by broken GPS catalogues (work continues to find these broken files). Ra and Dec indexes have now been added to FIRST, IRAS, ROSAT, GLIMPSE, PSSA and SkyServerV3 (USNOB already had them) meaning all DBs now have these indexes. Amenhotep performance imporved by removing Ahmose link and moving tempdb to an emptier, less troubled disk." NCH asked that the archive provenance be updated asap to enable 06A QC to start; RSC noted that a few issues concerning duplicate files and reprocessed files need to be checked today. ACTION: JB and RSC to update the archive provenance, then let everyone know when it's done ACTION: MAR to contact SJW and arrange for 06A QC to start asap next week. Hardware: JB noted that the NAS box installed though not yet live, and thumose lost two disks from the same partition but both came back fine and no data seems to have been lost (phew). NCH enquired about storage space on thutmose as regards rebuilding the final, full-blown (4TB) SSA. JB suggested this should be possible with a little juggling; PMW noted that we can always get another JBOD and NCH agreed that there is certainly plenty of expansion capability on thutmose. In any case, this is a relatively low priority activity compared to WFCAM archiving. Software: RSC reported: "Completed script to set the detection quality bit flags for all but the cross-talk artefact flag. This is all documented in the TWiki article: QualityBitFlags. Developed a fix for a bug in ProvenanceFiller, identified during QC, where a component file's multiframeID may be assigned for an alternate processing revision version of a FITS file. Implementation of this fix awaits an update to the FITS header keyword design of the deep stacks we've produced. Also, added in a check to ProvenanceFiller to ensure that individual science frames do not have multiple Multiframe entries. Improved various developer helper scripts, SyncTestDb, pySQL, and BinaryToCsv with the help of bug reports from Eckhard and Johann. Supported Nick with development of recalibration scripts using the new software framework." ETWS reported refactoring of CU3 and 4 to enable parallel runs. NJC reported checking of codes updateFITS and reCalibration this week. Made sure that all the updates do not change the block size on FITS files (to optimise for speed). MAR reported updating the MultiGetImage service to provide a link to check status of results rather than accessing email, and other minor tweaks to the User Interface in the light of user feedback and requests. Survey Data Release: Reiterating the proto-schedule from last week, minus another week of transfer/ingests: Complete transfer of 06A : Done " ingest " " : +1 (CU4 for GPS is the bottleneck) Photometric recal : Still not finalised by the Calibration WG, but following discussions with SJW it has been agreed that QC can proceed without the the final calibration, so this is no longer on the critical path (but will be if not delivered by mid-December at the latest) Quality bit flagging : +1 (can design & test in parallel with previous) QC1 : +1 (05A to be done now; 06A in December following ingest) CU7 (source merging) : +3 (again totally dominated by GPS, and has to be done from scratch because of q-bits and recal) DXS CUs : not on the critical path, since much can be done in parallel with previous; later CUs run fast on the relatively small amount of data) CU16 : +1 QC2 : +1 Final CUs : +3 ... plus a couple of weeks for Xmas. Things are reasonably on track. NCH asked that Quality Flagging and QC1 (for 06A) now start in earnest; we aim to apply whatever photometric recalibration details are supplied to us by mid-December over the Xmas period. ACTION: RSC to apply the quality error bit flagging procedure as it stands to the WSA. NCH reported that 05A jpeg eyeball results are now in for DXS and GPS. ETWS completed the production and DB updates for 06A jpegs and the green light has been given to UKIDSS Survey Heads to complete eyeballing of those (deadline mid-December). Automated QC for 05A has been completed apart from one remaining deprecation issue concerning LAS Y catalogue data (an email has been sent to SJW concerning this). NCH will check in to the CVS the 05A script before departing for UKIRT, and this script should be used as the basis for the 06A QC1 procedure. Non-survey Data Release: JB noted that he intends to get two non-survey DBs out to their respective users within the next week. Astrogrid deployment: The team discussed the DSA access to various DBs on the WSA public batch server. MSH and ETWS will be adding new metadocs for various ancilliary DBs; the conesearch SQL is in the process of being optimised for speed; MAR and JB have generated spatial indexes for all catalogue holdings to expedite incoming conesearches from the AG DSA. MSH asked if we should impose a conesearch query timeout on the DSA; 10 min was suggested as a suitable value. Miscellaneous: Nothing else this week.