From nch@roe.ac.uk Tue Jan 16 20:49:05 2007 Date: Fri, 12 Jan 2007 14:33:46 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , John Taylor , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 12/01/07 Minutes of WFAU VDFS Science Archive meeting: 12th January 2007 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, PMW, ETWS, RSC, LGR, MAR, NJC, RGM Apologies: JPE, JMS, JDT, JB, BCW, AL, MSH DONM: 10am, Friday 19th January 2007 in the Plate Library Actions discharged this week: ----------------------------- ACTION: NCH & PMW to meet 10am Tues 9th to prepare the formal Q1 VDFS work plan. Discharged; see Project Management below ACTION: NCH to ask for help from IT Support in accessing thutmose network drives from Veritas BackupExec Discharged; ahmose instance of Veritas can now see thutmose. Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: ACTION: JB to rearrange external catalogues onto batch catalogue server Continues; ETWS and NCH are juggling the DR5 files during the absence of JB. ACTION: RSC to profile CU4 and investigate possible optimisation Continues; on hold until parallelisation and other refactoring is complete. Actions carried forward from 05/01/06 meeting: ---------------------------------------------- The following from last time continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH ACTION: NCH to discuss and review AstroWISE interfacing with MAR and JDT. Specific points and new actions: -------------------------------- Project management: NCH received a nudge from JPE in the week concerning liaising with VISTA PSP PIs over the content of their Survey Management Plans (SMPs; to be delivered to ESO by noon on the 16th Feb). Of particular concern is the requirement to deliver "survey products" to the ESO Science Archive Facility in Garching; but there are of course other operational issues. Anyway, NCH, AL, PMW, RGM and MAR had a very constructive conflab yesterday (Thurs pm) at which a plan of action for the next week or so was developed; the idea is to gently steer ESO in the direction we (VDFS) want. This fledgling plan of action will be circulated amongst the VDMT in the first instance to see what everybody thinks and for discussion at the next VDMT telecon. ACTION: NCH to summarise yesterday's deliberations concerning input into VISTA PSP SMPs and email around to VDMT As regards the next VDMT, the latest rumour is there may be one next week. Last quarter's progress charts have been filled in for presentation at this; PMW and NCH have created the Q1'07 plan of work and presented this to the team. Some tweaking is necessary and will be done by PMW; this plan will also be available for perusal by the VDMT. WFCAM update: Nothing new to report this week Comments and issues arising from CASU fortnightly minutes: No new minutes this year. Networking: DR5 installation on the batch catalogue server thutmose was proceeding well, until ... (see Hardware below). Unfortunately, 4 out of the 64 SQL backup files of the DR5 database may have been lost, and will have to be copied again from the States (but no problem - 140 GB in total and available from JHU, and now Chicago as well). WSA Operations: CU8 (photometric calibration) continues to grind away on the load server; it is now 82% complete (cf. ~70% at the same time last week). It seems almost inevitable that there will be a consequent delay of a few weeks to DR2 release: ACTION: NCH to email VDMT and SJW concerning the likelihood of a delay in the DR2 release schedule. Possible significant speed-ups have been identified by the team for if/when CU8 is rerun; these have been implemented by NJC. Hardware: Batch catalogue server thutmose suffered loss of RAID volume I: again (same array was lost just before Xmas) this week. This was not affected any online databases; it has just interupted installation of DR5. Work on diagnosing and fixing the problem is ongoing. Software: RSC reported further work on CU7 (for non-surveys) and cross-talk quality bit flagging. Some input was expected from the Consortium Survey Scientist on the latter, and would be very useful if the offer is still open... NJC reported CU8 optimisation work and refactoring of CU14 ETWS reported continuing refactoring of CUs 1-4, and work on implementation of new CU0 (control for automation and optimisation of transfer/ingest CUs). Finally, NCH noted that IT Support have set up a new SVN source repository including the "trac" web interface that may be useful at some point in the future. Survey Data Release: Reiterating the proto-schedule from last time: Complete transfer of 06A : Done " ingest " " : Done Photometric recal : As noted above, was expected to be complete by now but may take several more weeks ... Quality bit flagging : Done except for cross-talk; development will continue as long as photo recal runs but if not finished by recal completion then crosstalk flagging may not be implemented for DR2 ... QC1 : +1 - stalled due to photo-recal locking up the ingest DB to all other modifications. CU7 (source merging) : +3 (again totally dominated by GPS, and has to be done from scratch because of q-bits and recal) DXS CUs : not on the critical path, since much can be done in parallel with previous; later CUs run fast on the relatively small amount of data) CU16 : +1 QC2 : +1 Final CUs : +3 As noted above, photometric recalibration is still running. As much QC scripting is being done in the meantime: eyeball results are in for DXS, GCS and GPS (awaiting final confirmation of where to cut); we are awaiting LAS results. Non-survey Data Release: Nothing to report this week. Astrogrid deployment: RGM noted that the CAS-jobs "MyDB" concept is coming to the fore in AG discussions (this would extend the functionality currently offered in MySpace/VOSpace) and we will likely need to work with the AG team on development and implementation of such a service at WFAU. Miscellaneous: The team cast an eye over NJC's beautiful colour posters created from Serpens calibration observations with WFCAM. PMW suggested we should frame them and put them on permanent display somewhere.