From nch@roe.ac.uk Tue Jan 23 14:29:00 2007 Date: Mon, 22 Jan 2007 12:44:48 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , John Taylor , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 19/1/07 Minutes of WFAU VDFS Science Archive meeting: 19th January 2007 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, PMW, ETWS, RSC, LGR, MAR, NJC, RGM, JB, BCW, AL, MSH Apologies: JPE, JMS, JDT DONM: 10am, Friday 26th January 2007 in the Plate Library Actions discharged this week: ----------------------------- ACTION: NCH to summarise yesterday's deliberations concerning input into VISTA PSP SMPs and email around to VDMT Discharged; summary was discussed at VDMT and NCH took an action to finalise the archive notes and then forward to CASU for similar pipeline notes to be added before forwarding to the Vista PSP PIs. ACTION: NCH to email VDMT and SJW concerning the likelihood of a delay in the DR2 release schedule. Discharged; this has precipitated a heated debate amongst UKIDSS and related interested parties (see below). Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: ACTION: JB to rearrange external catalogues onto batch catalogue server Continues; ETWS and NCH are juggling the DR5 files during the absence of JB. ACTION: RSC to profile CU4 and investigate possible optimisation Continues; on hold until parallelisation and other refactoring is complete. Actions carried forward from 12/01/07 meeting: ---------------------------------------------- The following from last time continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH ACTION: NCH to discuss and review AstroWISE interfacing with MAR and JDT. Specific points and new actions: -------------------------------- Project management: The main news this week follows yesterday's VDMT telecon. PMW and NCH presented progress reports for the last quater and the forward plan for the current one. NCH's proto-notes for Vista PSP PIs were also discussed, VDMT suggested a few changes, and NCH took an action to progress these with CASU before forwarding on the PIs for use during their preparation of SMPs. At the end of the VDMT, RGMcM raised his suggestion concerning UKIDSS releases in the context of the current archive-end recalibration log-jam (more of which below) and the fact that the GPS is dominating most processing and curation activities. NCH suprised everybody (I think) by agreeing that RGMcM's idea of splitting releases into project-based rather then monolithic UKIDSS-based DBs was a good one, and went along with the VDMT's endorsement of the same. NCH put this idea to the WSA team for their reaction, which was also generally very positive from an operational point of view. The team consensus is that staggered, separate releases could go a long way to reducing the stress of limited, all-in-one big splashes. One provisio is that complication in the user interface, and potential confusion of users, must be borne in mind with proliferation of release DBs, but in principle there is no show-stopping problem. AL agreed with the above operational point-of-view, but, wearing his UKIDSS PI hat, raised some cautionary political views. In any case, AL has taken on board RGMcM's suggestion and the VDMT endorsement, as is now seeking a debate amongst the UKIDSS Survey Heads to try to reach some kind of agreement there as to what VDFS should be asked to do. NCH emphasised that it is the WSA team's job to furnish the UKIDSS Survey Heads with as much relevant information as possible regarding the implications of various options. NCH noted that Valentin Ivanov, co-I on the VISTA PSP known as VVV (note to self: too many TLAs in these minutes) will be visiting Edinburgh on the 7th Feb to talk to the archive team concerning requirements and SMP for that survey. NCH and NJC will be available for those discussions. WFCAM & VISTA updates: WFAU note with much sympathy the issues raised by CASU in their VDMT report concerning screwed-up WFCAM raw data. We will keep an eye out for associated funnies when the 06B processed data start to arrive at this end. Good news for Vista: VIRCam was shipped from the UK on the 17th and is due to arrive in Chile on the 26th: http://www.pparc.ac.uk/Nw/Vista_camera.asp Comments and issues arising from CASU fortnightly minutes: No new minutes this year. Networking: JB reported: "Have been in touch with Sam at EUCS and Peter at CASU about UKLight. Sam had a little news; he has been in contact with UKERNA (whom we are waiting for technical approval for hardware from and then waiting on the actual UKLight connection) and David Salmon is apparently on holiday at the moment so he had a quick chat with David's boss, but no news. Sam's man, Dunc, will be up to give our connection a once over as and when the necessary with UKERNA has been sorted out." NCH noted that Clive Davenhall (remember him?) is visiting from NeSC on Tuesday 23rd, and will talk to JB and ETWS about the current impasse concerning implementation of a CASU-WFAU UKLight connection. This was again raised at the VDMT by CASU, and RGMcM has agreed (after consultation with AL) to draft a gripe-note to be sent from on high at the IoA to AL for his use in nudging appropriate queso-grandes at this end. In fact, there is renewed hope of succesful resolution within a Hubble time, since after last Friday's meeting and before typing of these minutes, UKERNA have spoken to EUCS and it seems we may get a date for for implementation of the link sometime next week ... hope springs eternal. ETWS has been attempting to recover 4 lost SDSS DR5 DB backup files from Chicago, but their insistance on using their bespoke "sector" transfer software is causing more of a bottleneck than the network link ... WSA Operations: The main (bad) news last week was the continuing log-jam over CU8 (photometric recalibration). The WSA catalogue load server suffered one of it's recurring fits-a-pique on Wednesday, causing the month-old CU8 to fall over and cheerfully roll back every single data modification done in that period. The good news is that this happened very quickly, and a refactored CU8 was finalised, tested and set running at the end of the week. At the time of writing these minutes, the procedure is 90% complete and we anticipate completion some time today (22nd). Hence the new CU8 takes around three days cf. the old which took more than thirty; particular thanks are due to RSC, NJC and ETWS for pulling off this feat of optimisation. As regards the release schedule, see below. Hardware: JB noted: "Ahmose has been given a well deserved reboot (it does have a BIOS time sync. issue though that will be properly corrected at next reboot (beginning of February)). Amenhotep has been behaving admirably but Thutmose's "I" array has failed again... possible plans for fixing this are in the pipeline. NAS planning continues." NCH pointed out that the WSA load server got rather close to hitting the disk storage buffers recently (mainly through bloated transaction logs). Now is probably a good time to order a new 16x300GB = 4TB (net) JBOD ACTION: PMW and JB to order new JBOD for ahmose JB noted that there is a rumour flying around concerning planned power outages in March for equipment safety testing; more of this nearer the time. Software: RSC reported: "I've almost completed development of the cross-talk artefact flagging algorithm, but have been hampered so far by not being able to test it at all due to CU8 database locks (and subsequent CU8 redevelopment). Of course, I won't know how much work is left to be done or how successful it can be until initial testing has been done. Most of the work in progress is logged in the TWiki under QualityBitFlags. Since we have known that CU8 will need to be restarted I've been tirelessly ensuring it can be safely and reliably interrupted and restarted, without loss of database updates to date or data corruption. To aid the operator I've prepared a guide to running CU scripts, specifically CU8, in a manner where they can be safely interrupted and restarted, it may be found on the TWiki under RunningCUs. I've also reshuffled the order of the files it works through so that they are grouped into individual survey programmes, in case we gain any benefit through caching by updating just one detection table at a time." NCH has been working on the archive paper write-up. Survey Data Release: Reiterating the proto-schedule from last time: Complete transfer of 06A : Done " ingest " " : Done Photometric recal : Should finish today (22nd) Quality bit flagging : Done except for cross-talk; since time is pressing, suggest we defer cross-talk flagging to later releases (still no external input from CSS concerning implementation details) QC1 : +1 - hopefully will be completed this week (still awaiting LAS eyeball results tho'...) CU7 (source merging) : +3 (totally dominated by GPS) DXS CUs : To commence this week following appropriate QC1 CU16 : +1 QC2 : +1 Final CUs : +3 Final release date(s) now hinge on exactly how they are done. As originally planned, a monolithic release can be achieved by the end of March; if the survey releases are split, the LAS, GCS and DXS probably can be done by end Feb (with GPS to follow around end Mar / beginning Apr). Non-survey Data Release: Nothing to report this week. Astrogrid deployment: Nothing to report this week. Miscellaneous: TLA = Three Letter Acronym