From nch@roe.ac.uk Fri Feb 9 17:32:06 2007 Date: Fri, 9 Feb 2007 16:23:37 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , John Taylor , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 9/2/07 Minutes of WFAU VDFS Science Archive meeting: 9th February 2007 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, PMW, ETWS, RSC, LGR, NJC, JB, MAR, RGM Apologies: JPE, JMS, JDT, AL, BCW, MSH, LGR DONM: 10am, Friday 16th February 2007 in the Plate Library Actions discharged this week: ----------------------------- ACTION: NCH and JB to prepare for WSA backup by scrubbing the old source tables and associate frame sets for GPS, LAS and DXS. Discharged ACTION: PMW to obtain a quote for an LTO-3 drive for the Overland tape library backup unit. Discharged; no response from Eclipse as yet. Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: ACTION: JB to rearrange external catalogues onto batch catalogue server Continues; Juggling of the DR5 is complete and restore will take place over this weekend; several other DBs remain to be uploaded (possibly from tape) onto the batch server thutmose: ACTION: JB/MAR to install UKIDSSDR1PLUS on thutmose ACTION: RSC to profile CU4 and investigate possible optimisation Continues; on hold until parallelisation and other refactoring is complete. o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; NCH, RSC and MAR discussed this in the week and because of some subtle complexities it will be put on hold until after DR2. RSC has made some code changes to make the curation applications robust against this very minor schema change when it does happen. ACTION: NCH to check the GCS source merging for DR2 Continues; first checks indicated all is well, further checks will be done today. Actions carried forward from 2/02/07 meeting: ---------------------------------------------- The following from last time continue: ACTION: NCH to discuss and review AstroWISE interfacing with MAR and JDT. Specific points and new actions: -------------------------------- Project management: NCH and NJC reported on a very constructive meeting with VISTA EPS VVV co-I Valentin Ivanov. New archive-end features required by this survey ideally included provision for light-curve extraction from difference-imaged frames and some UI enhancements to allow automated bulk export of light curve data to enable analysis outwith the archive. VVV have suggested investigation of standard third-party SW packages (ISIS and summat else); tests on WFCAM data and interaction with the VVV consortium will be necessary to home in on the final method implemented. NCH noted other interaction with VISTA EPS PIs: Ultra-VISTA (co-PI Jim Dunlop) is homing in on a working model very similar to the current WSA-UDS one where the intricate stacking work is done outwith the archive (Terapix/AstroWISE are interested in doing this work), so the current design needs no major modifications. For VMC (PI Maria-Rosa Cioni) the major extra curation task is in stacking under-exposed strips and re- source extracting a wide(ish) survey tiled in such a way that these strips are covered fully using combinations of tiles. Although we have the toolkit codes in place to do such a thing, the top-level curation procedure needs to be designed and implemented (this is likely to be useful in several of the wide VISTA surveys in fact). In any case, generally speaking, a wish-list of enhancements is being assembled and this will be factored into the Q2 et seq forward plan of work, and also used (along with the FDR report) to lever further development cash out of STFC under the aegis of the WFAU rolling grant when the latter is renewed in Q2 this year. NCH asked NJC to also review the SMP drafts as they come in to check off the current VSA DB design against requirements. ACTION: NCH to forward VISTA EPS SMP drafts to NJC for additional scrutiny WFCAM & VISTA updates: Nothing new to report this week. Comments and issues arising from CASU fortnightly minutes: The team noted the minutes of the meeting of 1/2/07. WFAU apologise for the frequency and verbosity of our minutes, and congratulate PSB on his recent good (?) news. Networking: JB noted nada from UKERNA. The team wonders if AL has received the gripe note from IoA yet (not that this will do much good since the delay in implementation of the UKLight link is not a University problem...) WSA Operations: ETWS noted finishing copying BestDR5 to thutmose, and running CU13/14 on the DXS successfully and ingested the new files for DR2. NJC is preparing some summary stats for the DR2 DXS to circulate amongst UKIDSS. Hardware: Nothing of note this week. Software: ETWS reported starting to include CU1 and 2 into the new data ingest design, with a view to greater concurrency/efficiency in the curation process. RSC reported making a test dataset for checks on the prototype cross-talk error bit flagging; the suggestion is to let UKIDSS check this out independently to test whether it is working adequately. ACTION: RSC to point SJW to his test DB to check out the prototype implementation of cross-talk error bit flagging. MAR reported modifying the UI to take account of the fact that DR2 will only be one database (DR2plus) eg adding options for selecting views of source tables whilst maintaining compatability with previous releases. NCH noted that a new project has appeared in the top-level WFAU CVS repository. This "morphology" needs to be shunted down several levels (or expunged if it shouldn't be there!) to keep things logical and tidy. ACTION: ETWS to do some CVS administration to help NJC in socially responsible use of the CVS...! NCH and RSC discovered and fixed some quality error bit flag bugs; ETWS and NCH tweaked CU3 to automatically apply old deep stack deprecations and otherwise fettle the DXS at each release point. Survey Data Release: As far as the DR2 schedule is concerned, everything is on course despite a minor hiccup caused by the discovery of minor quality error bit flagging bugs (which resulted in a couple of days back-track): DXS: all stacked, catalogued and ingested and being checked by NJC & UKIDSS; next steps are provenance update and quality flags prior to source merging (JB); in all cases these procedures add up to a few hours of processing so will be interleaved with more heavy CUs for the wider surveys. LAS: will be source merged this weekend (JB) GCS: final checking on the merged source table to be done today (NCH) UDS: nowt to do. GPS: on hold until after initial UKIDSS DB release. The final CUs 16 (external catalogue cross-matching) and 18 (indexing and views) et seq. will take place later next week. DR5 should be restored well in time for an LAS cross-match; NJC noted that the MGC is all present and correct to be included; NCH has contacted Mike Watson in Leicester concerning possible inclusion of an XMM-2p cross match with the imminent release. MAR volunteered to check all the CU16 curation information to ensure a smooth procedure ACTION: MAR to review external catalogue cross-matches in prepartion for CU16 next week. Non-survey Data Release: Nothing to report this week. Astrogrid deployment: Nothing to report this week. Miscellaneous: RGM noted that LGR's new data exploration/visualisation work is now ready to be integrated into the WSA web site; MAR is now in discussion with LGR on how best to do this.