From nch@roe.ac.uk Fri Aug 4 14:47:05 2006 Date: Fri, 4 Aug 2006 14:00:49 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU WSA weekly meeting minutes, 4th August 2006 Minutes of WFCAM Science Archive meeting: 4th August 2006 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, PMW, RSC, ETWS, JMS, MAR, MSH, NJC Apologies: JPE, AL, RGM, JB DONM: 10am, Friday 18th August 2006 (several team members away on the 11th) Actions discharged this week: ----------------------------- ACTION: MAR to adjust the WSA pages to provide clear top-level warnings to users of spurious source issues, with links to CASU pages. Discharged; clear navigation bar link and links to relevant CASU pages are now present. ACTION: MSH to create accounts for RSC, ETWS and NJC on 64-bit VOTech server. Discharged; accounts on thor now available; see also Hardware below The following from last week all discharged: a) Switch on fast flat-file access to the ingest DB for the UKIDSS (ESO) community - ACTION: MAR c) Change reprocessed file deprecation to bit setting to avoid overwriting QC deprecation codes - ACTION: NCH e) Debug general procedure for index creation/drop to ensure indexes persist into released survey DBs - ACTION: NCH f) Ensure all constraints are present & correct in the WSA ingest DB including both primary and foreign keys - ACTION: NCH g) Revise view creation procedure - tidy up, and automate into CU19, redesigning the defined views in the light of recent experience - ACTION: ETWS/NCH h) Encode quality issues, post ingest (or at ingest) into post-processing bit-wise error flags, e.g. edge proximity, cross-talk images etc. - ACTION: RSC, NCH, MAR and NJC to meet offline, Tues 1st Aug, 2pm in the Plate Library to progress image quality error flagging j) Design procedure and code up new software to cope with CASU-derived photometric recalibration; review photometric recalibration generally - ACTION: NJC, NCH, MAR to meet Thurs 3rd Aug 2pm Plate Library to review photometric recalibration procedures. n) Enhance CU13 to create a "deepleavstack" in DXS by copying through a single intermediate stack product when there is only one - ACTION: NJC Actions partly discharged but continuing: ----------------------------------------- ACTION: JB to review Hardware/OS/DBMS design doc to note areas that need updating and itemise new sections required Continues; JB & MSH have been accumulating much info, updating the TWiki pages in the process. See Hardware. The follwing from last week partly done but continue: p) Release calibration and transit survey data as non-survey databases - ACTION: MAR, NCH & JB l) Review global archive schema to remove extraneous objects - ACTION: ALL, as we go along. Actions carried forward from 28/07/06 meeting: ---------------------------------------------- ACTION: NCH to remove udsSource/udsDetection and associated tables from the EDR, pointing users to DR1 or Nottingham EDR on those web pages - CONTINUES; URL not yet received from Nottingham. ACTION: MSH, JB, PMW, NCH (& RGM?) to have a hardware brainstorming session on Tuesday 15th August at 2pm in the Plate Library - CONTINUES The following from last week continue: b) Merge in the development CVS branch to the main - on hold until RSC fully tested development software - ACTION: RSC d) Debug CU7 for deprecated frame sets and deletions on rerun - ACTION: NCH i) Enhance seaming to use quality bit information - ACTION: RSC (on hold until after h) k) Ensure ingest code can cope with 06A missing data quirks and any associated new attributes - ACTION: ETWS m) Insert Millenium Galaxy Catalogue as external survey; rearrange external catalogues onto batch catalogue server - ACTION: NJC & JB o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Specific points and new actions: -------------------------------- Project management: PMW and NCH presented the Q3 plan to the team, with the main effort allocated to preparing documentation for the October review. Progress for July will be filled in offline, at which time any final issues in the plan can be tidied up. WFCAM update: Nothing new to report this week Comments and issues arising from CASU fortnightly minutes: No new minutes as of AM today Networking: NCH noted that he has received a sample FITS file from CASU, and that there is now no excuse for not completing the ICD. JMS asked if this was a "processed" file or a raw file from the instrument; NCH replied that this was a very good point, and he would find out. JB and PSB have been benchmarking some hyper-scp transfers between CASU and WFAU, with some spectacular results, albeit for relatively small amounts of data transfered at unusually quiet times (e.g. when servers at both ends are quiet). The record transfer rate currently stands at 18 MByte/s; further work will be done when JB and PSB return from holidays. WSA Operations: No news this week. Hardware: RSC has volunteered to produce an ADASS contribution investigating issues with astronomical software support for 64-bit since this is such a generic issue (e.g. python doesn't fully support 64-bit as regards > 2GB sized files). Software: NCH reported much work on CU19 to prepare transit and calibration survey databases for release. RSC reported: "Resolved the problem of programme name ambiguities, and merged UKIRTCAL and CAL into a single programme, in the C++ code, based on Eckhard's Python code algorithm. Various code improvements in the development branch. One of note is that it is now much more simple to move the operations of CU16 to a different server, as is explained in the new TWiki document "RunningCU16". Have also made error reporting in CUs more clear for the operator. For developers, important changes include a new interface for ExternalProcess.py and DataOps.py has been replaced by WfcamsrcInterface.py. Please refer to the on-line code documentation: http://www.roe.ac.uk/~rsc/wsa/" MAR reported: "Put up initial webpage noting know issues with WSA data (i.e. cross-talk and persistence image contamination). Opened up achive listing page for UKIDSS to load server (WSA) allowing for flat file access. Worked on incorporating the new PTS and calibration databases within the UI. A bit messy as they fall bewteen survey and non-survey but nearly done." ETWS has been working on metadata issues with regard to publishing DBs through the AG DSA (see below). NJC has been tidying up CUs 13 & 14 with a view to making them much more automated and operator-friendly. NCH, RSC, NJC and MAR met in the week to discuss quality bit-flagging, and came up with the following list of issues given experience with WFCAM data and with SuperCOSMOS and S-Extractor source extraction software (note that place-holder attributes errBits and ppErrbits are in place in the archive to hold error-bit flags produced by source extraction software and archive-end post-processing respectively): Archive-end post-processing issues (with a relative importance ranking): ---------------------------------------------------------------------- Rank Image artefact and/or quality issue ---------------------------------------------------------------------- 2 Spurious persistence image 3 Scattered light halo / diffraction spike / satellite trail etc. 6 Near edge (e.g. within dither offset) of image area 7 Near very bright star 8 One or more apertures off edge of image area ---------------------------------------------------------------------- Archive-end post-processing OR generated at source extraction stage: ---------------------------------------------------------------------- Rank Image artefact and/or quality issue ---------------------------------------------------------------------- 1 (residual) cross-talk image or affected by same 4 Incomplete image (pixels hit edge of image area) 5 Saturated image ---------------------------------------------------------------------- and finally a bunch of things that can only be generated at the source extraction stage: o Image parameterisation failure (e.g. calculation failure in determining Petrosian/Kron/Hall radius; profile fit failure; moments calculation failure; object too big to deblend or too large for analysis; etc.) o Local sky background determination failure o Image pixels combined confidence below some threshold o Object has been deblended o One or more apertures contain bad pixels WFAU would be interested to hear of any additional issues from those folks reading these minutes; the plan is to implement as many of the archive-end determinable issues as possible and factor relative quality information into, for example, the seaming algorithm in overlap regions. Survey Data Release: NCH suggested that he send an email splash to UKIDSS pointing out the new fast-file access. ACTION: NCH to send an email splash around noting the new archive facility for flat file access in "archive listing", but also pointing out caveats along the lines of the data overview pages. Non-survey Data Release: Nothing new this week. Astrogrid deployment: MSH reported that DSA at astrogrid.roe is ready for testing the UKIDSS taster dataset. ETWS has sent the schema metadata, and the service will be tested next week in advance of the IAU demo. Access to DR3 has been road-tested by several AG developers and appears to be working well. Miscellaneous: PMW noted that ADASS abstract deadline of the end of next week. Both NJC and RSC plan to present work at ADASS in October, and will circulate abstracts asap.