From nch@roe.ac.uk Fri Jul 28 18:23:45 2006 Date: Fri, 28 Jul 2006 15:23:58 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU WSA weekly meeting minutes, 28th July 2006 Minutes of WFCAM Science Archive meeting: 28th July 2006 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, PMW, RSC, ETWS, JB, MAR, MSH, NJC, Apologies: JPE, AL, RGM, JMS DONM: 10am, Friday 4th August 2006 Actions discharged this week: ----------------------------- None this week Actions partly discharged but continuing: ----------------------------------------- ACTION: JB to review Hardware/OS/DBMS design doc to note areas that need updating and itemise new sections required Continues; JB & MSH have been accumulating much info, updating the TWiki pages in the process. See Hardware. Actions carried forward from 21/07/06 meeting: ---------------------------------------------- None this week Specific points and new actions: -------------------------------- Project management: PMW kicked-off the meeting with a glass of bubbly for all to congratulate on the UKIDSS DR1 release. NCH & PMW reported back from the VDMT meeting on Monday PM. Main issues were: i) review is now scheduled for 24/25 October at the IoA in Cambridge; ii) request for top-level, clear links in the WSA pages to CASUs known issues problems in the WFCAM data (cross-talk, persistence etc); iii) NCH to contact VISTA PSP PIs to check archive functionality matches user expectations. MAR and NCH are dealing with ii) and NCH (with help from JPE) has sent out a communication to the VISTA PSP PIs. JPE has been in contact with PPARC over the possibility of an extension to the VDFS development grant, given that the current VEGA-VDFS grant ends at the end of Sep'07 and that VDFS will not have had enough time to fully shake down the system with real VISTA survey data. ACTION: MAR to adjust the WSA pages to provide clear top-level warnings to users of spurious source issues, with links to CASU pages. NCH, PMW & NJC also attended this week's UKIDSS consortium meeting, at which NCH gave an archive update. Generally, comments made on usage of the WSA seemed to be positive. It transpires that the sub-optimal UDS EDR catalogues have been used, and UKIDSS has requested that these be taken offline, with users being pointed to the DR1 (or if they insist on an EDR catalogue, then a link provided to the Nottingham UDS web pages). ACTION: NCH to remove udsSource/udsDetection and associated tables from the EDR, pointing users to DR1 or Nottingham EDR on those web pages On the basis of the two meetings, NCH has compiled a list of enhancements to the archive that will be tackled over the comming weeks - see Software below. Since the review date has now been fixed, and the priorities for work post-DR1 are crystalising, NCH suggested that he and PMW meet offline to tidy up the Q3'06 plan of work. WFCAM update: Nothing new to report this week Comments and issues arising from CASU fortnightly minutes: The team noted the minutes of the meeting of 18th July. NJC has been in touch with JRL to follow up on our previous worries over the catalogue extraction software; face-to-face chats with MJI at this weeks UKIDSS meeting and comments in the CASU minutes imply a rather uncertain status for CASU-Extractor. WFAU is now waiting for a final, stable version; we look forward to receiving this well in advance of any need to run it for the DXS in UKIDSS Data Release 2. The team also noted the issues related to transfer of data. JB has started the requested single-thread copy experiments in collaboration with PSB. We also note that because of outstanding source extraction issues, possible YZ recalibration and standard network transfer costs at the CASU end, no 06A data has been flagged as OK_TO_COPY and we have been asked to hold off transfering 06A until these issues are resolved and the UKLight connection is up and running (assuming the latter is quick to implement). Networking: See previous paragraph. WSA Operations: JB noted that routine backups of the ingest WSA DB and system partitions have all proceeded as normal. Release DB backups are scheduled. Rereretransfer of rerereprocessed 05A data is complete to the end of May, with healthy average transfer rates of up to 14 MB/s. Hardware: The 4TB JBOD purchased from an Astrogrid contribution has been attached to the high capacity catalogue server. NCH asked the team what should be done about support for 64-bit architecture & software. MSH volunteered to set up accounts for RSC, ETWS & NJC on a VOTech 64-bit SUSE linux server for some experiments in advance of disturbing the 64-bit server (with 32-bit Debian linux) khafre, since JB is worried about interruptions to it's file storage capability. ACTION: MSH to create accounts for RSC, ETWS and NJC on 64-bit VOTech server. JB requested a hardware brainstorming session to review all the information gathered about SAN/NAS in advance of preparing the October review documentation. ACTION: MSH, JB, PMW, NCH (& RGM?) to have a hardware brainstorming session on Tuesday 15th August at 2pm in the Plate Library Software: RSC reported: "Tagged software in the main branch of the CVS repository with a DR1_FINAL tag, to denote the version used for the latest data release. Mostly been bogged down testing last week's software enhancements to make database outgests and ingests safer. Meanwhile, I learnt some interesting things about obtaining column metadata from the database, which I have shared in an article in the database section of the TWiki." MAR reported: "Did the user-interface side of things for the UKIDSSDR1 release early in the week. Tweaked web docs and forms and answered some user queries. A few users had problems querying DR3, tried a few fixes and it seems to be working for now." NCH noting that he has assembled a list of issues that have arisen over the past few weeks in the run up to DR1 and from the meetings earlier this week. In no particular order, they are as follows: a) Switch on fast flat-file access to the ingest DB for the UKIDSS (ESO) community - ACTION: MAR b) Merge in the development CVS branch to the main - on hold until RSC fully tested development software - ACTION: RSC c) Change reprocessed file deprecation to bit setting to avoid overwriting QC deprecation codes - ACTION: NCH d) Debug CU7 for deprecated frame sets and deletions on rerun - ACTION: NCH e) Debug general procedure for index creation/drop to ensure indexes persist into released survey DBs - ACTION: NCH f) Ensure all constraints are present & correct in the WSA ingest DB including both primary and foreign keys - ACTION: NCH g) Revise view creation procedure - tidy up, and automate into CU19, redesigning the defined views in the light of recent experience - ACTION: ETWS/NCH h) Encode quality issues, post ingest (or at ingest) into post-processing bit-wise error flags, e.g. edge proximity, cross-talk images etc. - ACTION: RSC, NCH, MAR and NJC to meet offline, Tues 1st Aug, 2pm in the Plate Library to progress image quality error flagging i) Enhance seaming to use quality bit information - ACTION: RSC (on hold until after h) j) Design procedure and code up new software to cope with CASU-derived photometric recalibration; review photometric recalibration generally - ACTION: NJC, NCH, MAR to meet Thurs 3rd Aug 2pm Plate Library to review photometric recalibration procedures. k) Ensure ingest code can cope with 06A missing data quirks and any associated new attributes - ACTION: ETWS l) Review global archive schema to remove extraneous objects - ACTION: ALL, as we go along. m) Insert Millenium Galaxy Catalogue as external survey; rearrange external catalogues onto batch catalogue server - ACTION: NJC & JB n) Enhance CU13 to create a "deepleavstack" in DXS by copying through a single intermediate stack product when there is only one - ACTION: NJC o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH p) Release calibration and transit survey data as non-survey databases - ACTION: MAR, NCH & JB phew. Survey Data Release: DR1/DR1+ were released aproximately to schedule. Overheads on creating and maintaining a subset database (e.g. DR1) in addition to the full database (e.g. DR1+), when the only difference is incomplete frame sets, was discussed. NCH suggested that in future we would like to not have these operational overheads. For example, given that SELECT COUNT(*) FROM lasSource AS s, lasMergeLog AS l WHERE s.frameSetID=l.frameSetID AND l.ymfID>0 AND l.j_1mfID>0 and l.hmfID>0 and l.kmfID>0 in UKIDSSDR1PLUS is entirely equivalent to SELECT COUNT(*) FROM lasSource in UKIDSSDR1 then as data volumes grow, maintaining both is a luxury we cannot afford. Besides, we should be educating our users in getting the most out of a normalised database by routine use of join queries, perhaps helping the poor blighters out by defining views predicated along the lines of the first query above. Let's see how folks (SJW in particular) react to this proposition ... Non-survey Data Release: Nothing new this week, apart from the requirement to get the calibration and transit survey datasets out as pseudo non-survey datasets. Astrogrid deployment: MSH noted that SDSS DR3 is now online through the Astrogrid DSA deployment at Edinburgh, with a few metadata issues (e.g. views and UCDs) remaining to be sorted out. NCH & MAR noted that they couldn't get simple queries to run through the workbench task launcher, but this may be down to their ignorance of ADQL and use of the application generally. Miscellaneous: MSH asked for help with the IAU poster; NCH suggested a rough hack of RSC's ADASS poster followed by review by NCH and RGM.