From nch@roe.ac.uk Fri Jan 20 17:08:19 2006
Date: Fri, 20 Jan 2006 12:57:22 +0000 (GMT)
From: Nigel Hambly <nch@roe.ac.uk>
To: WFCAM Science Archive Team -- Eckhard Sutorius <etws@roe.ac.uk>,
Johann Bryant <jb@roe.ac.uk>, Mike Read <mar@roe.ac.uk>,
Nigel Hambly <nch@roe.ac.uk>, Nicholas Cross <njc@roe.ac.uk>,
Bob Mann <rgm@roe.ac.uk>, Ross Collins <rsc@roe.ac.uk>
Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence
<al@roe.ac.uk>, Andy Adamson <a.adamson@jach.hawaii.edu>,
John Taylor <jdt@roe.ac.uk>, Jim Emerson <j.p.emerson@qmul.ac.uk>,
Malcolm Stewart <jms@roe.ac.uk>, Martin Hill <mch@roe.ac.uk>,
Mike Irwin <mike@ast.cam.ac.uk>, Peredur Williams <pmw@roe.ac.uk>,
Stephen Warren <s.j.warren@ic.ac.uk>
Subject: WFAU WSA meeting minutes, 20th January 2006
Minutes of WFCAM Science Archive meeting: 20th January 2006
-------------------------------------------------------------
-------------------------------------------------------------
Present: NCH, ETWS, NJC, RSC, PMW, JB, RGM
Apologies: JPE, JDT, AL, MAR, JMS
DONM: 10am, Friday 27th January 2006, plate library
Actions discharged:
-------------------
None this week.
Actions partly discharged but continuing:
-----------------------------------------
ACTION: JDT to set up a Datascope Launcher for the WSA on the AG
workbench pages.
Continues.
Actions carried forward from 13/01/06 meeting:
----------------------------------------------
ACTION: NCH to investigate the usefulness of a NAS solution for medium
term archive mass storage.
- CONTINUES
Specific points and new actions:
--------------------------------
Project management:
Nothing to report this week.
WFCAM update:
See the stop press pages linked from the TWiki for latest news.
Comments and issues arising from CASU fortnightly minutes:
No new minutes as of 20/1/06 AM.
Networking:
An email from Peter Clarke (NeSC) has been received concerning
UKLight, asking for a major push to make use of the facility.
NCH suggested we revisit this when the heat is off of the
impending survey data release (we have no spare effort to look
into this for the next couple of weeks).
WSA Operations:
JB reported:
"CU3 was run and run again and it went in. CU21 was run, fixes made and
finished correctly, GLIMPSE was uploaded to the DB on Amenhotep and is now
accessable (possibly even alongside FIRST etc. for the public). CU4 was
run just for the DXS and UDS and went fine. The Provenance Table was
updated, the CU7 fix was made as well as the CU16 fix to add the GLIMPSE
Tables and progress has been in automating the disection of the lists of
files which CU3 didn't manage to ingest.
In an interesting experiment ETWS independantly wrote code to do the
same thing (the above mentioned disection) and comparisons of results
helped greatly with developing both pieces of code and has resulted in
more information about the problem files being available."
JB & NCH reported that a post-ingest, post-QC1 disk backup of the main
WSA DB has been made yesterday to guard against loss of the last two
week's work (in lieu of standard tape backups which await the
resolution of the ahmose login problem - see below).
NCH asked about the current provision for "helpdesk" facility in the
archive system (something raised by SJW during his recent visit).
An email system is in place (we believe) but it might be a good idea
to review this prior to going live with the ESO-wide data release.
ACTION: JB to investigate the current provision of "helpdesk"
facilities in the WSA, and look into off-the-shelf options for
a more fancy system for future reference.
Hardware:
The ahmose login problem (reboots on login) remains unsolved owing to
no downtime on the server being available; otherwise all is fine
on the hardware front. PMW reported that the order for the new
filestore server "djedefre" will be placed as soon as ITSG have
OKed the spec.
Software:
ETWS reported:
"Split up the CU1 code in two distinct parts to run it decoupled from the
database and ingest transfer information at a later time.
Fixed the jpeg code and added more output, so that we can check easier
for missing files in the database as well.
Added a script for CU4 to create programme specific lists, so that
ingestion can be instantiated easier."
RSC reported:
"Further development of helper scripts to fill out the Provenance table and
update the database with schema changes. Also some enhancements to the
curation codes (e.g. wfcamsrc) to handle the latest "issues" brought up by
the current data release drive."
NJC reported:
"I have revamped the stacking software with updated DQC info, fixed
the problem of more than 1 program ID and hacked it to use the priority
list given by Alastair. I have written much of the software to get the
zeropoints for deep stacks and I have tested much of the catalogue
software provided by Cambridge."
NCH has been closing out the remaining QC1 issues in close collaboration
with SJW, has implemented the same for 05A survey data and has tweaked
CU7 (source merging) for the impending UKIDSS EDR release (switch off
multi-epoch passbands etc.)
Survey Data Release:
Things are going well with UKIDSS Early Data Release preparations.
With reference to the schedule sketched out last week:
The schedule over the next couple of weeks to release is
therefore:
05B JPEGing (CU2) : Will be completed within a few days
05B image ingest (CU3) : Currently working, will finish by 14th
Non-survey set-up (CU21) : Monday 16th morning (very quick)
AperCor fix ingest DB : " " " " "
05B cats ingest (CU4) : Start Mon 16th for a day or two
Post-ingest QC : Nearly complete for 05A; 05B by end of
next week (after CU4 has completed for
05B)
- all completed with the exception of the 05B quality control
(awaiting input from UDS and DXS folks) and the final couple of
days worth of JPEGS.
Source merging (CU7) : Over next weekend (?)
- has started overnight. NCH has produced the GCS source catalogue
and is testing now; in the meantime the LAS is being done during
today and GPS will be done over the weekend. NCH noted that the
source merging itself is very quick (less than an hour for LAS/GCS)
but that "seaming" the resulting catalogue to flag duplicates in
overlap regions is dominating the run time (owing to a high
number of DB transactions).
DXS/UDS stacks (CU13/14) : Initial work in parallel starting Tues next
week; schedule a little unclear at present
(will be clarified with survey heads and
NJC mid-week next week)
This is progressing well, with both DXS and UDS representatives
having come to WFAU earlier this week to thrash out the plan of
work. The idea is to have QC1 finished for both of these at the
beginning of next week to enable stacks to be produced; in the
case of UDS, Swarp is being used to create the final mosaiced
product, while CASU toolkit codes are being used for the DXS
multiframe stacks (UDS wide dithering strategy and requirements
for source extraction results in the requirement for a single
mosaiced product to be used as opposed to individual detector
stacks). In both cases, the CASU source extractor and subsequent
software (classification and astrometry) will be used to create
a catalogue for archive ingest.
Cross-matching (CU16) : Mon 23rd ish (for a few hours)
Indexing (CU18) : Tues 24ish (very quick)
Pre-release QC : Week of 23rd for 05A and 05B
Can all start when source merging is complete.
Release DB prep (CU19) : Starts around end of week of 23rd.
Awaits completion of all
Code enhancements have been done for CU7 (minor) but not
CU19 (major), and the second QC procedure needs to be implemented
(manually in SQL; outline procedure has been discussed and agreed
with SJW and SD). The schedule is getting tighter...
Non-Survey Data Release:
No news this week.
Astrogrid deployment:
MAR has been attending the AG developer's workshop this week.
Miscellaneous:
Nothing else to report this week.