From nch@roe.ac.uk Fri Sep 21 17:43:01 2007
Date: Fri, 21 Sep 2007 16:36:29 +0100 (BST)
From: Nigel Hambly <nch@roe.ac.uk>
To: WFCAM Science Archive Team -- Eckhard Sutorius <etws@roe.ac.uk>,
Johann Bryant <jb@roe.ac.uk>, Mike Read <mar@roe.ac.uk>,
Nigel Hambly <nch@roe.ac.uk>, Nicholas Cross <njc@roe.ac.uk>,
Bob Mann <rgm@roe.ac.uk>, Ross Collins <rsc@roe.ac.uk>
Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence
<al@roe.ac.uk>, Andy Adamson <a.adamson@jach.hawaii.edu>,
Brian Walshe <bcw@roe.ac.uk>, John Taylor <jdt@roe.ac.uk>,
Jim Emerson <j.p.emerson@qmul.ac.uk>, Malcolm Stewart <jms@roe.ac.uk>,
Lorenzo Rimoldini <lgr@roe.ac.uk>, Mike Irwin <mike@ast.cam.ac.uk>,
Mark Holliman <msh@roe.ac.uk>, Peredur Williams <pmw@roe.ac.uk>,
Stephen Warren <s.j.warren@ic.ac.uk>
Subject: WFAU VDFS Science Archive weekly project meeting minutes, 21/9/07
Minutes of WFAU VDFS Science Archive meeting: 21st September 2007
-------------------------------------------------------------------------
-------------------------------------------------------------------------
Present: NCH, RSC, NJC, JB, MSH, LGR, ETWS
Apologies: BCW, JPE, AL, JMS, MAR, JDT, RGM, PMW
NB: DONM: 10am, Friday 28th September 2007 in the Plate Library
Actions discharged this week:
-----------------------------
ACTION: JB and RSC to check the archive linux curation servers for
leaked memory.
Discharged; RSC and JB checked and agreed things seem to be OK;
furthermore last weekend's power cut (see below) rebooted everything
anyway. There is some suspicion that there is a tendency for the
linux servers to gradually slow down if not rebooted for a very
long period.
ACTION: NCH to check the final DR3 tests of CU7 done by RSC.
Discharged; all seems to be well for DR3.
ACTION: ETWS to liaise with CASU over the timing of the GPS 06A retransfer
to avoid any collisions with header updates
Discharged
ACTION: NCH to email a gentle reminder to the UKIDSS Survey Heads about
jpeg eyeballing.
Discharged; DXS and GCS are all in and LAS and GPS promised within the
next couple of weeks.
Actions partly discharged but continuing:
-----------------------------------------
The following from last time partly done but continue:
o) Add in a default row for every detector appearing in every detection
table (for schema consistency when querying merged sources and
individual detections)
- ACTION: RSC & NCH
Continues; will get sorted (honestly!) as part of the general
ingest DB schema revamp to facilitate rapid generalised
photometric/astrometric recalibration.
ACTION: RGM to look through requirements docs to see if any users have
mentioned GALEX.
Continues; RGM supplied GALEX details; will now check WSA SRAD and VISTA
URD for any mention of this.
Actions carried forward from 14/09/07 meeting:
----------------------------------------------
ACTION: JB to restore BestDR1 to thutmose from LTO tape at a convenient
time (but no rush!).
Specific points and new actions:
--------------------------------
Project management:
A final draft MoU has been received from Director JAC; the official
version for signing is anticipated in a couple of weeks time.
NCH noted that the ESO Workshop schedule is now available on the
web pages
http://www.ukidss.org/esoworkshop/programme.shtml
and that travel should be booked early to avoid the Xmas rush.
WFCAM & VISTA updates:
With regard to resumption of WFCAM observations on the 11 Sept, NCH
noted that the following cryptic observing comment had been received
as part of the usual OMP system observer info emails:
"18:26 HST: Tom Kerr
NB. (CASU, please take note). Fibres and cables have been swapped
between the controllers for cameras 2 and 3. If the channel edge
problem comes from within the controller, this effect should now be
seen on array 2. Data from camera 2 still go to WFDR2 and camera 3
go to WFDR3, but controller 3 now controls camera 2, and controller
2 controls camera 3. Probably clear as mud..."
It will be interesting to see if there are any knock-on effects in
the processing...
Comments and issues arising from CASU fortnightly minutes:
No new minutes this week.
Networking:
JB noted that UKLight has been up and down a bit over the last
week or so, but final retransfers of odds and sods of fixed
data for DR3 have not been affected.
WSA Operations:
JB reported:
"Following the WSA fix last week the MGC, XMM and TRANSIT were all put back
online on Ahmose and the TRANSIT work completed by copying it across to
Amenhotep and Thutmose. The obsolete copy of UKIDSSDR2PLUS on Ahmose has
been removed in preparation for DR3.
CU2 and CU4 for DR3 have now finished being ingested and the GPS 06A
reprocessed data has also been dealt with and ingested. The script for
20070101 has been run and the is being reingested as we speak. Finally a
couple of broken files discovered in the above work have been sorted out
as well as the few other fixes remaining from the main ingestion work from
last week."
Hardware:
A major power outage last weekend caused some WSA interface downtime
and has precipitated a breakdown of one of the 3 (redundant) power
supplies on public catalogue server amenhotep. Eclipse are trying
to source a replacement of this old-style PSU, without any luck so
far.
MSH noted that hatshepsut was hit by the power cut last weekend, but
that the SSA loads have been reinstigated.
Software:
RSC reported further work on UDS quality error-bit flagging.
NJC reported working on his poster for ADASS this week, and making
progress on the galaxy photometry paper and classifying variable /
non-variable objects.
ETWS reported poster work and low-level SW maintenance to cope with
the residual 06A/06B transfers and ingests.
JB noted that the updatescript for 20070101 was checked in to the SVN
having been modified to work in our environment with a couple additional
notes added on usage. Thanks to ETWS for fixes to the few bugs found
this week.
Survey Data Release:
Revised schedule (weeks remaining) in the light of the past week's
progress:
Weeks What:
1.0 CU8 (photometric zeropoint recalibration)
2.0 QC1 (much work to be done in parallel with previous tho')
0.0 CU5 (diff images for the GPS - after QC, but in parallel with the
following and it's very quick anyway)
1.0 Quality bit flagging
3.0 CU7 (source merging from scratch again, unfortunately)
0.0 CU13/14 for DXS/UDS (in parallel with shallow survey CU7s)
2.0 Final CUs.
Note that the degree to which some of QC1 can run in parallel is
a little uncertain and the above estimates are generally pessimistic;
the worst-case scenario of is DR3 release by the end of November.
Everything will be done to expedite and arrive sooner.
At this point, we are still on schedule but have a small hiatus while
awaiting the final zeropoint recalibration info from CASU.
ACTION: NCH to pester STH/MJI about any revised timescale for receipt
of zeropoint recalibration info from CASU.
Eyeball QC is complete for DXS and GCS with results communicated to MAR
for his return next week; LAS and GPS are anticipated within the next
few weeks. To use the time over the next few days constructively, the
team discussed a few DB housekeeping items that can be run, including
checking the SVN trunk sql schema scripts against the ingest DB schema
(for recent updates); scrubbing reprocess-deprecated data; truncating
any tables (eg source) that are to be recreated; shrinking filegroups
to recover unused space on the server; checking the 06A GPS reprocess
deprecations have been applied and propagated; and finally provenance
updating and GPS quality error-bit flag updating. NCH, RSC and MAR
will progress things during the next week as the ops team are
attending ADASS.
Non-survey Data Release:
NCH noted that an enquiry has been received from Richard McMahon
concerning the calibration field data taken with WFCAM, and in
the ensuing email exchange including AA at JAC, it seems that
there is probably a case for facilitating access to these data.
NCH has suggested that we enable flat-file access in the first
instance and then make a prepared database release using NJC's
CU6 schema and procedures once the dust has settled on DR3.
Astrogrid deployment:
Nothing new to report this week.
Miscellaneous:
NJC, ETWS and JB all noted that they had produced their finalised
poster contributions for ADASS.
Nothing else (and no crap jokes) this week.