From nch@roe.ac.uk Thu Nov 8 13:50:08 2007
Date: Thu, 8 Nov 2007 13:41:45 +0000 (GMT)
From: Nigel Hambly <nch@roe.ac.uk>
To: WFCAM Science Archive Team -- Eckhard Sutorius <etws@roe.ac.uk>,
Johann Bryant <jb@roe.ac.uk>, Mike Read <mar@roe.ac.uk>,
Nigel Hambly <nch@roe.ac.uk>, Nicholas Cross <njc@roe.ac.uk>,
Bob Mann <rgm@roe.ac.uk>, Ross Collins <rsc@roe.ac.uk>
Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence
<al@roe.ac.uk>, Andy Adamson <a.adamson@jach.hawaii.edu>,
Brian Walshe <bcw@roe.ac.uk>, Jim Emerson <j.p.emerson@qmul.ac.uk>,
Malcolm Stewart <jms@roe.ac.uk>, Lorenzo Rimoldini <lgr@roe.ac.uk>,
Mike Irwin <mike@ast.cam.ac.uk>, Mark Holliman <msh@roe.ac.uk>,
Peredur Williams <pmw@roe.ac.uk>, Stephen Warren <s.j.warren@ic.ac.uk>
Subject: WFAU VDFS Science Archive weekly project meeting minutes, 8/11/07
Minutes of WFAU VDFS Science Archive meeting: 8th November 2007
-------------------------------------------------------------------------
-------------------------------------------------------------------------
Present: NCH, RSC, NJC, MAR, PMW, JB, ETWS, LGR
Apologies: BCW, JPE, AL, RGM, JMS, MSH
DONM: 10am, Friday 23rd November 2007 in the Plate Library
Actions discharged this week:
-----------------------------
ACTION: MAR to put a note on the WSA downtime pages.
Discharged
ACTION: JB & MSH to make a decision as regards the next chunk of
archive mass-file storage (SAS/NAS...)
Discharged; NAS it is as existing NAS boxes probably are under-spec'd
for attaching more SAS.
ACTION: JB to manually defrost the aircon unit behind the archive
servers lest it start peeing water on them from a great height
Discharged; NCH noted that he had also done a manual defrost on
Monday this week, and got Chris Griffin to check the gas levels in
the system. Seems to be performing better now.
Actions partly discharged but continuing:
-----------------------------------------
The following from last time partly done but continue:
o) Add in a default row for every detector appearing in every detection
table (for schema consistency when querying merged sources and
individual detections)
- ACTION: RSC & NCH
Continues; will get sorted (honestly!) as part of the general
ingest DB schema revamp to facilitate rapid generalised
photometric/astrometric recalibration after DR3.
ACTION: RSC to progress implementation of automated replication of
the ingest DB file metadata on a public server.
Continues; RSC has located and fettled the appropriate scripts, and
will now test against the restored ingest DB. A similar
solution is proposed for proprietary-lapsed non-survey data
access (see Non-survey Data Release below). Some discussion took
place as to the names of these databases: FlatFiles? WFCAMfiles? ...?
Actions carried forward from 02/11/07 meeting:
----------------------------------------------
ACTION: JB & ETWS to restore SegueDR6 on thutmose at their convenience
(i.e. no hurry given the present circumstances)
Continues; team agreed to postpone until after DR3 now.
ACTION: JB to set dfg_1 shrinking in the WSA this weekend.
Continues; on hold until after DR3 phase-1 unless there's a convenient
point at which to do it.
Specific points and new actions:
--------------------------------
Project management:
NCH & PMW will attend this afternoon's VDMT at 2pm. PMW has assembled
the usual reporting materials.
WFCAM & VISTA updates:
No news this week.
Comments and issues arising from CASU fortnightly minutes:
The team noted the minutes of the meeting of 6th Nov; in
particular the issues regarding potential reprocessing of data
affected by occasional whacky flats, and the decision on holding
on to monthly chunks of data to be able to refine the photometric
zeropoints. Both procedures seem most reasonable to WFAU; if any
users complain about slight delays in processed data appearing for
flat-file access in the archive they can be informed of the
operational changes.
Networking:
Mike Watson from University of Leicester has pointed us to the
latest "2XMM" source catalogue; this is being bolted into the
WSA for cross-match with UKIDSS DR3 source tables by ETWS and
JB. The pairing radius for XMM has been setup to be 30 arcsecs
in all the necessary places following advice from Mike W.
JB noted that UKLight appeared to stutter at some point over the weekend
and was hanging on Monday morning, this seemed to clear up however and
transfers have resumed using scp (due to a problem with date stamps with
ftp).
WSA Operations:
ETWS reported:
"Fixed broken CU4 ingest for the last remaining 2 reprocessed days.
Revisited the logging procedure in CU1, will apply the changes made to CUs 2
to 4 in due course.
Finished broken (UKLight) CU1 download and discovered that ftp is not
keeping file timestamps. These have to be updated when there is spare time.
Downloaded XMM, ran our code to translate nullvalues into our format and add
HTMIDs, and started writing the schema."
JB reported:
"The 28/29122006 data is now fully in the WSA with the ingest stage now
officially closed out with the final Provenance run.
QC having been finished for some programs, CU5 and Quality Bit Flagging
are now running with CU7 now been trial run on GCS.
CU13 is also being run for the DXS."
Otherwise, NCH noted that QC is now closed out for DR3 save the
issue of repeat frames in the LAS (awaiting input from SJW). (MAR noted
rolled back QC a couple of steps and applied code 80 which was initially
left out.)
Hardware
JB reported:
"MSH and I have discussed the hardware for the next NAS box, we've decided
on a new NAS box rather than SAS box due to worries about the power of the
current machines given they are still currently under heavy use.
The RAID batteries on Amenhotep have successfully be replaced.
Another disk has now been successfully transfered to a NAS box with the
concommitant reduction in mount points."
Software:
NJC reported:
"I have made good progress on a classifier for variable stars. I have been
testing a lot of the statistical methods and making them fast and reliable.
However the whole lot has to be fully put together and run through with
fake/real data still."
MAR reported:
"Modified the archive listing/flat file access and nonSurevy registration to
allow flat file access without programmeIDs needing to be set-up. Primarily
this was to allow access to the UKIDSS/B2 data where the programmeId might
cause issues but it will also lead to faster nonSurvey access."
RSC reported:
"I've prepared the detection quality bit flagger to flag the DXS/UDS
intermediate stacks, now that they will be released in DR3. However, I
still need to test the calibration of the cross-talk flagger for these
intermediate stacks. Once this is done I will merge the changes into
existing release 3 branch.
Also, I've investigated a revised way for curation scripts to handle table
indices in the WSA now that they no longer need to be maintained for release
purposes, as the indices are now attached to the copied release database
only. Once we have a mirror of the WSA metadata for pre-release access
ready, we will no longer maintain indices on the WSA. Scripts that bulk
ingest data will drop any existing indices without restoring them, and
scripts that require specific indices will add just those indices at the
appropriate time. This way indices should never adversely affect curation.
These changes will be implemented for the release 4 branch of the software
that will be released between phase 1 and phase 2 of DR3."
JB noted that the IRAF installation on disk01 has been moved to be
locally installed on djoser as it isn't friendly when run over the
network and it is only rarely needed by NJC on djoser; also
identified a number of bugs, problems and documentation errors in the
software being used and found solutions where possible and passed
the rest on to the appropriate parties.
Survey Data Release:
Updated schedule following on from recent work over the last week:
Weeks What:
0.0 CU5 (diff images for the GPS - after QC, but in parallel with the
following and it's very quick anyway)
1.0 Quality bit flagging
3.0 CU7 (source merging from scratch again, unfortunately)
0.0 CU13/14 for DXS/UDS (in parallel with shallow survey CU7s)
2.0 Final CUs.
The team reviewed DR3 readiness as follows:
LAS: NCH will hand over to MAR to close out the remaining issue of
repeat frame deprecations; quality-bit flagging can take place
before or after this prior to the source merging run.
GPS: ETWS is running the difference image creation; following this,
preparations are on hold until after DR3 phase-1
GCS: NCH ran the quality error bit flagging over night (small issue
over indexes screwing up performance were sorted out yesterday). JB
will set source merging going today.
DXS: JB and NJC are running the stacking and will then co-ordinate
cataloguing (CU13), ingest of images/cats (CUs 3/4). RSC noted
that he has reset the intermediate quality bit status values to
"not done" in advance of running the newly fettled error bit
flagging for DXS/UDS intermediate stack catalogues.
UDS: Delivery of DR3 stacks is imminent; NJC will co-ordinate with
Notts folks and oversee cataloguing/ingest etc.
Generally, ETWS will ensure jpeg creation (CU2) runs as and when
appropriate for image products, and oversee CU3/4 ingests.
Non-survey Data Release:
No news this week.
Astrogrid deployment:
No news this week.
Miscellaneous:
NCH noted that revision #1 of the WSA paper is now back with MN and
it's fate is in the hands of the referee.