From nch@roe.ac.uk Fri Mar 12 16:18:25 2010
Date: Fri, 26 Feb 2010 17:57:05 +0000 (GMT)
From: Nigel Hambly <nch@roe.ac.uk>
To: WFCAM Science Archive Team -- Eckhard Sutorius <etws@roe.ac.uk>,
Mike Read <mar@roe.ac.uk>, Mark Holliman <msh@roe.ac.uk>,
Nigel Hambly <nch@roe.ac.uk>, Nicholas Cross <njc@roe.ac.uk>,
Rob Blake <rpb@roe.ac.uk>, Ross Collins <rsc@roe.ac.uk>
Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence
<al@roe.ac.uk>, Andy Adamson <a.adamson@jach.hawaii.edu>,
Jim Emerson <j.p.emerson@qmul.ac.uk>, Keith Noddle <ktn@star.le.ac.uk>,
Lorenzo Rimoldini <lgr@roe.ac.uk>, Mike Irwin <mike@ast.cam.ac.uk>,
Peredur Williams <pmw@roe.ac.uk>, Bob Mann <rgm@roe.ac.uk>,
Stephen Warren <s.j.warren@ic.ac.uk>
Subject: WFAU VDFS Science Archive weekly project meeting mins, 26/02/10
Minutes of WFAU VDFS Science Archive meeting: February 26th 2009
-------------------------------------------------------------------------
-------------------------------------------------------------------------
Present: NCH, PMW, ETWS, RGM, RSC, RPB, MSH, MAR, KTN
Apologies: JPE, AL, NJC
DoNM: 10am, Friday March 5th 2010 in the VISTA Hut
Actions discharged this week:
-----------------------------
ACTION: NCH to book the Vista Hut meeting room for 2pm Wed 24th.
Discharged; see below for summary of meeting.
ACTION: All the consider possible poster/talk contributions by next
Friday at which point we'll have a look-see and plug any gaps.
Discharged:
NCH: SIA services poster contributed to Software parallel session;
MSH: VO " " " " " " "
RSC: VSA talk/poster contribution to the VISTA session
All attendees should register asap to get early-bird discount on day-rates.
Actions partly discharged but continuing:
-----------------------------------------
None this week.
Actions carried forward from 19/02/10 meeting:
----------------------------------------------
ACTION: RPB & NJC (with help when required from RSC) set up cross-neighbour
tables between Stripe 82 and WSA/VSA surveys as appropriate.
Done for the WSA; same should be done for VISTA VHS, VIKING (& possibly
VIDEO) in the VSA.
Specific points and new actions:
--------------------------------
Project management:
NCH welcomed KTN to the team (official start date is May 1st at which
point early doors will be arranged).
WFCAM & VISTA updates:
NCH noted the constructive meeting earlier in the week concerning
operation/procedural/SW mods required following on from the VISTA PSPIs
meeting in Cambridge. The priorities discussed were:
1) Survey team support during the notional proprietary period. MAR noted
that a science-ready DB (not just metadata with possibly basic catalogue
tables) needs to be released to the survey teams well within their
proprietary period to enable early science exploitation. Hence in addition
to the daily/monthly updating ingest DB mirrors (enabling for example
input into QC etc.) the proposal is that WFAU will create for the survey
science teams a science-ready release DB (merged sources, cross-neighbours
and all the usual bells and whistles associated with UKIDSS releases)
within N months of the last data to be included arriving processed at
the archive, where N depends on the amount of data required to be included.
(The suggestion from WFAU is to make the first release a small Early Data
Release for hopefully rapid turn-around.) The remaining 18 (12) - N months
is the survey team's proprietary opportunity to do headline science etc.
for the first (subsequent) data releases, in line with the "general
conditions" laid down by ESO. Alignment of releases with standard ESO
observing periods, and/or follow-up telescope proposal deadlines is a
possibility, but to start the ball rolling a possible schedule could be:
| 2010 | 2011 | 2012 |
| | | |
| Q1 | Q2 | Q3 | Q4 | Q1 | Q2 | Q3 | Q4 | Q1 | Q2 | Q3 | Q4 | ...
a --b-- c d
<--EDR-->
<--DR1-------------->? e f
where:
a: start of survey ops
b: cut-off in contents of EDRs: somewhere in Q2/Q3 2010
c: EDRs release to PIs
d: EDRs release to the world (within 18 months of start of survey ops)
e: DR1s release to the PIs
f: DR1s release to the world (within 18+12 months of start of survey ops).
and we would be interested in hearing reaction from with upstream in VDFS
on this.
2) Tile issues: the team discussed the required schema changes related to
the baseline assumption that catalogues will be delivered from all of
paw-prints, unfiltered and filtered tiles. the team agreed that all
detections should go into one detection table but with a new attribute
frameCode to make it possible to unpick the required catalogue detections
without recourse to a relational join with Multiframe. Some specific
requirements from RGMcM for VHS were discussed and NJC sought and received
clarification (thanks!) that some kind of pointers linking the detections
would be good for QC and science: the porposal is to define new tables
for these and using the existing source merging SW (rather than inefficient
neighbour tables) to create the necessary info. But WFAU notes that
rolling catalogue updates to PIs ingest DB mirror are only possible on
the same period as the data are delivered from the pipeline (i.e. monthly,
not daily).
3) ESO-SAF interface: all agreed (in the absence of political arguments
from RGM/AL) that this is not a high priority at present, and we should
concentrate on doing what we do, and doing it well.
4) QC changes: MAR noted that generally the PIs seemed happy with his
presentation at the meeting, and that the baseline approach will be to
apply those QC filters appropriate to VIRCam as are currently defined
for WFCAM, then to iterate with the PIs.
Finally, the team went through the detailed points noted by the attendees
on the WFAU TWiki topic. Some noteworthy items:
i) ppErrBits for VIRCam: RSC noted that new bits can be defined for
detections from underexposed regions of tiles ("ears" now is it?!), for
detections from poor regions of detectors (e.g. upper third of detector
16?) and for propagation of the new average confidence level from the
standard catalogues.
ii) detailed metadata/catalogue schema changes arising from pipeline
developments (e.g. background subtraction option keywords, filtering
parameters, ESO VIRCam QC1 data from the headers as written by CASU)
iii) integration of ISIS difference imaging, maybe defined in collaboration
with VVV and/or Eamonn Kerins [TBC]
iv) Split of VHS into separate survey DBs for DES (JHK), GPS (JK) and
ATLAS (YJHK)
Subject to the reaction from upstream in VDFS, some kind of communique
with the PIs (possibly via JPE to coordinate conflicts in priorities)
will be made on all the above.
Comments and issues arising from CASU minutes:
The team noted the minutes of the meeting of 16th Feb; there were no
major comments.
Networking:
ETWS noted that the last lot of transfers from 09B (Jan 2010) from CASU
are grinding to a halt for some reason. MSH volunteered to investigate
the network cards at the WFAU NAS box end to see if there is a problem
since that's where we're writing to and there were network problems
earlier in the week...
WSA/VSA Operations:
Ingests of 09B catalogue data are going ahead. RPB and ETWS noted
that this is the main bottleneck in operations these days, and that
we should seriously consider splitting the monolithic VSA into
separate survey DBs to enable parallelisation of ingests on different
instances of SQL Server. Since this is such a big operational
change, RPB and ETWS will do a few tests and think a bit more about
the ramifications before we plough ahead with any changes.
ETWS noted:
- Updated the thumbnail creation software to use multiple processors, this
closes the software upgrade of early CU1/CU2 quality control checks.
- Together with MAR we fixed the problems in the listdriven photometry tool
where some of the data wasn't returned.
- Started transfers of 09B January data, but transfer speed is very slow.
Also started CU2 on the data that has arrived.
Hardware and Systems:
NCH and RPB noted that the current public catalogue server design of
multiply-attaching release DBs to several SQL Server instances for
performance has fallen foul of the inability of SQL to keep query plan
execution statistics up-to-date as nothing can be written to the
read-only DBs. RPB has a bodged solution whereby the UI logs that
report missing statistics can be parsed to create scripts that must be
run periodically on the servers. This has been done for DR7+ for all
column stats logged so far, but this has not prevented one of the WSA
paper standard queries from timing out in the current release. Obviously
this is one to keep an eye on over the coming months.
MSH noted that he's trying to update the Win infiniband drivers at
the moment in the hope that it cures some of the file-copy performance
bugs when transfering data between SQL Servers.
A beefy new curation server (8-core, 16GB ram) and also a slightly lower
spec VO Services server are in the process of being ordered; the new
96TB (raw) NAS box has been ordered.
Software:
RSC/NJC have continued to discuss and work on the curation infrastructure
mods necessary to support more flexibly the requirements for the VISTA
surveys.
Survey Data Release:
UKIDSS DR7+ is released (LAS, DXS and GCS) and has been announced to
wsa-announce (but not yet to the UKIDSS consortium - we assume SJW
will do the usual).
Non-survey Data Release:
RSC asked that the retrospective deblend fix for non-survey data
earlier that 08A be closed off if at all possible. NCH noted that
09B non-survey prepared catalogue DB releases will be done once
all the 09B data are ingested. (all 09A non-surveys have been
released apart from one by RPB and MAR).
Astrogrid deployment & Data Analysis services:
MSH noted that a DSA will be set up for secure access to UKIDSS DR7+
via the VO.
Miscellaneous:
Nothing else this week.
=============================================================
Nigel Hambly Tel: +44-131-668-8234
Institute for Astronomy Fax: +44-131-668-8416
School of Physics and Astronomy
University of Edinburgh Email: nch@roe.ac.uk
Royal Observatory
Blackford Hill
Edinburgh EH9 3HJ
The University of Edinburgh is a charitable body, registered
in Scotland, with registration number SC005336.
=============================================================