From nch@roe.ac.uk Mon Feb  5 12:47:03 2007
Date: Fri, 2 Feb 2007 17:04:04 +0000 (GMT)
From: Nigel Hambly <nch@roe.ac.uk>
To: WFCAM Science Archive Team -- Eckhard Sutorius <etws@roe.ac.uk>,
     Johann Bryant <jb@roe.ac.uk>, Mike Read <mar@roe.ac.uk>,
     Nigel Hambly <nch@roe.ac.uk>, Nicholas Cross <njc@roe.ac.uk>,
     Bob Mann <rgm@roe.ac.uk>, Ross Collins <rsc@roe.ac.uk>
Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence
    <al@roe.ac.uk>, Andy Adamson <a.adamson@jach.hawaii.edu>,
     Brian Walshe <bcw@roe.ac.uk>, John Taylor <jdt@roe.ac.uk>,
     Jim Emerson <j.p.emerson@qmul.ac.uk>, Malcolm Stewart <jms@roe.ac.uk>,
     Lorenzo Rimoldini <lgr@roe.ac.uk>, Mike Irwin <mike@ast.cam.ac.uk>,
     Mark Holliman <msh@roe.ac.uk>, Peredur Williams <pmw@roe.ac.uk>,
     Stephen Warren <s.j.warren@ic.ac.uk>
Subject: WFAU VDFS Science Archive weekly project meeting minutes, 2/2/07

Minutes of WFAU VDFS Science Archive meeting:  2nd February 2007
-------------------------------------------------------------------------
-------------------------------------------------------------------------

Present:       NCH, PMW, ETWS, RSC, LGR, NJC, JB
Apologies:     JPE, JMS, JDT, RGM, AL, BCW, MSH. <AR

DONM: 10am, Friday 9th February 2007 in the Plate Library


Actions discharged this week:
-----------------------------

ACTION: NCH & MAR to close out QC1 early next week
Discharged apart from LAS (see below)

ACTION: JB & RSC to commence source merging (after backup?)
         after QC1 close-out
Discharged; GCS is done, LAS is next but awaits QC close-out

ACTION: NJC & ETWS to commence CUs 13,14,2,3,4 for DXS deep stacks
Discharged; CU13 (stacking) done; CU14 (source extraction) now
underway


Actions partly discharged but continuing:
-----------------------------------------

The following from last time partly done but continue:

ACTION: JB to rearrange external catalogues onto batch catalogue
         server
Continues; Juggling of the DR5 files continues; GLIMPSE and TWOMASS 
remain to be uploaded (possibly from tape) onto the batch server 
thutmose

ACTION: RSC to profile CU4 and investigate possible optimisation
Continues; on hold until parallelisation and other refactoring is
complete.

o) Add in a default row for every detector appearing in every detection
table (for schema consistency when querying merged sources and 
individual detections)
  - ACTION: RSC & NCH
Continues; NCH, RSC and MAR discussed this in the week and because
of some subtle complexities it will be put on hold until after DR2.
RSC has made some code changes to make the curation applications
robust against this very minor schema change when it does happen.


Actions carried forward from 26/01/07 meeting:
----------------------------------------------

The following from last time continue:

ACTION: NCH to discuss and review AstroWISE interfacing with MAR
         and JDT.


Specific points and new actions:
--------------------------------

Project management:

The main news this week was fall-out from this week's UKIDSS Survey
Heads telecon, where two items of interest to WSA operations were
extensively debated: a) split UKIDSS releases, and b) the matter
of preparing two DBs per release (the so-called "plus" and 
non-plus DB, where the later comprises complete frame sets).

Concerning a), after lengthy debate UKIDSS have requested that
a release be made first that includes LAS, GCS and DXS (and of 
course UDS the same as it was for DR1), followed by a completed
DB release asap afterwards that includes the GPS. NCH tentatively 
suggested release dates of end Feb for the former, and end Mar
for the latter, and the WSA team have agreed that this should be
feasible.

Concerning b), the Consortium Survey Scientist seems very worried 
about users getting in a mess concerning complete or incomplete 
datasets. The WSA team appreciate this, but the sad fact is that
we do not have the hardware (i.e. disk space) nor operational
resources to do this in perpetuity. NCH announced that he has
decided to dictate the way forward, viz. from now on, only one
DB will be prepared per release (e.g. UKIDSSDR2PLUS in the
current instance). Views will be created in this DB for 
selections from complete frame sets, and the user interface and
online documentation will be tweaked to make it clear what is
going on. Any complaints, please email to dev.null@black.hole.com
where they will be dealt with asap.


WFCAM & VISTA updates:

Nothing new to report this week.


Comments and issues arising from CASU fortnightly minutes:

No new minutes this year.


Networking:

JB notes:
"Sam Wilson forwarded a conversation he had with David Salmon of UKERNA to
  Peter Bunclark and myself; basically nothing much has yet happened at
  UKERNA but Dave Tinker now knows that a connection needs to be made (Dave
  Tinker is the engineer who sorts these things out at the coal face),
  Jeremy Sharp (David Salmon's boss) was CC'd in on all of this.  Clive
  Davenhall has been in contact and says Colin Greenwood is willing to help
  if he can though he recons the best person to speak to would be Sam
  Wilson..."


WSA Operations:

MAR and NCH closed out the QC for the coming release bar the LAS
eyeball deprecations (promised for today).

ETWS has run the source merging for the GCS and stacking for the
DXS with no problems.

JB reported:
"System backups went as normal, Ahmose and Amenhotep were linked to test
  CU16 and in preparation for the release runs of CU16, Thutmose was reboot
  on Monday due to unknown glitch and the version number update to the
  Multiframe table was done (took about 80 minutes).  I kept an eye on CU7
  also due to a command mistype when it was started.
  Copies of the Non-Surveys, UKIDSSR1, UKIDSSR2 and BestDR2 were made on
  Thutmose as part of the catalogue rearrangement."

The team discussed what to do this weekend for DR2: if LAS jpeg eyeball
results arrive in time for QC1 close out before the end of the day, then
the LAS can be source merged over the weekend, otherwise ingest DB
backups will be performed.

ACTION: NCH and JB to prepare for WSA backup by scrubbing the old source
         tables and associate frame sets for GPS, LAS and DXS.


Hardware:

With reference to all the tape activity at the moment, PMW suggested
that he get a quote for an LTO-3 drive for the Overland library to
expedite bulk backups. JB welcomed this suggestion; all agreed that
LTO-4 is a little too bleeding-edge to be considered at the present
time.

ACTION: PMW to obtain a quote for an LTO-3 drive for the Overland
         tape library backup unit.


Software:

ETWS reported finishing the refactoring of CUs 3 and 4.

RSC reported:
"Set up a bug tracking system for the project using the new ROE source forge
  site "trac". Here's the URL: https://forge.roe.ac.uk/trac/VDFSarchive/
  Updated CU7 to apply the Bonifacio correction to the extragalactic survey
  galactic extinction values (see trac ticket 8). Also gave CU7 an option to
  turn off verbose logging.
  Revised the detection quality bit flag algorithms for the deep surveys (see
  trac ticket 6), and further optimised cross-talk artefact flagging
  algorithm. Current plan and tests results are posted in the QualityBitFlags
  TWiki article.
  Updated all CUs to handle correctly the addition of default rows for every
  catalogue in the detection tables (see trac ticket 1)."

RSC also noted the need for a curation client lock system redesign to
facilitate more concurrent curation activities: a new design is
detailed on the new bug tracking system:

https://forge.roe.ac.uk/trac/VDFSarchive/ticket/19


Survey Data Release:

Reiterating the schedule from last time, noting that we are
now aiming for end Feb for a release without GPS merged sources etc:

QC1                      : Done apart from LAS
CU7 (source merging)     : +1 (for LAS and DXS; GCS has been done)
DXS CUs                  : CU13 done; CU14 running now; CUs 2,3,4 next week
CU16                     : +1
QC2                      : +1
Final CUs                : +1

... so we're on target(ish) for end Feb.

ACTION: NCH to check the GCS source merging for DR2


Non-survey Data Release:

Nothing to report this week.


Astrogrid deployment:

Nothing to report this week.


Miscellaneous:

Nothing else this week.