From nch@roe.ac.uk Sat Apr 8 15:51:11 2006
Date: Fri, 7 Apr 2006 14:10:12 +0100 (BST)
From: Nigel Hambly <nch@roe.ac.uk>
To: WFCAM Science Archive Team -- Eckhard Sutorius <etws@roe.ac.uk>,
Johann Bryant <jb@roe.ac.uk>, Mike Read <mar@roe.ac.uk>,
Mark Holliman <msh@roe.ac.uk>, Nigel Hambly <nch@roe.ac.uk>,
Nicholas Cross <njc@roe.ac.uk>, Bob Mann <rgm@roe.ac.uk>,
Ross Collins <rsc@roe.ac.uk>
Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence
<al@roe.ac.uk>, Andy Adamson <a.adamson@jach.hawaii.edu>,
John Taylor <jdt@roe.ac.uk>, Jim Emerson <j.p.emerson@qmul.ac.uk>,
Malcolm Stewart <jms@roe.ac.uk>, Martin Hill <mch@roe.ac.uk>,
Mike Irwin <mike@ast.cam.ac.uk>, Peredur Williams <pmw@roe.ac.uk>,
Stephen Warren <s.j.warren@ic.ac.uk>
Subject: WFAU WSA weekly meeting minutes, 7th April 2006
Minutes of WFCAM Science Archive meeting: 7th April 2006
-------------------------------------------------------------------------
-------------------------------------------------------------------------
Present: NJC, JB, MAR, ETWS, NCH
Apologies: AL, JPE, JMS, PMW, MSH, RSC
DONM: 10am, Friday 21st April 2006
Actions discharged this week:
-----------------------------
ACTION: MAR to liaise with Eclipse regarding installation
Discharged
Actions partly discharged but continuing:
-----------------------------------------
None this week
Actions carried forward from 24/03/06 meeting:
----------------------------------------------
ACTION: JB to review Hardware/OS/DBMS design doc to note areas that
need updating and itemise new sections required
- CONTINUES
ACTION: NCH to investigate the usefulness of a NAS solution for medium
term archive mass storage.
- CONTINUES
ACTION: ETWS and JB to put a TWiki note on the intranet detailing
the outcome of the UKLight meeting at NeSC.
- CONTINUES
Specific points and new actions:
--------------------------------
Project management:
NCH reminded everyone that next Tuesday, there is a VDMT face-to-face
meeting in Edinburgh at which MJI, STH and JPE will be present.
NCH reported back from the UKIDSS survey heads meeting at NAM in
Leicester on Thursday. This was mostly concerning preparations for the
next UKIDSS survey observing with WFCAM, but a few VDFS issues arose
and were discussed. The main issue for WFAU was the release date of
the Data Release 1 in the summer: this has now been fixed for Friday
the 14th July. Some discussion ensued as to whether the same
DR1/DR1+ databases would be required this time around; JB suggested
that DR1 views could be implemented within a DR1+ database to
avoid having to prepare two databases. SJW's preference (obtained
after the meeting) is for the same as EDR: two databases prepared
for release: DR1 and DR1PLUS.
WFCAM update:
NCH reported a brief chat with AA concerning this problem, which
occured at the end of WFCAM semester 05B. Apparently a controller
problem (which is now fixed) was the cause of one detector's data
not being written. AA volunteered to send a list of affected dates
so that we can watch out for any problems, but we note that we do
not anticipate any archive issues.
NCH noted that a sixth UKIDSS survey is now implemented (under the
project identification U/06A/52). This concerns time-resolved
photometry of late-type stars for the detection of planetary
transits, and makes use of poor-seeing UKIDSS WFCAM time.
ACTION: NCH to install the top-level curation metadata for the new
UKIDSS survey
ACTION: NJC to close out the data modelling for arbitrary multiple
epoch surveys
NCH noted that WFCAM is back on telescope at the end of April for
observing May/June/July.
Comments and issues arising from CASU fortnightly minutes:
The team noted the minutes of the meeting of 23/3/2006. Regarding the
transfer rate issue and UKLight, we are somewhat confused over the
CASU comments on our minutes. With reference to worries over costs,
is it a manpower issue? As we understand it, UKLight itself doesn't
cost anything (at the moment); WFAU capital HW outlay costs are
estimated at 4.3K plus 0.33K per annum maintenance, so surely this isn't
a problem? (We suspect CASU costs would be similar or less, but note
that CASU need to contact Chris Cheney in Cambridge to discuss).
As regards bottlenecks, if apm7 "grinds to a halt" when WFAU are
transfering, then this sounds like a bottleneck at the CASU end to us,
and this will be compounded if the machine is being used for other
purposes while we are attempting to transfer. Finally, the figure
of 3 weeks does not make allowance for the necessarily low xfer
duty cycle since the data need to be written at the CASU end, and
ingested at the WFAU end, outside the transfer time. Further, any
interuptions (power cuts, as have occured at both ends, etc)
require manual fix-ups and further reduce the duty cycle. Finally,
reprocessing (several times!) necessitates retransfer...
Bottom line: we now have much evidence that there is no bottleneck
at WFAU. The VDFS cannot currently cope with both WFCAM and the projected
VISTA data volume at the same time, so we believe that we should press
ahead with UKLight. This issue will be discussed at VDMT next week.
Networking:
JB reported:
"UKLight discussions have been progressing and I'm currently putting
together a possible application to UKLight for our bandwidth assignment,
however this is something VDMT have to agree on before it gets the go
ahead I think due to the hardware outlay and what seems to be a reticence
on CASU's part to go ahead with the project. More statistics will soon
be available with Eckhard's new CU1 script (hopefully by Monday)"
WSA Operations:
JB reported:
"Backup operations continued as normal, a backup of the WSA is scheduled
for early next week (overnight like last time). Ahmose was rebooted last
week and I used the monthly downtime to shift BestDR2 around on Amenhotep.
I ran a test of the new CU7 and worked fine apparently. CU3 and CU4 have
been run through to the 9th January 2006 with a final CU1 run scheduled
for tonight to complete 05B data transfers.
BestDR3 is almost entirely copied over to Amenhotep and should be finished
by the end of the day along with BestDR2 having been split up on Amenhotep
we may even have space to build BestDR3 on Amenhotep assuming it can be
split into four files.
I've discussed various SQL issues (both support calls and interface
upgrades) with MAR and have hopefully helped! :)
A couple of new Non-Surveys have come in but have not been added as yet
but can probably be released in the next week or so as long as nothing
much comes up."
Hardware:
MAR reported that the new catalogue server is in the process of being
installed in C1.
Software:
JB reported:
"Eckhard and I have fixed a number
of bugs in CU3, CU16 and the make files arising from the lastest code
changes. Eckhard and I have discussed a a number of fixes and features
including working on a new script to run in tandem with CU1 and a fix for
CU3 MfID problems which he is currently implementing."
ETWS reported:
"Fixed some new bugs in the C++ code for CU3. Refactored CU3 taking
account of the problem with wrongly identified production frames.
(15% of ingested frames had a problem with the cross-referencing
of associated dark/sky/confidence frames). Wrote a script (and ran it)
to identify problematic entries in the DB."
NJC reported:
"Ross and I have added in the extinction correction code to CU7 and updated
the schema.This has been tested and now run on the WSA, updating the LAS,
DXS and UDS.
I have been comparing standards stars taken in WFCAM to UFTI, IRCAM and
2MASS. I haven't completed this yet, but I have found that Ser-EC84, Ukirt
FS144, is a variable, with variation over several months. It has a
variation of ~0.05 mag in J, ~0.06 mag in H and ~0.1 mag in K.
I have fixed up the confidence images in the UDS. This does not change the
errors in the catalogues as I expected, but does increase the number of
objects detected."
MAR reported:
"Mainly worked on the SQL scripts/stored procedures for implementing
a query queue. Also progressed the jsp pages/forms that'll allow users
to construct querys with the external databases. Will need to get the
parsing script changed to produce different names for the checkboxes
in the WFCAM and external tables."
Survey Data Release:
No news this week; see Project Management above concerning release
schedule etc. for DR1.
Non-survey Data Release:
Nothing to report this week
Astrogrid deployment:
USNOB has been reinstated for the time being for AG applications software.
Local AG folks have upped the row limit in the DSA/PAL applications
accessing the WFAU database servers. Finally, no more news on a visit
from Guy.
Miscellaneous:
NCH noted that the UKIDSS session at NAM was well atended and went
very well, finishing of with a good archive demo from RSC.
JB noted that it may be a good idea to raise the profile of the
WSA within the University by incorporating the WSA home page into
the MyEd portal.
ACTION: JB to see that the WSA homepage is incorporated into the
MyEd portal.