From nch@roe.ac.uk Fri Oct 27 14:54:51 2006
Date: Fri, 27 Oct 2006 14:32:10 +0100 (BST)
From: Nigel Hambly <nch@roe.ac.uk>
To: WFCAM Science Archive Team -- Eckhard Sutorius <etws@roe.ac.uk>,
Johann Bryant <jb@roe.ac.uk>, Mike Read <mar@roe.ac.uk>,
Nigel Hambly <nch@roe.ac.uk>, Nicholas Cross <njc@roe.ac.uk>,
Bob Mann <rgm@roe.ac.uk>, Ross Collins <rsc@roe.ac.uk>
Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence
<al@roe.ac.uk>, Andy Adamson <a.adamson@jach.hawaii.edu>,
Brian Walshe <bcw@roe.ac.uk>, John Taylor <jdt@roe.ac.uk>,
Jim Emerson <j.p.emerson@qmul.ac.uk>, Malcolm Stewart <jms@roe.ac.uk>,
Mike Irwin <mike@ast.cam.ac.uk>, Mark Holliman <msh@roe.ac.uk>,
Peredur Williams <pmw@roe.ac.uk>, Stephen Warren <s.j.warren@ic.ac.uk>
Subject: WFAU Science Archive weekly project meeting minutes 27/10/06
Minutes of WFAU VDFS Science Archive meeting: 27th October 2006
-------------------------------------------------------------------------
-------------------------------------------------------------------------
Present: NCH, MAR, JB, PMW, BCW, NJC, ETWS, RSC, JMS
Apologies: JPE, RGM, JDT, MSH, AL
DONM: 10am, Friday 3rd November 2006 in the Plate Library
Actions discharged this week:
-----------------------------
d) Debug CU7 for deprecated frame sets and deletions on rerun
- ACTION: NCH
Dropped as very low priority (for DR2) since sources to be
remerged from scratch given requirements from UKIDSS for
quality bit flagging and photometric recalibration.
k) Ensure ingest code can cope with 06A missing data quirks and any
associated new attributes
- ACTION: ETWS
Dropped since this will happen over the next few weeks anyway
as we ingest 06A.
ACTION: NCH to read the CASU overview and pipeline paper documents
Discharged
ACTION: NJC (on behalf of NCH) to read the calibration document
Discharged
ACTION: NCH to forward the review comments to RSC, MAR, RGM and
PMW for their perusal.
Discharged; NCH thanks all for their input into the responses
for those comments.
Actions partly discharged but continuing:
-----------------------------------------
The following from last week partly done but continue:
ACTION: JB to rearrange external catalogues onto batch catalogue
server
Actions carried forward from 18/10/06 meeting:
----------------------------------------------
The following from last week continue:
i) Enhance seaming to use quality bit information
- ACTION: RSC
See Software below.
o) Add in a default row for every detector appearing in every detection
table (for schema consistency when querying merged sources and
individual detections)
- ACTION: RSC & NCH
ACTION: NCH to circulate a release schedule for DR2 to all
concerned.
- CONTINUES; see Survey Release below.
ACTION: AL to email Peter Clarke in NeSC to see if a little pressure
from above helps to expedite implementation of UKLight at
the EUCS end.
- CONTINUES
ACTION: JB and ETWS to scrub the constituent frames of interleaves
for archived UKIDSS data only (where it has not been done
already).
- CONTINUES
See Operations below.
ACTION: NCH to email Jan at JHU to enquire about this (and
database file sizes at the same time).
- CONTINUES
Specific points and new actions:
--------------------------------
Project management:
NCH reported back from the VDFS FDR, where the general impression
seemed to be very good. We are anticipating written feedback
soon.
PMW presented the Q4 plan, and NCH emphasised the need to
prioritise those bits of development needed for UKIDSS DR2.
WFCAM update:
Nothing to report this week
Comments and issues arising from CASU fortnightly minutes:
No new minutes this week; however with reference to the
recalibration requirements identified previously, the
team discussed some implementation details and it became
clear that we need the information and procedural details
at the archive end as soon as possible in order to
expedite the DR2 schedule.
ACTION: NJC to contact STH directly concerning photometric
recalibration of 05A, 05B and 06A.
NCH further reported a highly conciliatory chat with MJI and
RGMcM on Wednesday at CASU concerning their worries over
timelags between processed data being made available and
flat file access being available at the archive end. A
review of software and procedures has been instigated at
the archive end to address these concerns (see Software
below).
Networking:
NCH noted that on the external catalogue front, we have
again received offers of help from Portsmouth with SDSS-DR5;
a request for inclusion of the NVSS radio catalogue in the
archive suite of external datasets; and an offer of the
current version of GSC-II from STScI on a FireWire disk.
ETWS reported that attempts were still being made to
copy DR5 from Chicago using the new "Sector" software,
but that if these come to nothing over the next few weeks
then that might be the time to take Portsmouth up on their
offer; NVSS can go in in advance of DR2 without much
trouble; and finally given the local availability of the
SSA and the fact that no users have expressed the need
for GSC-II, we should not worry about that one.
ACTION: ETWS & MAR to include NVSS into the WSA external
catalogue suite, but as a very low priority and
without bugging JB.
JB reported:
"We have received a reply from EUCS about the diplexers and they are to be
tested soon, they apologised for the delay. UKLight is still to deliver a
piece of hardware to EUCS that will be needed."
ACTION: NCH to prod AL to prod Peter Clark concerning this
frustrating lack of progress from the UKLighters.
WSA Operations:
JB reported:
"Backups continue as normal with the exception of khafre (due to it's
lockups) though plans for new backups of UKIDSS data were scuppered due to
illness.
CU4 runs of 05Av3 are continuing (with a delay caused by khafre having
problems), likewise CU1 runs. CU21 (for Non-Surveys) has been run again
to update opur list of Non-Surveys with two surveys waiting on an 05Av3
CU4 run to finish before they are updated (the others don't need updated
as they are from 06A and 06B).
A couple of difficult to find and curious bugs were run to ground with the
help of ETWS and RSC.
Discussions abou the Operator environment and ways of improving it and
efficiency have been soundly discussioned. My thanks to ETWS for the
speedy implementation of some of these. Other imporvements are being
considered or in the pipeline."
Hardware:
JB reported:
"The shutdown due to the planned power outage over last weekend caused a
curfuffle with khafre, it hard locked when it ran it's backup on Monday
night, it is now being monitored having had everything we could think of
checked but we won't know more certainly until next Tuesday if there is
still anything wrong with it. This is a problem as most of the data we
are currently working with and the free disk space is on khafre's disks.
Ahmose's slow periods seem to still be causing a problem but they may be
due to Thutmose, I have unlinked the two servers in the mean time to see
if this has an effect.
Options for the NAS were considered, the maintenance costs of the current
solution as opposed to NAS continue to make themselves known, in this case
problems encountered in shutting down the machines before the weekend
power-down."
The team made the decision to order an 8TB NAS brick to service
storage requirements.
ACTION: PMW to order an 8TB NAS storage brick.
Software:
RSC reported:
"Software/Operations:
* Improved CU16 to produce neighbour tables for specific external
surveys, and/or specify neighbour tables.
* Ran CU16 for XMM neighbour tables.
* Recreated TRANSIT survey neighbour tables.
Reports:
* Completed investigation into large FITS file support on 32-bit PCs, and
64-bit support. Full report and summary may be found in TWiki article
"LargeFileSupport".
* Prepared poster contribution for ADASS based on large-file support
investigation. See "TalksPostersPapers" TWiki article (also merged in
Nigel's talks from "TalkSlides").
* Attended ADASS conference, and am writing a report based on my notes
from ADASS; it will be found in the TWiki article "ReportOnADASS2006".
* Attended VDFS FDR meeting, and have put the review comments on to the
"ProjectManagement" TWiki page for future reference.
* Created TWiki page "StaffAbsences" so we will all know in advance when
other team members will be away."
ACTION: All to fill in their absences over the next few months
on the TWiki.
MAR reported:
"Worked on a more "streaming" web access via sql query so that users
can submit queries less interactively i.e. using wget and pipe/save the
results on the fly. Not sure if it'll be affected by timeout issues.
Also worked on MultiGetimage to see how if it can be made to work readily
with programmes that don't have source tables."
ETWS reported:
"Simplified scos' login and started software update on djoser.
Created xml metadata files of the DR1 for Astrogrid and created an
update of our browser pages."
NCH emphasised the need to concentrate on those developments
necessary for DR2 at the moment. The main issues are i) expediting
CUs 1-4 to enable fast turn-around on flat file access and get
the 06A data ingested as quickly as possible; ii) designing and
implementing some archive end quality bit flagging (in collaboration
with SJW) and iii) implementing IAU names in the database.
ACTION: ETWS, JB & NCH to meet on Monday 2pm to sort out current
software and operation modes for CUs 1-3 in order to
optimise and expedite flat-file turn around.
ACTION: RSC to profile CU4 and investigate possible optimisation
ACTION: NCH, MAR and RSC to meet Monday 4pm to finalise design
of archive end quality bit flagging for DR2.
ACTION: MAR to implement a DB stored procedure to encapsulate
the production of IAU names based on queried table and
celestial co-ordinates.
Survey Data Release:
NCH noted that SJW and he had set a deadline of 25th of October
for any further gripes from UKIDSS survey heads over 05Av3
processing; no further worries have been raised so from that
point of view it's all systems go for DR2 release of the
current processed versions of 05A, 05B and 06A.
A large part of this week's meeting was taken up with a discussion
over the DR2 schedule. From the archive curation point of view,
the cumulative time (in weeks) required from now is estimated
as follows:
Complete transfer of 06A : +2
" ingest " " : +6 (CU4 for GPS is the bottleneck)
Photometric recal : Critical - we don't as yet have the info
from CASU about this, cannot start
QC1 until we do, and it's impact on CU4
is not known.
Quality bit flagging : +1 (can design & test in parallel with
previous)
QC1 : +1 (but cannot start 05A and early 06A
until photometric recalibration is done;
do we need to redo 05B owing to photometric
recalibration?!)
CU7 (source merging) : +3 (again totally dominated by GPS, and
has to be done from scratch because of
q-bits and recal)
DXS CUs : not on the critical path, since much
can be done in parallel with previous;
later CUs run fast on the relatively
small amount of data)
CU16 : +1
QC2 : +1
Final CUs : +3
... adding in a couple of weeks for Xmas leaves us with the
worryingly high figure of 20weeks from now. The major bottleneck
is likely to be CUs 4 & 7 for the GPS (assuming delivery of
photometric recalibration does not hold up QC1, and QC1 itself
does not undergo any revision over previous procedures). It is
possible that some operational and software mods can improve
the CU4 & 7 bottlenecks... investigations are in progress.
See elsewhere in these minutes for relevant actions on specific
individuals.
Non-survey Data Release:
JB noted that several new (late) 05B registrations had come in
and that flat file access was available for them; MAR will
inform the PIs.
Astrogrid deployment:
THe first attempt at a secure DSA was implemented over the last
week or so, and RGM, NCH and MAR all tested it and found the
same bug that prevent some authorisation layer (?) from
working. Details were passed back to the AG development team,
and work is on-going to try to fix in advance of the 7th
November review panel meeting.
Miscellaneous:
Nothing else this week.