From nch@roe.ac.uk Fri Dec 12 17:55:00 2003
Date: Fri, 12 Dec 2003 13:49:14 +0000 (GMT)
From: Nigel Hambly <nch@roe.ac.uk>
To: WFCAM Science Archive Team -- Eckhard Sutorius <etws@roe.ac.uk>,
Harvey MacGillivray <hmg@roe.ac.uk>, Ian Bond <iab@roe.ac.uk>,
Mike Read <mar@roe.ac.uk>, Nigel Hambly <nch@roe.ac.uk>,
Bob Mann <rgm@roe.ac.uk>
Cc: CCs for WSA weekly meeting minutes distribution -- Clive Davenhall
<acd@roe.ac.uk>, Andrew Lawrence <al@roe.ac.uk>,
Andy Adamson <a.adamson@jach.hawaii.edu>, Peter Shillan <gps@roe.ac.uk>,
John Taylor <jdt@roe.ac.uk>, Jim Emerson <j.p.emerson@qmul.ac.uk>,
Martin Hill <mch@roe.ac.uk>, Mike Irwin <mike@ast.cam.ac.uk>,
Peredur Williams <pmw@roe.ac.uk>
Subject: WFAU WSA weekly meeting minutes: 11 December 2003
Minutes of WFCAM Science Archive meeting: 11th December 2003
------------------------------------------------------------
------------------------------------------------------------
Present: NCH, ETWS, MAR, IAB, PMW, RGM, JDT
Apologies: JPE, HTMG, JMS, AL, MCH, GPS
Actions discharged:
-------------------
ACTION: PMW to check out the time/place of the next INT-WFC Ha survey
meeting and attend if possible.
ACTION: NCH to write a TWiki report, and IAB & ETWS to contribute to
this, concerning the two day visit to Astro-WISE at Groningen.
Actions partly discharged but continuing:
-----------------------------------------
ACTION: IAB to check that all necessary SW for curation client
functionality is present on djoser.
ACTION: ALL to review risk register and think of any new internal or
external risks that should be documented in the risk register.
Defered until top level plan is explicitly defined. PMW tabled
the VISTA risk register as an example.
ACTION: RGM to progress design of a web cookbook based around the 20
Queries.
ACTION: JPE to design and set up centralised VDFS web pages. Progressing.
(there is now a link to it from our WSA Twiki).
ACTION: NCH to stage intermediate Source files in thoth, and email
Francois Oschenbien at CDS when the first file(s) is/are ready
for transfer.
- continuing; thanks for progressing these.
Actions from 5/12/03 meeting:
------------------------------
Specific points and new actions:
--------------------------------
Project management:
Correction to the last minutes:
"PMW updated the team with the latest funding situation, where the GSC..."
GSC should read PPRP (Project Peer Review Panel).
NCH reported that Andy Adamson had visited yesterday and had been given
an update on the status of the WSA project, including a verbal progress
report against the top level plan (VDFS Phase A) and a demo of work
done, user interfaces, and also shown the hardware. AA seems happy with
progress, but has suggested that it would be good to have a regularly
updated progress summary on the website along with the GANTT to keep an
eye on progress on timescales longer than weekly (as detailed in these
minutes).
Comments and issues arising from CASU fortnightly minutes:
(no meeting since last WFAU meeting).
Hardware:
NCH reported some suspicious hardware errors on the SATA RAID arrays on
djoser, in the form of error warnings from the 3ware 3DM monitoring
software. NCH suggested that if this is due to a manufacturing problem
in the immature SATA technology (ie. like the previous Seagate firmware
bugs) then this must be identified as soon as possible, and if
necessary we should fall back on ATA-133.
ACTION: ETWS (in consultation with NCH & HMG) to look into the djoser
SATA RAID errors.
ETWS reported further network tests with djoser:
"I've tested datatransfer rates from djoser to Cambridge again, downloading
about 4.8 GB of data. scp parameters were changed by a "config" file in
"~/.ssh". The performance of the default compressed download (Compression
level 6) was the same as without compression, about 3MB/s. Decreasing the
compression to a level of 2 to 4 (no real difference between them) gives a
transfer rate of ~6MB/s, the same as with ftp.
Also I've tested the internal network downloading a 2.9GB file from
srif112 to ericht and then uploading it to djoser. Here the best option is
using the cipher arcfour (only available in ssh2, but blowfish is in the
same league) and NO compression. With this settings it was possible to
reach a transfer rate of 10.8 MB/s.
Debian linux tcsh shows the same problems with large files (>2GB) as
RedHat, so if one wants to create large files (copying is fine!) one has
to use bash instead."
Software:
NCH reported that Samba SMBFS seems to have a 2 Gbyte file size limit as
installed at the moment; furthermore the fileshare between the linux
and Windows servers is not writable from the linux side.
ACTION: NCH to sort out write access to the linux/Windows file shares
between the WSA servers.
IAB reported good progress with the ingest tests. MJI generated first-cut
WFCAM FITS files which were checked against the WSA ICD and schemas; minor
schema typos etc have been corrected and now the FITS data pass initial
ingest parser filtering. MJI has generated a batch of FITS files for
ingest. IAB reported that a few procedural issues need to be resolved
concerning attributes computed at the time of ingest; then a database load
can be tested. NCH added that the curation client / DB server file share
between djoser and the load server is not quite ready (see above).
SSA:
NCH reported that neighbour-table creation in the SSA wsa still giving
problems (suspect that the high row counts are the source of the problems)
and workaraounds are ongoing.
RGM is investigating simplified indexing schemes on amenhotep to give a
baseline indexed full-blown SSA to attach to the web interface.
Miscellaneous:
Nothing else this week.
DONM: 10am, Wed 7th January 2004, plate library.