From: Nigel Hambly [mailto:nch@roe.ac.uk]
Sent: 01 August 2003 13:03
To: WFCAM Science Archive Team -- Clive Davenhall; Eckhard Sutorius;
Harvey MacGillivray; Ian Bond; Mike Read; Nigel Hambly; Bob Mann
Cc: Andrew Lawrence; Peredur Williams; Andy Adamson; Jim Emerson
Subject: WFAU WSA weekly meeting minutes: 1st August 2003
Minutes of WFCAM Science Archive meeting: 1st August 2003
---------------------------------------------------------
---------------------------------------------------------
Present: HMG, JMS, IAB, NCH, MAR, RGM, ETWS, PMW
Apologies: AL
Actions discharged:
-------------------
ACTION: PMW to draft an action plan in the light of the review panel
report.
ACTION: RGM to look over the current version of the SSA UCDs in the
schema scripts in the CVS.
ACTION: NCH to start checking in the CU16 codes into the CVS.
ACTION: PMW to approach compute support to request that MAR & ETWS be
allowed NT administrator and Unix root priviledge on WSA
systems.
ACTION: NCH & IAB to prototype client/server functionality using CU16.
ACTION: ETWS to set up a TWiki page and all to edit in required software
components/URLs and any specific set-up parameters as they crop
up.
- all done and discharged (thanks to all concerned!)
Actions partly discharged but continuing:
-----------------------------------------
ACTION: ALL to review risk register and think of any new internal or
external risks that should be documented in the risk register.
- continuing; thanks for progressing these.
Actions carried forward from 25/07/03 meeting:
----------------------------------------------
ACTION: PMW to liaise with AL to make sure that Compute Support are
made aware, from the highest "official" level, of
our requirements, re. 100 Gbyte/day transfer from CASU, and to
impress on them that the 1 Gbit/s FW should be their highest
priority.
- CONTINUES
Specific points and new actions:
--------------------------------
Project management:
The monthly VDFS management group meeting takes place next week. PMW has
assembled the progress report, and has produced a finer-grained version
of the task breakdown against which progress is measured. Several tasks
have been completed to 100% for July so the report looks good; the priorty
for the next month is to achieve a working SSA.
Note added after meeting: PPARC have informed us that the WFAU Grant
announcement is in the post (finally).
Hardware:
Eclipse are currently in setting up the new catalogue and web server
hardware & infrastructure. NCH suggested the cabinet layouts; some
problems experience with the neceassry bolts (?!) to rackmount the
units, but the work will be completed early next week (the experimental
server "eclipse" has been removed to be reconfigured). At which point
RAID sets will be configured and SSA loading can begin.
PMW reported that approaches to Computing Support about admin/root status
for a coupple of named WSA team members received a positive response on
the Windows side, and a sympathetic, but guarded response from the Linux
side. HMG suggested that it was important the HME sets up the web server
and associated internal firewall; a more flexible system for the linux
mass storage system (a la cosaxp6) was envisaged as workable.
NCH asked the team about remote network access to C1 servers; this should
be possible in principle but needs looking into.
ACTION: NCH to get advice from JNTD about Windows terminal services client
access to C1 servers.
Software:
IAB reported:
"Following the Astro-WISE philosophy, I have designed classes to
represent the passbands, time stamps, and curation use case ids.
The idea is that all representations (integers, strings etc) and
descriptions are encasulated in the classes. These are communicated
amongst the different modules and scripts by passing these as
objects (ie instances of these classes).
The FitsReader.py module has been replaced by a new module
sciencefile.py. This defines a ScienceFile class that represents
a WFCAM FITS file which can contain pixel and/or catalogue data.
I tested out XML-RPC between Windows and Linux. A client running
on Linux was able to successfully call functions provided by a server
running on the Windows side."
The prototyped linux client / Windows server curation
connection using XMLRPC and Python seems to work well; NCH
asked the team if we should also look into SOAP as an alternative
to give as experience in VO-compliant web service implementation.
Consensus was that we should ask advice from experienced locals
before making a decision.
ACTION: IAB to talk to experienced local software gurus (eg. AJV,
MCH, GPS) about SOAP and its use for curation remote
client services.
NCH has tested CU16 catalogue load server Python and SQL codes and
committed them to the CVS. The problem over the connection timeouts
and other quirks of PyMSSQL have been circumvented by switching to
a different DB interface module, namely mxODBC. This is far superior,
standard across different DBMS (eg. Oracle & DB2) and does not have
the connection limitations of PyMSSQL. The only issue is about
licensing for this higher quality middleware.
ACTION: NCH to contact eGenix and inquire as to the licensing requirements
for our use of eGenix Python middleware.
MAR reported:
"Re-factored some of the servlet code to separate out the formation of the
SQL statements. Worked on the JSP pages for the menu driven access."
ETWS has set up a software installation TWiki page, and the team are
editing in all the off-the-shelf software components that are being
download (along with URLs and any special installation requirements)
as the archive systems are being prototyped.
Networking:
ETWS reported:
"I've tested two download accelerator programs, prozilla and axel.
They both uses multi-threading via an ftp connection. The transfer rates
reached with both programs are up to 7MB/s. But the drawback of the
commandline version of prozilla is, that the downloaded file has to be
rebuilt from the downloaded parts which took about the same time as the
downloading, so the overall rate was again 4 MB/s.
Whereas axel loads all parts in the right places in the file since Unix
can write after EOF marks.
As a followup I've written a python script which controls the download
according to the file size since we may have as well very small as very
big files in the archive at CASU. I'm testing it for the best threshold
for single or multi threaded download."
RGM suggest ETWS should update the TWiki notes as to the results of all
the various network tests he has been doing - these may well be very
useful for external parties (eg. AstroGrid folks).
ACTION: ETWS to update TWiki networking pages with new test results.
SSA:
NCH has implemented a create-database script and committed to the CVS.
This includes details of SQL Server filegroups, their striping across
logical volumes, and the organisation of the RAID LVs themselves. NCH
requested that RGM & ETWS look over this and discuss with him after
this meeting.
Miscellaneous:
ADASS XIII: October 12-15th. Team discussed attendance at this; RGM
suggested a strong presence for the SSA (oral contribution) and
WSA (poster?) would be desirable. NCH volunteered to attend and
offer an oral presentation on the SSA. PMW informed the team that
we could afford to send two people; RGM is attending anyway.
ACTION: NCH to register, and submit SSA abstract, for ADASS XIII.
ACTION: NCH to type up and circulate these minutes.
DONM: 10am, Friday 8th August, plate library.