From: Nigel Hambly [mailto:nch@roe.ac.uk] Sent: 01 August 2003 13:03 To: WFCAM Science Archive Team -- Clive Davenhall; Eckhard Sutorius; Harvey MacGillivray; Ian Bond; Mike Read; Nigel Hambly; Bob Mann Cc: Andrew Lawrence; Peredur Williams; Andy Adamson; Jim Emerson Subject: WFAU WSA weekly meeting minutes: 1st August 2003 Minutes of WFCAM Science Archive meeting: 1st August 2003 --------------------------------------------------------- --------------------------------------------------------- Present: HMG, JMS, IAB, NCH, MAR, RGM, ETWS, PMW Apologies: AL Actions discharged: ------------------- ACTION: PMW to draft an action plan in the light of the review panel report. ACTION: RGM to look over the current version of the SSA UCDs in the schema scripts in the CVS. ACTION: NCH to start checking in the CU16 codes into the CVS. ACTION: PMW to approach compute support to request that MAR & ETWS be allowed NT administrator and Unix root priviledge on WSA systems. ACTION: NCH & IAB to prototype client/server functionality using CU16. ACTION: ETWS to set up a TWiki page and all to edit in required software components/URLs and any specific set-up parameters as they crop up. - all done and discharged (thanks to all concerned!) Actions partly discharged but continuing: ----------------------------------------- ACTION: ALL to review risk register and think of any new internal or external risks that should be documented in the risk register. - continuing; thanks for progressing these. Actions carried forward from 25/07/03 meeting: ---------------------------------------------- ACTION: PMW to liaise with AL to make sure that Compute Support are made aware, from the highest "official" level, of our requirements, re. 100 Gbyte/day transfer from CASU, and to impress on them that the 1 Gbit/s FW should be their highest priority. - CONTINUES Specific points and new actions: -------------------------------- Project management: The monthly VDFS management group meeting takes place next week. PMW has assembled the progress report, and has produced a finer-grained version of the task breakdown against which progress is measured. Several tasks have been completed to 100% for July so the report looks good; the priorty for the next month is to achieve a working SSA. Note added after meeting: PPARC have informed us that the WFAU Grant announcement is in the post (finally). Hardware: Eclipse are currently in setting up the new catalogue and web server hardware & infrastructure. NCH suggested the cabinet layouts; some problems experience with the neceassry bolts (?!) to rackmount the units, but the work will be completed early next week (the experimental server "eclipse" has been removed to be reconfigured). At which point RAID sets will be configured and SSA loading can begin. PMW reported that approaches to Computing Support about admin/root status for a coupple of named WSA team members received a positive response on the Windows side, and a sympathetic, but guarded response from the Linux side. HMG suggested that it was important the HME sets up the web server and associated internal firewall; a more flexible system for the linux mass storage system (a la cosaxp6) was envisaged as workable. NCH asked the team about remote network access to C1 servers; this should be possible in principle but needs looking into. ACTION: NCH to get advice from JNTD about Windows terminal services client access to C1 servers. Software: IAB reported: "Following the Astro-WISE philosophy, I have designed classes to represent the passbands, time stamps, and curation use case ids. The idea is that all representations (integers, strings etc) and descriptions are encasulated in the classes. These are communicated amongst the different modules and scripts by passing these as objects (ie instances of these classes). The FitsReader.py module has been replaced by a new module sciencefile.py. This defines a ScienceFile class that represents a WFCAM FITS file which can contain pixel and/or catalogue data. I tested out XML-RPC between Windows and Linux. A client running on Linux was able to successfully call functions provided by a server running on the Windows side." The prototyped linux client / Windows server curation connection using XMLRPC and Python seems to work well; NCH asked the team if we should also look into SOAP as an alternative to give as experience in VO-compliant web service implementation. Consensus was that we should ask advice from experienced locals before making a decision. ACTION: IAB to talk to experienced local software gurus (eg. AJV, MCH, GPS) about SOAP and its use for curation remote client services. NCH has tested CU16 catalogue load server Python and SQL codes and committed them to the CVS. The problem over the connection timeouts and other quirks of PyMSSQL have been circumvented by switching to a different DB interface module, namely mxODBC. This is far superior, standard across different DBMS (eg. Oracle & DB2) and does not have the connection limitations of PyMSSQL. The only issue is about licensing for this higher quality middleware. ACTION: NCH to contact eGenix and inquire as to the licensing requirements for our use of eGenix Python middleware. MAR reported: "Re-factored some of the servlet code to separate out the formation of the SQL statements. Worked on the JSP pages for the menu driven access." ETWS has set up a software installation TWiki page, and the team are editing in all the off-the-shelf software components that are being download (along with URLs and any special installation requirements) as the archive systems are being prototyped. Networking: ETWS reported: "I've tested two download accelerator programs, prozilla and axel. They both uses multi-threading via an ftp connection. The transfer rates reached with both programs are up to 7MB/s. But the drawback of the commandline version of prozilla is, that the downloaded file has to be rebuilt from the downloaded parts which took about the same time as the downloading, so the overall rate was again 4 MB/s. Whereas axel loads all parts in the right places in the file since Unix can write after EOF marks. As a followup I've written a python script which controls the download according to the file size since we may have as well very small as very big files in the archive at CASU. I'm testing it for the best threshold for single or multi threaded download." RGM suggest ETWS should update the TWiki notes as to the results of all the various network tests he has been doing - these may well be very useful for external parties (eg. AstroGrid folks). ACTION: ETWS to update TWiki networking pages with new test results. SSA: NCH has implemented a create-database script and committed to the CVS. This includes details of SQL Server filegroups, their striping across logical volumes, and the organisation of the RAID LVs themselves. NCH requested that RGM & ETWS look over this and discuss with him after this meeting. Miscellaneous: ADASS XIII: October 12-15th. Team discussed attendance at this; RGM suggested a strong presence for the SSA (oral contribution) and WSA (poster?) would be desirable. NCH volunteered to attend and offer an oral presentation on the SSA. PMW informed the team that we could afford to send two people; RGM is attending anyway. ACTION: NCH to register, and submit SSA abstract, for ADASS XIII. ACTION: NCH to type up and circulate these minutes. DONM: 10am, Friday 8th August, plate library.