From j.p.emerson@qmul.ac.uk Thu Aug 14 14:58:41 2003 Date: Thu, 14 Aug 2003 14:55:12 +0100 From: Jim Emerson To: Malcolm Stewart , Mike Irwin , Richard McMahon , Simon Hodgkin Subject: FW: WFAU WSA weekly meeting minutes: 8th August 2003 for information Minutes of WFCAM Science Archive meeting: 8th August 2003 --------------------------------------------------------- --------------------------------------------------------- Present: AL, JMS, IAB, NCH, MAR, RGM, ETWS, PMW Apologies: HMG Actions discharged: ------------------- ACTION: NCH to get advice from JNTD about Windows terminal services client access to C1 servers. (shouldn't be any problems, in principle) ACTION: IAB to talk to experienced local software gurus (eg. AJV, MCH, GPS) about SOAP and its use for curation remote client services. (experiments with SOAPpy look very good; client/server curation to be implemented using this). ACTION: NCH to contact eGenix and inquire as to the licensing requirements for our use of eGenix Python middleware. (eGenix have informed us that a free licence will cover our use of mxODBC, mxDateTime, etc.) ACTION: NCH to register, and submit SSA abstract, for ADASS XIII. (NCH has registered and submitted an SSA abstract for an oral presentation). - all done and discharged (thanks to all concerned!) Actions partly discharged but continuing: ----------------------------------------- ACTION: ALL to review risk register and think of any new internal or external risks that should be documented in the risk register. - continuing; thanks for progressing these. Actions carried forward from 1/08/03 meeting: ---------------------------------------------- ACTION: ETWS to update TWiki networking pages with new test results. ACTION: PMW to liaise with AL to make sure that Compute Support are made aware, from the highest "official" level, of our requirements, re. 100 Gbyte/day transfer from CASU, and to impress on them that the 1 Gbit/s FW should be their highest priority. - CONTINUES Specific points and new actions: -------------------------------- Project management: NCH & PMW attended the monthly VDFS management meeting. NCH suggested that it would be good if these meetings were consistently minuted and the minutes available; more generally it would help a great deal if there was a central set of web pages where this and other information (eg. VDFS proposals etc) where held for access by the entire project membership. NCH felt that it would be particularly good if there was a "business plan" in place that illustrates the big picture (as it currently stands) to aid in planning work. ACTION: NCH to email JPE to suggest the ideas of minutes and centralised web pages of the VDFS management group. PMW also noted that there was now some uncertainty over the exact grant mechanism since PPARC were now refering to the WFAU RG as a renewable, rather than a rolling, grant. The team is of course particularly concerned about the likely timescale for submitting renewals, and fitting proposal preparation into the work plans. ACTION: PMW to contact PPARC to enquire as to the likely requirements on WFAU concerning renewing the WF grant in future. Hardware: NCH reported that Eclipse would finish setting up the new hardware today. An LTO 1/2 compatibility test had been successfully completed: We wrote 5 Gbytes on a Seagate LTO-1 drive, Maxell tape. We then loaded the tape into our Overland storage ("NEO series") LTO-2 30-slot library, (HP drive) and read the tape with no problems at all. Operating system in both cases was Red Hat linux. So, no problems so far; the LTO technology is great. The Overland library system is really compact, and easy to use (at least under Linux). PMW reported that he has added the new kit to the IfA equipment inventory for insurance purposes, etc. Software: IAB reported: "I have been looking into SOAP for the internal network programming functionality that we will require. I have tried out the Python SOAPpy modules and it is a very straightforward procedure to set up simple functions for remote procedure calls. More elaborate functionality could be added if required. Using threads appears an effective way of dealing with the issue of remotely calling procedures that take a long time to finish. The server provides a function that runs the required task in a different thread and returns telling the client that the thread has started. The network connection does not need to be up while the thread is running. The server provides another another function to the client which polls the thread asking if it has finished. A test run using a Linux client and Windows server worked well." NCH suggested that we now implement CU16 as a test case using SOAPpy, including robust handling of exceptions and communication of captured log information from the catalogue server side. ACTION: IAB to implement a client/server test of CU16, including log info communication and robust exception handling. MAR reported: "This week mainly worked on the SSA xmatching/file upload interface. Which appears to be working. So now pretty much have radial, SQL by menu, SQL raw and XMatch all working. The first three run under one servlet, the XMatch sits under a separate one as the file upload is handled a bit differently as is the SQL formation. Code and html needs tidying up." NCH suggested that the software structure (as regards repository storage, and installation within the deployed system) needs some attention - the team needs to agree a modular hierarchy now that three of us (NCH, IAB & ETWS) are all working on curation software. ACTION: NCH, ETWS and IAB to meet to finalise the software system modular hierarchy before ETWS goes on leave (Thurs 14th). MAR asked about the setup of the web server as regards tomcat etc. ACTION: MAR to liaise with HME concerning set-up of the WSA web server AL asked whether the Perl/Python debate was now settled; the team informed him that we were now progressing implementation using Python and that this has been communicated to all concerned (via these minutes) with no more objections coming back. Networking: ETWS reported: "I've written the 2MASS SQL Schemas and filled in the UCDs. The tables can be found in the WSA CVS. Also I've downloaded the 2MASS data from IPAC (Infrared Processing and Analysis Center/California Institute of Technology). The transfer rate was 1.3MB/s which is three times the rate I've had for the EDR data. The used script is the first raw version of CU1." AL asked about the team's attitude to GridFTP. Consensus was that we should go with what works, write a concise report of how we've achieved the required performance, and move on. Team does not have time to indulge in open-ended testing of GridFTP. MAR suggested that he contacts IPAC to let them know we intend to server 2MASS data, and to check if that is OK with them. (Offline: RGM emphasised the need to adhere to the WSA "no NULLs policy" which means that we need to choose default values for NULL fields in 2MASS data and encode these into the 2MASS schema). ACTION: MAR to contact 2MASS concerning ingestion and serving of their data in the WSA. SSA: NCH reported that no progress has been made on loading since it has taken all week for Eclipse to set up the new hardware. A 1 Gb/s net card was being investigated to aid in the transfer of the 1 Tbyte of intermediate ingest data from the SSS server cosaxp6. AL emphasised the need to set short term goals for deployment of the SSA. Team agreed that we should aim for a working system at the end of August. NCH asked about documentation and indexing for the SSA. MAR suggested that he and ETWS take care of linking together the schema and access pages; RGM hopes to get on to indexing next week. In the meantime, NCH is to progress server & DB set-up, RAID configuration and loading. NCH asked RGM about the licence keys for Windows 2003 AS (Enterprise). ACTION: NCH & RGM to get the required keys for W2003 AS. Miscellaneous: ADASS XIII: October 12-15th. Team again discussed attendance at this; consensus was that RGM & NCH attending was good enough and that NCH should give the WSA and wider picture a good plug in his SSA presentation. ACTION: NCH to type up and circulate these minutes. DONM: 10am, Friday 15th August, plate library.