From j.p.emerson@qmul.ac.uk Fri Aug 15 14:39:29 2003 Date: Fri, 15 Aug 2003 12:26:49 +0100 From: Jim Emerson To: Malcolm Stewart , Mike Irwin , Richard McMahon , Simon Hodgkin Subject: FW: WFAU WSA weekly meeting minutes, 15 August 2003 Minutes of WFCAM Science Archive meeting: 15th August 2003 ---------------------------------------------------------- ---------------------------------------------------------- Present: AL, IAB, NCH, MAR, RGM, PMW, HMG Apologies: JMS, ETWS Actions discharged: ------------------- ACTION: NCH to email JPE to suggest the ideas of minutes and centralised web pages of the VDFS management group. (Positive reponse from JPE; he will attend to this on return from holiday at the beginning of September). ACTION: PMW to contact PPARC to enquire as to the likely requirements on WFAU concerning renewing the WF grant in future. (Email sent to Colin Vincent; not expecting immediate response) ACTION: IAB to implement a client/server test of CU16, including log info communication and robust exception handling. (Wrapper and server modules now coded up and tested locally). ACTION: NCH, ETWS and IAB to meet to finalise the software system modular hierarchy before ETWS goes on leave (Thurs 14th). (met up on Mon; new hierarchy proposed and noted on TWiki) ACTION: MAR to liaise with HME concerning set-up of the WSA web server (HME is now attending to this) ACTION: MAR to contact 2MASS concerning ingestion and serving of their data in the WSA. (IPAC contacted via Tom Jarret; very positive response - happy to let us ingest 2MASS and serve; and answer any technical questions). ACTION: NCH & RGM to get the required keys for W2003 AS. (no problems; keys now with CDs and server names noted against keys). - all done and discharged (thanks to all concerned!) Actions partly discharged but continuing: ----------------------------------------- ACTION: ALL to review risk register and think of any new internal or external risks that should be documented in the risk register. - continuing; thanks for progressing these. Actions carried forward from 8/08/03 meeting: ---------------------------------------------- ACTION: ETWS to update TWiki networking pages with new test results. ACTION: PMW to liaise with AL to make sure that Compute Support are made aware, from the highest "official" level, of our requirements, re. 100 Gbyte/day transfer from CASU, and to impress on them that the 1 Gbit/s FW should be their highest priority. - CONTINUES Specific points and new actions: -------------------------------- Project management: Nothing new this week Hardware: NCH suggested we should now look at the plan for procuring a small part of the mass storage system for set-up, tests and integration. HMG informed the meeting that new inexpensive IDE/ATA technology was available that may be better than the existing specs. ACTION: HMG to communicate latest info concerning inexpensive disk storage, and NCH, HMG & PMW to discuss offline the next hardware acquisition. NCH reported that initial tests of the new catalogue servers had produced adequate results, but nothing spectacular. Likely that some optimisation of hardware set-up is likely. The meeting agreed that the best way forward now was to deploy a baseline system for the SSA and begin loading the data; at the same time the clone catalogue server can be used to investigate optimisation of the set-up. Software: NCH reported that ETWS, IAB and he had met to discuss the modular hierarchy of the WSA codes, and agreed a new framework to keep things tidy in the CVS and to make a logically arranged software system that is exportable and deployable in a straightforward manner. A TWiki note (under coding philosophy) gives details. ACTION: NCH to rearrange WFAU/WSA/src along the lines described in the TWiki following deliberations with ETWS and IAB. IAB reported: "A python framework has been written as follows: (-> denotes dependency) DbHandler.py (deployed on curtion server) DbRpcServer.py (deployed on load server) -> CreateExternalJoinThreader.py -> nch's python scripts to make the joins CreateExternalJoinThreader.py defines a python threading class that calls the create join methods. Also defined is a method that starts the thread and another method which checks to see if that thread is alive. These two functions are served for RPC by DbRpcHandler and are called on the curation server side by DbHandler. These are ready to be put through the testing stages." ACTION: IAB and NCH to test the RPC server functionality between linux and Windows for CU16 (Fri PM). MAR reported: "Worked on handling and limiting long queries (time and volume). Queries taking longer than 90 secs will be put put into the background if a user has supplied an email address. The initial defaults are 15,000,000 parameters written to file i.e. 5,000,000 rows with 3 parameters or 1,000,000 of 15 parameters. 60,000,000 rows returned and cycled through before query is aborted 90 arcmin radius for radial search (might be lowered depending on speed of full system) Need to think about limit for Xmatch maybe just 1000 objects max radius 5 arcmin. In testing several concurrent queries ran into memory problems, with help from Peter upped the memory the JavaVM uses under Tomcat, seems to have stopped the problem. The new webserver has at least 4 times the physical memory of current so hopefully that will help further." Networking: ETWS reported (via NCH) that 2MASS ingest files are in the process of being produced, along with default values where IPAC quote nulls. ETWS will progress this and communicate our choice of defaults with IPAC on his return from leave (1/9/03). SSA: Nothing new this week, apart from hardware set-up (see above). Miscellaneous: ACTION: NCH to type up and circulate these minutes. DONM: 10am, Friday 22th August, plate library.