From j.p.emerson@qmul.ac.uk Fri May 2 14:11:43 2003 Date: Fri, 2 May 2003 12:27:17 +0100 From: Jim Emerson To: Malcolm Stewart , Mike Irwin , Peredur Williams , Richard McMahon , Andy Lawrence Subject: WFAU VDFS weekly meeting minutes: 2nd May 2003 VDFS Mgt team Please read these WFAU meeting minutes and feedback any comments to me. Jim -----Original Message----- From: Nigel Hambly [mailto:nch@roe.ac.uk] Sent: 02 May 2003 12:20 To: WFCAM Science Archive Team -- Clive Davenhall; Eckhard Sutorius; Harvey MacGillivray; Ian Bond; Mike Read; Nigel Hambly; Bob Mann Cc: Andrew Lawrence; Peredur Williams; Andy Adamson; Jim Emerson Subject: WFAU WSA weekly meeting minutes, 2/5/2003 Minutes of WFCAM Science Archive meeting: 02 May 2003 ----------------------------------------------------- ----------------------------------------------------- Present: NCH, ETWS, AL, IAB, MAR, RGM Apologies: PMW, HMG In Attendance: JMS Actions discharged: ------------------- ACTION: JMS to review software tasks in build GANTT and comment on the plan. ACTION: IAB to obtain UFTI IR images and test out compression options: gzip, bzip2 & RICE (within CFITSIO). ACTION: NCH to meet with AB to discuss general approach to coding. ACTION: NCH to liase with ETWS on network transfer work for the coming week. - all done and discharged (thanks to all concerned!) (none this week, mainly owing to review & Easter hols). Actions partly discharged but continuing: ----------------------------------------- ACTION: IAB to look at the highest priority use cases, and identify code modules. ACTION: ALL to review risk register and think of any new internal or external risks that should be documented in the risk register. - continuing; thanks for progressing these. Actions carried forward from 25/04/03 meeting: ---------------------------------------------- ACTION: PMW to liaise with AL to make sure that Compute Support are made aware, from the highest "official" level, of our requirements, re. 100 Gbyte/day transfer from CASU, and to impress on them that the 1 Gbit/s FW should be their highest priority. - CONTINUES Specific points and new actions: -------------------------------- Project management: NCH reminded the team to email a small paragraph indicating progress over the previous week to help in preparation of these minutes. NCH reported that AA had been in touch concerning the review report, and this was expected to be sent to WFAU within the next day or so. NCH asked AL about WFAU RG progress: AL reassured everyone that this was likely to be finalised within the next few days, and in any case if there was an urgent issue over hardware money (for example), a way around the lack of a formal grant announcement would be found. AL reported that the next VDFS monthly management meeting was due to take place (by telephone) on Wed 7th May. Hardware: NCH reported that initial tests with the IBM hardware had proved rather inconclusive, with problems with some disks and uncertainty over the provenance of the equipment. RGM expressed the worry that we shouldn't judge the ServeRAID adapter performance on the basis of sub-standard kit. RGM suggested it may be possible to benchmark the NeSC IBM machines, but NCH pointed out that a) they're not running Windows, and b) they're doing lots of other things as well, so benchmarks would be difficult to interpret. HMG reported (via NCH) that Eclipse have potentially come up trumps with a high IO SCSI solution for the catalogue servers, based around standard Ultra320 disks spread over 3 controllers. An extremely competitive quote has been given for these systems, and what's more they are willing to let us "try before we buy", and we can expect delivery of a server next week to benchmark. This system has been clocked at 400 Mbyte/s with MemSpeed. Eclipse will also deliver a 3Ware IDE system at the same time for us to check out the pixel server performance. Networking: "I've asked Horst to install GridFTP on srif112. Also I'm putting together instructions for Horst and Mike Irwin (or his sysadmin in Cambridge) to enhance the data transfer rates by changing the buffer sizes and window sizes (tcp tuning) on both ends of the transfer link. Ben Panter and I have installed Condor on wiglaf (node 3-7, condor master on node 4) and Ben is testing it." Software: NCH reported that a very constructive meeting had been held with AB and ETWS, MAR and IAB concerning software design. AB's advice was that given the tight time-scales, an "extreme programming" approach was indicated, whereby rapid prototyping followed by "refactoring", but at the same time keeping overall software architecture in mind, was called for. The main point to come out of the review documentation from the software point of view was the lack of an overall architecture design following the use case analysis in the DB design and the user interface doc. This should include and identification of the major packages/classes/interfaces, a prioritisation of them, and then a plan drawn up based on delivery of these major packages with milestones intermediate between the existing ones in the GANTT chart and the round of weekly meetings. On the plus side, we seem to be following this basic approach by instint, but a little more detail needs written down (AB suggested using the TWiki). ACTION: IAB to finish analysis of common components (packages/classes/ interfaces) in the WSA by Wed 7th ACTION: NCH to arrange a get-together for Thurs am with ETWS, MAR & IAB to write down the coding schedule for the next quarter, keeping in mind the existing prioritisation of the work in the review documentation. IAB reported initial results of compression tests with gzip and RICE: with UFTI images provided by Paul Hirst, ratios of 80% and 50% were achieved (ie. RICE compresses an image to half it's uncompressed size). NCH suggested that it is uncertain what the equivalent WFCAM ratios would be as this depends on the byte representation of pixels and the statistics of the pixel values in a given image. Some discussion ensued; but a detailed examination was thought to be unnecessary since the pixel storage hardware is phased anyway, and we can match disk purchase to requirements as we gain experience. The WSA hardware budget is based on a worst-case scenario for the pixel volumes. NCH reminded the team that everyone should begin using the CVS for all WSA and WSA related work. IAB reported thet he has downloaded Doxygen and it produces nice results quite easily. ACTION: IAB to put a small note on the TWiki about where Doxygen is on the linux file system, and how to use it. SSA: NCH reported that significant progress has been made on bulk loading of large amounts of data: it is possible to use "native" binary files for this, resulting in major savings in storage, transfer and ingest of intermediate files. HTM indices have been computed for the entire SSA using IAB's implementation of the SDSS HTM C++ routines: the procedure took 1 week of time on the heavily loaded SSS server (of order 10 billion co-ordinate pairs and approximately 2 TByte read / 0.2 Tbyte write). The SSA schema has been revised in the light of review comments on analogous relational design issues in the WSA; scripts are commented for ease of creation of online documentation; scripts are of course under CVS control. Miscellaneous: NCH reported that there is to be a WFCAM observing strategy meeting on 12/13th June at Edinburgh. The draft agenda is to examine issues concerning observing procedures and make sure that all processing and archive issues are covered (eg. calibration). IAB & MAR are pencilled in to attend this meeting (NCH & ETWS are on leave). AL expressed concern that the appropriate WG UKIDSS needs to take an active role in this meeting. NCH reported that this has already been suggested to MMC (who is co-ordinating the meeting). ACTION: AL to contact SJW concerning UKIDSS Implementation WG input and attendance at the strategy meeting. ACTION: NCH to type up and circulate these minutes DONM: 10am, Friday 9th May, plate library.