From nch@roe.ac.uk Tue Jul 13 12:10:53 2004 Date: Fri, 9 Jul 2004 11:26:48 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Harvey MacGillivray , Ian Bond , Mike Read , Nigel Hambly , Bob Mann Cc: CCs for WSA weekly meeting minutes distribution -- Clive Davenhall , Andrew Lawrence , Andy Adamson , Peter Shillan , John Taylor , Jim Emerson , Martin Hill , Mike Irwin , Peredur Williams Subject: WFAU WSA weekly meeting minutes, July 9th 2004 Minutes of WFCAM Science Archive meeting: 9th July 2004 ------------------------------------------------------------ ------------------------------------------------------------ Present: NCH, MAR, PMW, ETWS, MCH Apologies: JPE, HMG, GPS, JMS, AL, JDT, RGM DONM: 10am, Friday 16th July, plate library. Actions discharged: ------------------- ACTION: MAR to update the image astrometry tables in the WSA Discharged. Actions partly discharged but continuing: ----------------------------------------- ACTION: ALL to review risk register and think of any new internal or external risks that should be documented in the risk register. Defered until top level plan is explicitly defined. PMW tabled the VISTA risk register as an example. ACTION: JPE to design and set up centralised VDFS web pages. Progressing. (there is now a link to it from our WSA Twiki). - continuing; thanks for progressing these. Actions carried forward from 2/07/04 meeting: ----------------------------------------------- ACTION: ETWS to add in required ingest functionality for MAR's changes and check to see if the ingest code can cope with this schema evolution. ACTION: ETWS to set up CVS logins for MCH and JDT - CONTINUE Specific points and new actions: -------------------------------- Project management: NCH noted that a new VDUKURD (?) has been circulated from Vista. This has been placed on the TWiki under the VDFS topic; there were some new requirements on the archive that seem reasonable; archive uptime requirements have now been moved to goals as requested. PMW requested a little time from team members to fill in the Q2 report for next week's VDMT; also reported revisiting the overall plan in the light of IAB's departure and the new recruitments. Applications for the new posts stand at a healthy 20 for the astronomer-developer and 30 for the programmer-developer. PMW asked if any more team members should attend the VDMT on the 15th; NCH suggested that we arrange for a meet-up over lunch so that CASU, WFAU and VDMT management can network and discuss any relevant issues. Comments and issues arising from CASU fortnightly minutes: No new minutes as of today. Networking: NCH reported that the entire USNOB (230 GB) SQL Server database has been transfered across the Atlantic with an aggregate transfer speed of more than 1 MByte/s using multi-threaded http from the archive web server thoth. This has been achieved by using ETWS's prescription for widening TCP windows at both ends and simply launching multiple wgets (wget having been hacked to provide support for files bigger than 2 GB). Hardware: NCH reported a fruitful meeting between Eclipse and AL, PMW, JNTD, NCH & RGM. There is now some confidence (on the Eclipse side at least) that the Ultra SCSI problem has been isolated to cabling and connections in the server and JBOD chasses. U320 catalogue server "amenhotep" has been soak-tested here for the last week with the JBOD removed and continues to run stably; U160 server ahmose has not exhibited any more instabilities despite running with the old, copious cabling. Eclipse have taken amenhotep's JBOD back and are reconfiguring with minimal cabling. The idea is to reconfigure each of the other rackmount chasses using hardwired "backplanes" and minimal cabling, the hope being that this will cure the problems. NCH has suggested to Eclipse that these changes must be done and be demonstrated to be working by the end of September. Failing this, a solution based around U160 will be employed. Meanwhile, LSI Logic have restarted shipping of the 4-channel MegaRAID and Eclipse have sourced one from their UK supplier. This will be tested on amenhotep after the 2-channel card has been soaked for another week; first with 16 disks and then with the full complement of 32 disks in the reconfigured chasses. The remainder of the meeting was taken up with discussions on medium-term expansion options for the archive hardware. Upgardes to disk capacities (U320 will go from 146 GB to 300 GB soon; SATA will go from 250 to 400 to 500 GB within the next year or so; additional MegaRAID cards will allow the catalogue servers to expand while additional servers with 3Ware SATA RAID controllers will be required to keep pace with WFCAM pixel data storage requirements). The consensus is that these upgrade paths essentially solve the WFCAM storage problem; on the VISTA timescale we expect the SAN-style solution to be employed (possibly off-site). MAR & NCH suggested that the capacity of the http web server thoth could do with being expanded. ACTION: MAR to liaise with HME over the possibilities of expanding the disk storage available on the http server. Software: ETWS reported: "Finally mxODBC is running on djoser and the last problem is solved. After making sure that the BIGINT problem (cutting off the least significant digits) was not in the drivers by installing and testing iODBC and unixODBC on ericht (connecting to PSSA on MAR's toshnt71) - also testing Python version 2.3.2 and the latest (final) version 2.3.4 - and on djoser (connecting to WSA and SSA on ahmose), it came out that the parameter sqllen is set wrongly (to 8) in mxODBC on Linux. mxODBC assumes that the information for sqllen contains the original database information in case no converter function is set. (For numeric values this parameter normally holds the precision.) The solution will be a converter function which will set sqllen to 0 to have a dynamically growing data transfer buffer." MAR reported: "Extended the ImageSelect class to return all images a given RA/Dec appears on as well as the "best",also returns other useful values in a hashmap (eg min/max X/Y, WCS values). Changed the FITS image writing to produce a multiextension file (a copy of the primary HDU and the extracted sub-image). This code also accepts FITS cards to be modified i.e. so that if currentAstrometry contains revised WCS values they can replace the original ones." NCH has been working on-and-off on the source merging; much time has been taken up this week investigating the database interface layer problem that has now been resolved. New versions of the Python DB-API factory functions are being checked into the CVS before resuming development & testing of the source merging codes. SSA: No news (is good news) this week Astrogrid deployment: NCH revisited the question of registry XML docs from SSA schema files. MCH reported that the registry standards were rapidly converging and that he would be looking into the requirements for SSA schema translation soon. Miscellaneous: Nothing else this week.