From nch@roe.ac.uk Fri Nov 28 14:46:53 2003 Date: Fri, 28 Nov 2003 14:40:33 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Harvey MacGillivray , Ian Bond , Mike Read , Nigel Hambly , Bob Mann Cc: CCs for WSA weekly meeting minutes distribution -- Clive Davenhall , Andrew Lawrence , Andy Adamson , Peter Shillan , John Taylor , Jim Emerson , Martin Hill , Mike Irwin , Peredur Williams Subject: WFAU WSA weekly meeting minutes: 28th November 2003 Minutes of WFCAM Science Archive meeting: 28th November 2003 ------------------------------------------------------------ ------------------------------------------------------------ Present: NCH, ETWS, RGM, IAB, PMW, JDT Apologies: JPE, HTMG, JMS, AL, MAR, MCH, GPS Actions discharged: ------------------- ACTION: HTMG to get a quote for UPS for djoser Closed - see Hardware. ACTION: PMW to produce a progress GANTT and send to JPE. Closed ACTION: ETWS to contact Guy Rixon concerning a new GridFTP test. Awaiting installation of GridFTP on djoser; would consult Guy as to best version to install. Closed - installation (with HME) complete; tests underway. ACTION: ETWS to propagate ICD Issue 3 changes into WSA schemas. Closed ACTION: ETWS to again approach HME concerning system access to djoser. Closed - HME says no way because of potential security problems. Our only option is to pester HME for every little thing that we need installing or changing. ACTION: ETWS to produce a NonSurveyDetSchema for the WSA to curate non-survey default catalogue data. Closed ACTION: RGM to chase up the SDSS DR1 "sneaker net" boxes and check that Edinburgh is on the grand tour itinerary. Closed - see Miscellaneous below. Actions partly discharged but continuing: ----------------------------------------- ACTION: ALL to review risk register and think of any new internal or external risks that should be documented in the risk register. Defered until top level plan is explicitly defined. PMW tabled the VISTA risk register as an example. ACTION: RGM to progress design of a web cookbook based around the 20 Queries. Progressing ACTION: JPE to design and set up centralised VDFS web pages. Progressing. (there is now a link to it from our WSA Twiki). - continuing; thanks for progressing these. Actions from 7/11/03 meeting: ----------------------------- ACTION: NCH to stage intermediate Source files in thoth, and email Francois Oschenbien at CDS when the first file(s) is/are ready for transfer. Open - couldn't do anything until transfer protocols set up between Windows and Linux servers on the WSA private network (see Hardware) Specific points and new actions: -------------------------------- [Note added to minutes: JDT, MCH and GPS would be attending the WSA weekly meetings from today as part of integration of archiving and VO work at WFAU]. Project management: NCH reminded everyone that the monthly VDFS management telecon is taking place on Tues pm; PMW would be requesting the usual updates prior to the meeting to create the usual progress tables etc. Some communications have been received concerning the VOProcPlus (aka VEGA) e-science proposal; the funding model for the WSA work looks like becoming increasingly complicated and unclear. In any case, PPARC are requesting more materials from the VDFS and GAIA parts of the funding proposal, on a short timescale, from the project leader JPE. PMW will take care of requests for proposal materials from the WSA part of VDFS. Hardware: UPS has been purchased for djoser; HMG has negotiated with BS and RPL for the WSA hardware to be powered directly in C1 rather than be on the power trip. This should result in more uptime since the UPS will iron out any transient glitches as well as outages on minute timescales; the WSA will no loner rely on the trip being reset manually. ACTION: PMW to check with HMG that the WSA hardware is getting it's own power connection bypassing the C1 trip. NCH reported that he had asked Eclipse to look into 4-channel Ultra320 RAID adapters, and that Eclipse have come up trumps with an LSI controller that fits the bill. Once things have stabilised over the SSA, NCH suggested that a few experiments with these adapters may be in order as they will provide a relatively inexpensive, but significant IO rate (x2) upgrade in the medium term. Eclipse have also provided specs for a new 5U 24-bay JBOD that may be useful for upgrades. NCH reported that he was liasing with JNTD and HME over file shares between the linux and Windows servers, and had installed SSH transfer tools (PuTTY) for ease of shifting data between the various machines. Software: IAB reported: "Carried out major refactoring of the python utilities for the WSA curation. These are in the CVS under "wsatools". The aim is that the final package will allow one to implement any of the 20 curation use cases as well as design new ones. A bottleneck relevant to the FITS ingestion was identified. The python FITS utility "pyfits" is very slow. I have written a python extension in C that will read all FITS header keys into a python list of python dictionaries. This is ~20X faster than pyfits. This has been checked in the CVS under wsatools/FitsIO. Did some major refactoring of the middleware software from stripping information and data from FITS files according to a given database table schema. The new software in the CVS under wfcamsrc should work for any schema and is more readily extensible should our needs change for generating entries fro data attributes that depend in some way on other attributes (suchas HTM ID calculation)." ETWS asked about the source remeasurement schemas in the WSA, and NCH suggested that it is now possible to finish these schemas since the source remeasurement parameter set has been agreed and documented in ICD Issue 3. ACTION: ETWS to finish the WSA source remeasurement schemas for all UKIDSS surveys. IAB asked about the question of simulated WFCAM FITS data; NCH suggested we have a good chat with MJI about this next Wednesday during the Astro-Wise visit (see later). SSA: NCH reported that a small scalability problem with SQL Server had surfaced with regard to the final installation scripts (some utilities will not process row sets having more than 2^31 rows). This is a known bug, and Jim Gray from Microsoft informs us that it will be fixed in SP4. In the meantime, a patch is available. Also, the final installation scripts are having to be modified so as to create minimal transaction logging (otherwise the SQL transaction log fills up and crashes the transactions). SSA scripts are being modified to get around this and are being checked into the CVS. Miscellaneous: NCH reported giving an update to the UKIDSS consortium last Friday; this was well received. NCH has written up a contribution for the next UKIDSS newsletter detailing the URLs for SSA (and proto-WSA browser) and requests for test-driving and feedback from the community. NCH reminded everyone that he, IAB and ETWS were visiting Astro-Wise next Wed/Thur, and suggested that the same folks should read the Architectural Design document so as to go armed with some questions. RGM emphasised the desire to get as much feedback on their Oracle experience as possible, including their experience of integrating HTM into it. NCH has already communicated with Ed Valentijn the specific areas of WFAU interest, and also pointed them to our own development webpages so Astro-Wise folks can get a good idea of what we are doing. RGM reported that Ani Thakar is visiting NeSC on Mon/Tues, and is unlikely to visit ROE; informal chats over beers will be arranged. RGM also reported that the SDSS DR1 "sneaker net" boxes may be with us just before or after Xmas (it has to go to MPA-Munich and then Hungary before arriving in Edinburgh). NCH & RGM suggested we should try to prepare as much as possible before it arrives. ACTION: RGM to inquire as to storage requirements and more accurate schedule (especially ETA) for the SDSS DR1 sneaker net box. DONM: 10am, Friday 5th December, plate library.