Minutes of WFCAM Science Archive meeting: 12th September 2003 ------------------------------------------------------------- ------------------------------------------------------------- Present: ETWS, JMS, IAB, NCH, RGM, JPE Apologies: MAR, AL, HMG, PMW Actions discharged: ------------------- ACTION: ETWS to update TWiki networking pages with new test results. ACTION: NCH & IAB to meet offline to discuss the way forward for implementation of specific tasks now that development of generic wrappers and modules is coming to a close. (see Software below) ACTION: RGM to consult with MAR on exporting SkyServer EDR from grendel01 into a transferable file to set up on one of the load servers; and to enquire with JHU as to location of the DR1 "sneaker net" box. (discharged; EDR is now on new catalogue server ahmose, and installed; Alex has been emailed concerning DR1) ACTION: EVERYONE to examine prototype documentation, and to send comments and suggestions to MAR. (discharged - many comments and suggestions to MAR). - all done and discharged (thanks to all concerned!) Actions partly discharged but continuing: ----------------------------------------- ACTION: NCH to communicate readiness of 10% SSA for indexing asap to RGM (pending diagnosis of disk problems on ahmose) (delayed by hardware problems but nearly there...) ACTION: ALL to review risk register and think of any new internal or external risks that should be documented in the risk register. - continuing; thanks for progressing these. Actions carried forward from 05/09/03 meeting: ---------------------------------------------- ACTION: RGM to progress design of a web cookbook based around the 20 Queries. ACTION: PMW to liaise with AL to make sure that Compute Support are made aware, from the highest "official" level, of our requirements, re. 100 Gbyte/day transfer from CASU, and to impress on them that the 1 Gbit/s FW should be their highest priority. - CONTINUE Specific points and new actions: -------------------------------- Project management: NCH took the opportunity of JPE attending the meeting to ask about a set of centralised VDFS web pages, inclduing the top level "business plan" and a documentation archive. JPE agreed that this must be done, but he wouldn't have time immediately; however agreed to be formally actioned to do this. ACTION: JPE to design and set up centralised VDFS web pages. JPE pointed out two medium term conference opportunities: NAM'04 (April, Milton Keynes) and SPIE'04 (June, Glasgow). Both have sessions on new ground based telescope facilities and data management projects. The SPIE deadline for abstracts is end Nov'03. JPE has emailed more details to NCH and MJI to start the ball rolling on a co-ordinated submission. Hardware: NCH reported that, thanks to hard work on the part of JNTD, a solution to the disk reliability problems may have been found. The solution appears to be to run the Ultra320 controllers using the latest W2003 drivers from Adaptec (not MS CDROM) along with reduced speed to 160. NCH reported that because of the logical volume design (8 disks striped across 8 controller channels) and the memory copy bottle neck, the disk IO speed is not greatly compromised by this reduction in interface speed. Clearly, this is not a long-term solution but is fine for now. NCH is continuing with loading and DB set-up and will keep an eye on the Windows system logs to watch out for further IO errors, but the situation at the moment is that ahmose (which was giving constant errors before) has not logged a single disk error since being set up in this way. Software: IAB reported: "The following python modules have been checked into the CVS DataTypes/ CurationUseCase.py Filter.py ProgramID.py TimeStamp.py modules/ CatExDriver.py DbHandler.py DiaDriver.py ProcessDriver.py ScienceFile.py StackDriver.py TextFile.py dbrpc/ DbRpcServer.py TaskThreader.py And under appropriate sub-directories under invocations: cu1.py cu2.py cu3.py cu4.py cu13.py To do from now: Require methods/SQL for - DB locking and unlocking - updating the curation log - ingesting image metadata - ingesting catalog data Implmement procedure/policy for validating FITS files inhaled from CASU Check performance issues regarding stripping off data from FITS files to be ingested into the data base. pyfits seems rather slow. Once these issues are dealt with, then cu2, cu3, cu4, cu13 can be ticked off as being ready for real WFCAM data ingestion. " IAB suggested that we now need to populate a proto-WSA database with curation and image metadata to test out the early curation software. JPE pointed out the availability of a nights worth of CIRSI data that is broadly similar to WFCAM in terms of metadata. The consensus was that it would be a good idea to transfer this up as a first simulation of the WSA transfer/ingestion process. ACTION: IAB to contact MJI and JRL to arrange for transfer of CIRSI data. Networking: Nothing new this week. SSA: MAR reported (via NCH) that the WSA web server is online and he is now setting up low-level access software (tomcat etc). NCH reported that progress was now being made on the 10% and full SSA DBs now that the hardware issue seems to have been solved. The entire 1.2 Tbyte SSA has been loaded in 33 hours, comprising heaped bulk insert from the native binary format files followed by attachment of primary keys. NCH & MAR suggested that the "20 queries" documented as a cookbook might be a little esoteric for the complete novice user, since some queries are geared towards exercising aspects of database design. Suggested that additional worked examples of using the web form access points should also be included; NCH & MAR will look after this part of the cookbook. Miscellaneous: ACTION: NCH to type up and circulate these minutes. DONM: 10am, Friday 19 September, plate library.