From nch@roe.ac.uk Fri Feb 17 12:52:26 2006 Date: Fri, 17 Feb 2006 12:49:06 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Martin Hill , Mike Irwin , Peredur Williams , Stephen Warren Subject: WFAU WSA weekly meeting minutes, 17th February 2006 Minutes of WFCAM Science Archive meeting: 17th February 2006 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, RSC, PMW, JB, MAR, NJC Apologies: JPE, JMS, AL, JDT, RGM DONM: 10am, Friday 24th February 2006, plate library Actions discharged: ------------------- ACTION: ETWS and MAR to enhance the schema browser pages to cope with multiple browser versions, supporting previous releases. Discharged; enhanced browser with past schema versions will be going live very soon. ACTION: NCH to put a list of known issues and bugs up under the archive release history. Discharged; bugs were noted on the page at the same time as the release; NCH has now fixed the three major bugs and further noted the fixes in the same release notes. ACTION: MAR to put a note on the "downtime" web page to inform users of the small interuption to normal service. Discharged. Actions partly discharged but continuing: ----------------------------------------- ACTION: JDT to set up a Datascope Launcher for the WSA on the AG workbench pages. Continues. Actions carried forward from 10/02/06 meeting: ---------------------------------------------- ACTION: NCH to investigate the usefulness of a NAS solution for medium term archive mass storage. - CONTINUES Specific points and new actions: -------------------------------- Project management: NCH and PMW brought up the subject of scaling the archive for the Vista UK design review; some discussion ensued as to various issues concerning scalability etc. The pragmatic approach is to continue to update the design documents with scaling issues in readiness for any review: ACTION: JB to review Hardware/OS/DBMS design doc to note areas that need updating and itemise new sections required ACTION: RSC to review the Software Architecture Document (SAD) in the light of implementation differences and any new OO approach under development ACTION: ETWS to go thru the CUs one by one and note any issues as regards performance/scalability. WFCAM update: See the stop press pages linked from the TWiki for latest news. Comments and issues arising from CASU fortnightly minutes: No new minutes as of 17/2/06 AM. Networking: Good news this week - ETWS reported: "Started the transfer of BestDR3 from Chicago. Got a script from David Hanley, which transfers 4 parts in parallel. The transfer rate so far just below 9 MB/s which is exactly the same as 4 * 1.5MB/s, which was the rate for a single transfer, so we are not hampered by any transfer window issues." NCH reminded the team of the UKLight event at NeSC; JB and ETWS are registered to attend this on March 1st. JB noted that a new file server is ready to accept data, and that data transfers from CASU will resume this weekend. WSA Operations: JB reported: "System backups went as normal along with Database backups for the EDR, the upggraded EDR and EDRPLUS and WORLDR2. DR2 has also been restore to Ahmose so it can be spread across four disks and alleviate the artificial space problems on Amenhotep. Amenhotep required a reboot on Monday due to Operating System instability (a similar problem has occured a couple of times to Ahmose at the end of last year). A at-risk maintenance period of one hour on the first Monday of each month has been implemented so that this and other problems can be fixed or investigated at set times when users know the system may be down. CU4 (and all the preparitory work necessary to run it) is now running, though since the GPS is currently 170Gb this will take a while to ingest. Four new Non-Surveys have been added to the database and CU4 run for those that needed it." Hardware: JB reported: "Djedefre has been delivered and is now installed courtesy of IT support, it still needs our packages installed but we should be able to start running CU1 on khufu to download data to its disks. Helpdesk tickets have also been submitted to have djoser upgraded to the latest Debian (a ticket for thoth was submitted by MAR). Tickets have also been submitted to note that the switch and KVM will need upgrade/expanded at some point soonish. SRIF IP addresses are also running out however a plan has been made on how to reduce the need for these to be used as much by the storage servers." MAR reported getting quotes for a third catalogue server to relieve congestion on the existing public server; expansion to larger disks and higher spec CPU motherboard will be investigated, but the existing design will be followed broadly speaking. NCH raised the issue of the next storage server, and the likely requirements in the light of recent experience. The team agreed that we should look into 0.5TB SATA disks hanging off 3ware controllers that can handle RAID sets larger than 2TB; a 64-bit motherboard will be advantagous for large stacking/mosaicing tasks for the next iteration on the UKIDSS UDS. ACTION: PMW to start the ball rolling on getting quotes for a new storage server with an expanded spec over khufu/djedefre. Software: NCH noted the scalability issue of GPS ingest, and suggested that now would be a good time to switch from ASCII formatted CSV ingest of catalogue data to much more efficient binary. ACTION: RSC to implement binary catalogue ingest in CU4. RSC reported: "I've solved all of the problems for the new Cu16 script, and I have also been working on a new OO-framework for the curation task scripts." ETWS reported: "Worked on the new browser parsing, which will take different releases into account. Included the description of how to run the many new helper scripts for CUs 1 to 4 on the TWiki WSA operations pages." Survey Data Release: NCH noted that a consensus on the release policy for the next survey releases has yet to be reached within the consortium. In the meantime, we should continue with transfer/ingest of remaining 05B data, and SJW and NCH are to start to investigate 05B quality control. Non-Survey Data Release: JB asked how we know which non-survey datasets should be released when, given that flexible scheduling results in it not being obvious when observations for a given programme are complete. Some discussion ensued about piecewise releases, but NCH suggested that it would be a great help if we new when non-survey data were being taken to help us in deciding, on a case-by-case basis, when to release survey data to PIs (as opposed to waiting until the end of the WFCAM observing block). ACTION: NCH to contact AA at JAC to discuss possible input from JAC to help us time releases of non-survey data. Astrogrid deployment: Nothing new this week. Miscellaneous: NCH noted that he has been making steady progress on the WSA write-up Added after the meeting at the suggestion of RSC: ACTION: RSC write an abstract for NAM, aiming for a talk with the emphasis being to encourage people to both make use of the WSA and more importantly highlight the power and simplicity of SQL searches for data mining etc. Finally, NCH suggested early doors at 6pm in Cloisters this evening after work.