From nch@roe.ac.uk Mon Jul 20 11:04:46 2009 Date: Fri, 17 Jul 2009 17:19:12 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting mins, 17/07/09 Minutes of WFAU VDFS Science Archive meeting: 17th July 2009 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, MAR, NJC, ETWS, PMW, RPB, RSC, RGM, MSH Apologies: JPE, JMS, AL DoNM: 10am, Friday July 31st 2009 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: RPB to check out wsaro/worldro permissions on public survey DBs. Discharged; all is well, and scripting procedure reviewed to ensure things work as they should in future. ACTION: PMW and NCH to meet Mon 12pm to finalise the Q3 plan. Discharged; see below. ACTION: NCH to send a message to UKIDSS SHs pointing out the immediate implications of the slashed grants. Discharged; much sympathy, supportive comments and action in progress... ACTION: MAR to start a web page detailing probable image storage rationalisation policies and use of wsa-announce to let users know when things will disappear. Discharged; on hold until grant sorted out ACTION: RPB to fettle /mnt/hatshepsut mounts from linux curation servers Discharged; much activity on ironing out SQL 2005 differences in preparation for VSA deployment. Actions partly discharged but continuing: ----------------------------------------- ACTION: ETWS to parse that latest keyword info into the VSA metadata schema files. Continues; NCH emphasised that this is now top priority (see below) ACTION: NCH to draft an interface control document for VSA to ESO-SAF On hold owing to grant and political issues... Actions carried forward from 10/07/09 meeting: ---------------------------------------------- None this week. Specific points and new actions: -------------------------------- Project management: The team examined the Q3 plan, and noted that VSA work is now top priority. Some small changes were suggested, but otherwise the plan was ratified. The grant situation continues to change almost daily, but until we have something substantive in writing from the research council, NCH suggested we assume the worst as detailed in last week's minutes. RGM and MAR have placed a headline note on the WSA pages to encourage VDFS users to input support via the current GBFR: http://surveys.roe.ac.uk/wsa/GBFR.html NCH noted that NUAP have extended their consultation deadline. PMW, RGM and NCH attended a VDMT telecon on Weds pm with JPE, MJI, STH and RGMcM ... WFCAM & VISTA updates: ... at which JPE gave a status report on VISTA. JPE asks that we now make VSA work absolute top priority, as survey science verification data, followed closely by "shared risks" survey shake-down observations, will start to flow very soon. MJI noted that the latest version of Vista pipeline-processed data is available for observations taken in mid-June, and suggested these be used for metadata schema prototyping and ingest tests. Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: ETWS noted: - Transferred the 09A June data, transfer rates are back to normal. (new record of 14.5 Mbyte/s, albeit for relatively small dataset) WSA Operations: ETWS noted: - Created JPGthumbnails for QC for 09A Jan-May. - CU4 ingests for the reprocessed GPS are going slowly due to the mid week server hiccup. - The reprocessed 20080920 data is fully transferred and ingested as v3 data. Hardware: Earlier in the week, RPB reported: "I'm pleased to report that amenhotep has returned from its brief holiday feeling much better. All disks are now back online and pretty much all data seems to have been preserved. There was only one database (UKIDSSDR1PLUS) that was marked as suspect, but it should be a fairly straight-forward job to restore that with the copy on thutmose. The other problem was ahmose, which decided that it wasn't going to talk to either its G or H drives. After about 4 hours of trying various solutions, Ian from Eclipse and I were both amazed to find that it all seems to have been caused by one physical disk failing. I'm still not sure how one disk can wipe out all four channels of a Raid controller but, having replaced the disk, ahmose seems a lot happier." Now that the public servers are all apparently back to normal, RSC asked as to the status of the the WFCAM metadata mirrors, and noted that the open time and proprietary metadata DBs should be updated ACTION: RPB to update the metadata mirrors on the main public server. PMW noted that maintenance contracts have been updated on the venerable catalogue servers thanks to money from the IfA, and MSH noted that we should now review the overall plan for catalogue servers in the light of his visit to JHU, and the general poor reliability of the SCSI JBOD systems v. serial-attached storage. This will be discussed at this afternoon's cluster deployment meeting, 2pm in the Stobie Room. Software: RSC noted: "Testing curation on hatshepsut, the VSA server, for the first time. There were lots of server settings that needed adjustment and a few modifications to the curation software to enable it to work with the new SQL Server 2005 installation on hatshepsut. The detection and diagnosis of the new issues this installation poses is ongoing work, but we don't expect any major difficulties. I've also taken the opportunity to improve the performance and robustness of the auto-curation process whilst using it as a test for curation on hatshepsut." RPB noted that SQL 2005 server hatshepsut is running low on disk space, which is not helping shaking down and test procedures. ACTION: NCH to tidy up unwanted bulk outgest files and tempdb etc on hatshepsut. Survey Data Release: The team discussed survey release options in the light of progress reported above. MAR plans to close out QC1 for DR6 today, so everything now rests on the deblended extended flux fixes required to both flat files and the ingest DB. The team agreed that a DR6 phase 1 release should be possible by end Aug without the bug fix; NCH suggested that it's impossible to say for certain, but that it's likely that applying the fix could easily add a month to this. Final decision on to fix or not to fix should of course rest with UKIDSS; but the team noted that we're quite close to the strange situation of possibly making the DR6 phase-1 release (i.e. not fixed) before the DR5 phase-2, which may annoy some users at the expense of delighting others... Non-survey Data Release: Nothing of note this week. Astrogrid deployment & Data Analysis services: Nothing of note this week. Miscellaneous: MSH noted that transfer of 16TB of PanSTARRS data from Hawaii is presenting a challenge ... given MAR's imminent visit to the islands, one solution proposed was a few boxes of LTO4 tapes ... ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================