From nch@roe.ac.uk Mon Jul 13 09:12:18 2009 Date: Fri, 10 Jul 2009 17:06:45 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archives weekly project meeting mins, 10/07/09 Minutes of WFAU VDFS Science Archive meeting: 10th July 2009 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, MAR, NJC, ETWS, PMW, RPB, RSC, RGM Apologies: JPE, JMS, AL, MSH DoNM: 10am, Friday July 17th 2009 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: RPB to liaise with ITSG over diagnosing and curing a network speed problem at this end. Discharged (by ETWS); it seems the cable may have been plugged in the wrong hole... anyway, ETWS will keep an eye on the bandwidth of the next batch of transfers. ACTION: NCH to put a UKIDSS SV release note on the web pages with an appropriate health warning. Discharged (it's somewhat perfunctory...) ACTION: RPB to release (most) 08A open-time flat-file data via the usual procedure. Discharged (also see below for issues/new features) Actions partly discharged but continuing: ----------------------------------------- ACTION: ETWS to parse that latest keyword info into the VSA metadata schema files. Continues; this will go ahead using pawprint products following comments in the latest CASU minutes (see below) ACTION: NCH to draft an interface control document for VSA to ESO-SAF On hold owing to grant and political issues... Actions carried forward from 26/06/09 meeting: ---------------------------------------------- ACTION: RPB to check out wsaro/worldro permissions on public survey DBs. - CONTINUES Specific points and new actions: -------------------------------- Project management: The team filled in the progress charts for June. ACTION: PMW and NCH to meet Mon 12pm to finalise the Q3 plan. Much discussion this week concerning the fall-out from the STFC announcements of reduced support for the wide field units. RGM and MJI have already sent out a briefing note to the main stake holders... ACTION: NCH to send a message to UKIDSS SHs pointing out the immediate implications of the slashed grants. The bottom line is that the 45% reduction at WFAU means development effort moves to Gaia faster (0.6FTE already gone) which means that more VDFS operations effort will be needed for VSA shake-down, and consequently reduction in operations support for both WSA and VSA. IfA/University are being very understanding and helpful so no immediate loss of staff, but next 3yr will be tricky. The team discussed how we can get as much of the community as possible to input support to the current ground-based facilities review, and agreed that info and material should probably go onto the WSA web pages to publicise as widely as possible. WFCAM & VISTA updates: Nothing of note this week. Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: ETWS noted: - IT support fixed the very slow UKLight link, but it is still delivering only 90% of the speed it had before 13/06/09. HME reported that he had 'unplugged and re-plugged the fibre and the converter from two colours in one fibre to a fibre pair (but not the switch port module itself)' which improved the performance significantly. WSA Operations: ETWS noted: - Started ingest of GPS reprocessed data 9 days ago, and 70% of the data is now ingested. - Retransferred the data (58 files) that had been corrupted by a CU8 run that used by accident pyfits 2.1 instead of 1.4 - Transferred the UDS reprocessed data after IT support fixed the UKLight link. RPB noted that public server amenhotep has been giving problems, and that it was likely to remain offline over the weekend. PMW noted that he, RPB and MSH should take a close look at the maintenance contracts for these venerable servers, in the light of the recent grant cuts and their (un)reliability. The team agreed that in the light of problems with the primary public server, metadata mirrors WFCAMPROPRIETY [sic] and WFCAMOPENTIME should be created routinely also on the batch server thutmose. NCH also asked that calibration frames (CAL) be included with WFCAMOPENTIME to make those metadata and flat files easily available to interested (i.e. one!) users. Hardware: NCH noted that "distributed partitioned view" tests with SQL Server 2008 and the windows cluster have stalled, with diabolical performance for everything except the simplest case of a single partitioned table. Results have been communicated to MSH who is currently in Johns Hopkins to learn from SDSS/PanSTARRS experiences, but the current thinking is multiply-attached read-only DBs may be the only sensible option. RGM noted that IfA want a corner of the cluster as a temporary storage space (for 10TB). WFAU are happy, but getting stuff on/off these Windows machines may not be as simple as IfA are hoping... Concerning general storage issues, and continuing on from the rolling grant saga, it seems there may be no money in the grant for equipment for 09/10 so ETWS and RPB have been looking into rationalising and squeezing up the current storage, starting with removal of redundant ancestor frames (i.e. normals of leavs) for surveys, deletion of all old processed versions, through to keeping only science stacks. It was noted that users should be aware of what may happen, so ACTION: MAR to start a web page detailing probable image storage rationalisation policies and use of wsa-announce to let users know when things will disappear. Software: RSC noted: "I've implemented the software update of the new Multiframe.unfilteredID and subsequent special deprecation of these unfiltered multiframes. Both processes have now been applied to the WSA after testing and will be automatically updated in future. Various other small patches to improve error messages etc. have been applied to aid operations. Further development work for the consistent error message logging style that we agreed on last week as well as VSA development has been delayed due to the inability to test software changes: caused by amenhotep being down and an inability to create new test databases on ahmose due to the GPS CU4 ingests occurring there (these mostly consistent of constraint checks, which block other constraint checks on ahmose that are required to create test databases). So because of this, but also to reduce the load on amenhotep, our primary public catalogue server, we'll try to set-up test databases on hatshepsut in future. However, this will no doubt require lots of tweaking of database permissions etc. to get working fully. A third bug with the PyFITS 2.x series was discovered when NJC found some files had been corrupted that was tracked down to a single CU8 run that occurred during the short time we were using the new PyFITS. This was apparently caused by a failure of PyFITS to remove HISTORY keywords from the FITS header leading to header corruption. Equivalent runs subsequently using the PyFITS 1.x series revealed no such problems. We should submit requests to the PyFITS developers to fix all three bugs as the PyFITS 1.x series is not fully compatible with Python 2.6 that we are currently using for its effective 64-bit support. With respect to the VISTA data, I've checked all of our code that handles ZPN projections from WCS data and as far as I can see in all places where it is required we have support for TAN projections too. However, this will need to be tested in more detail at the appropriate stages." ACTION: RPB to fettle /mnt/hatshepsut mounts from linux curation servers Survey Data Release: Eyeball QC results are in for DR6 LAS, GPS and GCS; no news on UDS mosaics for DR5 from Notts. Non-survey Data Release: Nothing of note this week. Astrogrid deployment & Data Analysis services: Nothing of note this week. Miscellaneous: Nothing else this week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================