From nch@roe.ac.uk Fri Feb 8 14:57:36 2008 Date: Fri, 8 Feb 2008 12:31:58 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Nigel Hambly , Nicholas Cross , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 8/2/08 Minutes of WFAU VDFS Science Archive meeting: 8th February 2008 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, LGR, NJC, PMW, RGM, BCW, MSH, MAR, RSC Apologies: JPE, AL, JMS DONM: 10am, Friday 15th February 2008 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: NJC to look after LAS/GCS recal issues following DR3 release. Discharged ACTION: NCH to email the VISTA EPS PIs to get further input on database design for their respective surveys. Discharged; thanks for advice from ETWS, NJC, PMW and JPE. ACTION: ETWS to email SJW that the first month of JPEGs etc are ready for perusal by survey heads. Discharged (hasn't come back to NCH yet though...!) ACTION: PMW to take advice from MSH on expanding the kvm units. Discharged; MSH cannibalised cluster grendel (may it rest in pieces) for it's kvm switch so no worries here anymore. Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; will get sorted (honestly!) as part of the general ingest DB schema revamp to facilitate rapid generalised photometric/astrometric recalibration after DR3. ACTION: MAR to take forward the possibility of archive and/or on-the-fly colour image production. Continues; line put in the forward plan for Q1'08; MAR has been studying renaissance masterpieces in the Florence Uffizi for inspiration... Actions carried forward from 01/02/08 meeting: ---------------------------------------------- ACTION: MAR to incorporate basic info on using flat files Continues ACTION: RSC to coordinate migration of the residual CVS-tracked projects into SVN. Continues; RSC will coordinate with MAR at some convenient point to freeze folder docs; NCH will check in the latest full SSA scripts to be clear of the old repository. ACTION: MSH to liaise with IT Support over set up of NFS file share from hatshepsut Continues; MSH has root access (let the good times roll) on djoser and will investigate optimisation of NFS settings. ACTION: PMW to check that IT Support have our future computing accomodation requirements factored into plans for the new room Continues Specific points and new actions: -------------------------------- Project management: The team updated the January progress charts during the meeting for those on leave last week. PMW noted that update from STFC on likely operations resource level is now unlikely to appear before the end of the month. PMW also note that if any conflict between lower priority ops (e.g. non survey support) and VDFS development occurs, then the latter must take precedence with an appropriate explanatory note to any disgruntled users. WFCAM & VISTA updates: WFCAM is back on UKIRT; first UKIDSS night is the 10th. Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: ETWS noted that the SRIF network will be down on the 19/02/08 frm 7:00 till 9:00. ACTION: MAR to put a note on the downtime pages, and email an interuption of service warning to wsa-announce. MSH reported that he has set up the new NAS box and is experimenting with NFS / RAID stripe settings for optimal file transfer speeds. Comprehensive tests will be done in the first half of next week, but early signs are good in that NFS settings can make the difference between 2 MB/s (cf. existing general performance on the WFAU LAN) and 40 MB/s (cf. the kind of performance reported by CASU). WSA Operations: ETWS reported: - The file timestamps are now updated for all databses on the public servers as well. - Operations are going well, September and half of October are ingested now as well. NCH noted that weekly differential backups of the ingest WSA DB are now taking place on Sundays (pm) including DB consistency checks. NCH and PMW, under advice from JNTD in ITSG, have ordered updated Veritas BackupExec SW and client access licenses, which include support for SQL 2005. Hardware Monthly reboots of the SQL servers occured on Monday am this week; NCH also took the opportunity to clock thutmose's SCSI channels back to Ultra160. Turns out that ahmose is still operating at U320, so this needs sorting at next reboot. MSH noted that he is very keen to disk map and document all the catalogue server JBODs for easier maintenance, this will require a small amount of downtime sometime during the next week at the convenience of operations. Debian 4.0 upgrades are to continue with sneferu (MSH is liaising with ITSG over these and 32/64-bit issues etc). Software: RSC noted: * Successfully tested all curation software under Debian 4.0 on the newly upgraded djoser * Synchronised between branches SQL updates that affect the WSA * Created in the TWiki article ModifyingSchemas a guide to making SQL schema updates by collecting together and expanding upon some existing notes: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ModifyingSchemas * Helped NJC debug database connection issues in CU6 and various other minor issues in CU8 and ProvenanceFiller ETWS noted phase 2 of the parallelisation has been implemented successfully. NJC noted "This week I have run CU8 to fix the CU4 problem on all data in the WSA. I have added new scripts to do this and sorted out an incorrect selection on deprecation. I have also been working on CU6 and have sorted out (with much help from Ross) tangled database connections." NCH emphasised that implementation of the schema changes required for efficient low-level photometric recalibration must be top priority now. RSC noted that strategy and proof-of-concepts is in place (see TWiki note) and that work can start next week. Survey Data Release: With due regard to the time required to implement the recalibration changes, the team discussed an outline timescale for DR4 release. UKIDSS (i.e. SJW) has asked that we aim for June 1st this year for DR4, to include everything up to the end of 07B, photometrically recalibrated using the most recent tweaks available from CASU. The team discussed this in the light of timescales for transfer, ingest, QC, etc. and agreed that this should be OK. A detailed timetable will be sketched out in the near future... watch this space. Non-survey Data Release: No news this week. Astrogrid deployment: ETWS noted BestDR1 has been transferred from JHU and is now on its way to thutmose; NCH will attach, add access application usernames and backup to tape; ETWS will generate metadocs for a new DSA and pass over to MSH to deploy the service so that BCW/RGM can do their thing. RGM noted that a 2-day visit by ESAC VO folks is planned for later this month, at which SIAP/SSAP implementation issues will be discussed. MSH noted that major deployment tests of the latest AG infrastructure are to take place beginning later next week, so batch server thutmose will be getting much incoming from the DSAs. Miscellaneous: (Leaving this in again from last week:) NCH noted that the NAM 2008 at Queen's University Belfast is now open for registration, with wide field surveys (incl. Gaia & PanSTARRs) scheduled for the Thurs/Fri Apr 3/4: http://nam2008.qub.ac.uk/ Deadline for registration for oral presentations is 29th Feb. Current attendees: NCH, MSH, LGR