From nch@roe.ac.uk Fri Jan 25 14:31:04 2008 Date: Fri, 25 Jan 2008 14:00:11 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Nigel Hambly , Nicholas Cross , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 25/1/08 Minutes of WFAU VDFS Science Archive meeting: 25th January 2008 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RSC, MAR, ETWS, LGR, NJC, PMW, MSH Apologies: BCW, JPE, AL, JMS, RGM DONM: 10am, Friday 1st February 2008 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: JB & ETWS to restore SegueDR6 on thutmose at their convenience (i.e. no hurry given the present circumstances) Superseded by new actions below. ACTION: MSH in consultation with RSC to liaise with IT Support over linux upgrades. Discharged; djoser is the first to go and ETWS is currently testing SW installations. ACTION: PMW, MAR and MSH to liaise over progressing the new web server procurement. Discharged immediately after the meeting; several worries over the Esteem (sic) quote based on SunFire systems; PMW will up the ante with the beaurocrats to try to get satisfaction... ACTION: ALL to look at the synoptic data model from NJC and provide comments. Discharged; some minor tweaks required, but basically looking very good and NJC will take this forward. ACTION: NJC, NCH, MAR, ETWS and RSC to meet on Wed 23rd Jan at 2pm in the Vista Hut to brainstorm database aspects of Vista tiling. Discharged; see below. ACTION: LGR to coordinate a kick-off meeting between himself, RGM, NCH and Kona to discuss the way forward on list-driven IR/optical pixel analysis Discharged; LGR is progressing with some rapid prototyping of a demo service built on VO infrastructure. ACTION: NCH to suggest to AL/SJW that formal feedback on user priorities (perhaps via a web forum) might be helpful in planning WSA enhancements like user-defined junk flagging. Discharged; UKDISS CSS seems happy with informal, ad hoc arrangements so we continue on that basis. ACTION: MAR to include a link from WSA publications to UKIDSS science publications list maintained by SJW Discharged ACTION: NCH to pass the buck to AL/SJW/PWL concerning an extinction attribute in the GPS: "we'll do what you tell us, but if users screw up, on your heads be it" Discharged; ball is now in GPS Survey Head's court on this one. Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; will get sorted (honestly!) as part of the general ingest DB schema revamp to facilitate rapid generalised photometric/astrometric recalibration after DR3. ACTION: MAR to look after Windows system/DB backups prior to the new operator arrving Some inconsistencies and holes in the existing disaster recovery policy; needs further attention. ACTION: MAR to take forward the possibility of archive and/or on-the-fly colour image production. Continues; line put in the forward plan for Q1'08 Actions carried forward from 11/01/08 meeting: ---------------------------------------------- ACTION: MAR to incorporate basic info on using flat files Continues ACTION: RSC to coordinate migration of the residual CVS-tracked projects into SVN. Continues ACTION: ETWS to examine CU21 to see if it can be tweaked to propagate template schema changes into pre-existing non-survey static SQL scripts Continues Specific points and new actions: -------------------------------- Project management: WFAU Rolling Grant review took place on Tuesday with PMW, RGM, NCH and AL in attendance. Case for support for residual VDFS development and continuing operations was presented; general feeling was positive but it is not known when we will be notified of any decision... The team reviewed the Q1'08 plan in the light of recent prioritisations, and also looked ahead to the Q2'08 plan. Concerning VISTA Survey liaison, NCH noted that it was time to progress things and get further detailed input from EPS PIs over database requirements ACTION: NCH to email the VISTA EPS PIs to get further input on database design for their respective surveys. WFCAM & VISTA updates: No news this week. Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: MSH reported further investigations as to optimisation of the WSA LAN. NFS mount tune-up with reference to RAID stripe size seems to be producing reasonable performance boosts, so this is being pursued with all possible haste. MAR noted that copy performance between the Windows DB servers seems to be rather poor also; NCH noted that this has been noticed in the past when copying to 2TB LVs. ACTION: MSH to check RAID stripe sizes etc. on the Windows side as a check on file transfer performance when writing to 2TB LVs. MAR has tested out JDBC connectivity of new SQL2005 server hatshepsut and all appears well; the team decided to shunt SegueDR6 onto this machine rather than wait for SQL2005 installation on batch server thutmose. ACTION: MSH to liaise with IT Support over set up of NFS file share from hatshepsut WSA Operations: ETWS reported: - Created a backup directory to hold backups of database driven products (mosaics, stacks, diffs). - Started installation of third party software on djoser after its debian has been upgraded to debian 4. - Started update of all file timestamps and their entries in the WSA to be in sync with CASU files. - Resumed data transfers from CASU. NCH noted that it is important to keep up momentum on preview JPEG and image metadata ingests so that flat file access is enabled and UKIDSS can check early 07B data asap. ACTION: ETWS to JPEG and CU3 early 07B data asap RSC noted the necessity to tidy up some residual calibrations on retransfered/ingested data ACTION: NJC to look after LAS/GCS recal issues following DR3 release. Hardware Since new SQL2005 server hatshepsut is now up and running, the team decided to install SegueDR6 on that machine (rather than thutmose) ACTION: MSH to liaise with IT support over a network file system for hatshepsut server and linux clients. MSH enquired about maximum partition sizes given the current upgrade to Debian linux 4.0; RSC also pointed out that we'll have a mix of 32- and 64-bit systems for the foreseeable. The suggestion was to stick to the maximum LV size allowed by 32-bit OSs with due regard to optimal stripe sizes with respect to the NFS optimisations recently identified as above. MSH noted that Eclipse are building the new 48TB NAS box so this next chunk of storage should be arriving very soon. Software: The team met on Wed pm to discuss NJC's latest data model for synoptic surveys, and strategy for coping with VISTA tiling. On the latter, NCH will be seeking clarification on tiling strategy from VISTA EPS PIs while at the same time NJC will consult with Nuria Lorente (Survey Area Definition Tool SW developer) over likely operational modes before we decide on the archive end strategy. RSC reported: "Helped diagnose the problem in releasing gpsDetection, submitted a bug fix to CU19 that allowed the DR3 database to be completed and then I manually finished it off by adding the gpsDetection indices that were missed out during the interruption. Finished synchronising the various SVN branches - submitting all the important DR3 fixes that were left in trunk into the release_4 branch and tagging this release of the curation software as DR3_FINAL. Then updated trunk with the complete set of release_4 fixes, before creating a new release_5 branch that contains the enhancements to CU13/14 made for releasing the 10001 programme as well as the CU4 bug fix. I then re-synchronised the dev_1 branch that has lied stagnant since last August due to DR3 work, allowing me to start submitting the more far-reaching proper fixes to some problems that occurred during DR3, such as the revised use of database indices. I've also created a BranchNews wiki page on trac to keep everyone up-to-date with the latest SVN branching plans. On the TWiki, created a SoftwareInstallation page for everyone to submit the list of packages that they require on the curation servers, so we don't have to keep bugging IT support during the Debian upgrade. Also updated the CreatingNewNeighbourTables page and CuPerformance is now complete for DR3 bar the CU13/14 times that I've delegated to NJC." Survey Data Release: MAR reported that the full DR3 (including GPS) completed earlier in the week and is ready to copy on to the public database servers; some juggling of existing DBs is required to make room for it in order to minimise disruption to the archive service. Further unexplained problems with Windows in file transfers have been complicating what should be a simple process... NCH noted that this is a top priority and must trump any other conflicting activities. Non-survey Data Release: MAR noted that another user enquiry has been received concerning access to open time flat files outside their proprietary period. RSC suggested that a small mod to the existing metadata mirroring process should handle this, and also suggested that semester release may be adequate at this stage. ACTION: RSC to set up WFCAMOPENTIME with semester 05A as a test to enable flat-file access to non-survey data outwith it's proprietary date. NJC and ETWS have been putting the finishing touches to non-survey stack/mosaic support, using programme 05A/100 as a test case for Mark Casali and student at ESO. Astrogrid deployment: ETWS and MSH noted that BestDR1 is to be reinstalled on thutmose from backup files from JHU as it is required as a test-bed for a VO application. ACTION: MSH to set up new JBOD for batch server thutmose to receive further SDSS databases. MSH noted a heads-up in that AG are currently discussing reorganisation of community user registration via certification and that this is likely to impact DSA access through WFAU services at some point in the near future. Miscellaneous: NCH noted that the NAM 2008 at Queen's University Belfast is now open for registration, with wide field surveys (incl. Gaia & PanSTARRs) scheduled for the Thurs/Fri Apr 3/4: http://nam2008.qub.ac.uk/ NCH and MSH will be going; NCH asked the other team members to consider attending and then we will co-ordinate who is to present what (deadline for registration for oral presentations is 29th Feb).