From nch@roe.ac.uk Fri Oct 19 13:19:03 2007 Date: Fri, 19 Oct 2007 12:11:28 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU VDFS Science Archives weekly project meeting minutes, 19/10/07 Minutes of WFAU VDFS Science Archive meeting: 19th October 2007 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RSC, NJC, MAR, PMW, JB Apologies: BCW, JPE, AL, RGM, JMS, ETWS, LGR, MSH NB: DONM: 10am, Friday 26th October 2007 in the Plate Library Actions discharged this week: ----------------------------- ACTION: NCH and PMW to meet on Mon 4pm to attend to various management admin. Discharged; see Project Management below. ACTION: NCH, MSH and JB to work with Eclipse on stabilising the servers Discharged; all quiet on the Hardware front ACTION: MAR to contact AA to ask for his advice on PI-programme registration verification Discharged; AA pointed us to the relevant JAC web pages that contain PI data for cross-checking incoming registrations. Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; will get sorted (honestly!) as part of the general ingest DB schema revamp to facilitate rapid generalised photometric/astrometric recalibration after DR3. ACTION: RSC to progress implementation of automated replication of the ingest DB file metadata on a public server. Continues; RSC has located and fettled the appropriate scripts, and will now test against the restored ingest DB. A similar solution is proposed for proprietary-lapsed non-survey data access (see Non-survey Data Release below). Some discussion took place as to the names of these databases: FlatFiles? WFCAMfiles? ...? ACTION: JB to restore BestDR1 to thutmose from LTO tape at a convenient time (but no rush!). Continues; NCH found the tapes and noted that 3 out of 5 are unreadable (see Hardware below); JB suggested there *may* be another copy lurking somehwere amongst the general WSA external catalogue backups. There was some confusion over how urgent this action is, since BCW didn't seem too worried if it is not possible to restore BestDR1... Actions carried forward from 12/10/07 meeting: ---------------------------------------------- ACTION: JB & ETWS to restore SegueDR6 on thutmose at their convenience (i.e. no hurry given the present circumstances) NCH noted that it'll almost certainly have to be restored under SQL2005 rather than 2000, so new server hatshepsut will be needed rather than thutmose... ACTION: NCH to bug site services concerning unacceptable infrastructure problems. ACTION: ETWS to look into the examples of unavailable jpegs. Specific points and new actions: -------------------------------- Project management: The team filled in progress for end Q3 and reviewed the Q4'07 plan. WFCAM & VISTA updates: NCH noted that Luca Rizzi is with us next Wed (pm; arriving mid-afternoon), Thurs, and Fri (am) for general getting-to-know WFAU WSA operations. Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: JB noted that the UKLight connection is down at the moment (not known how long it will take to fix), but that we are not transfering 07B data yet anyway so no probs (the team noted that processing continues apace at CASU; nothing released as ok_to_copy though...) WSA Operations: NCH noted that tenacious efforts by JB have restored the WSA ingest database to it's state as at 2nd Oct when the major hardware failure happened, so DR3 preparation is back on track. Application of photometric recalibration has finished (CU8); see below for DR3 preparation schedule as it stands. Hardware As noted above, 3 out of 5 LTO-2 tapes written in February 2004 have been found to be unreadable while attempting to restore SDSS BestDR1 for a VO application test. Suspicion is that magnetic printing is the most likely culprit, so now is the time to consider a co-ordinated campaign of winding archival tapes (WFAU would be interested to hear what is being done at CASU and JAC for the raw WFCAM data tapes). ACTION: NCH to ask MSH to spec out a tape winding machine for LTO. JB noted that the copy from disk01 of the venerable file server djoser is complete, so this oldest machine in the archive can be retired from active duty in storing and serving WFCAM data (it will remain as a general development server). ACTION: JB to get IT Support to redirect the disk01 mount point to the relevant NAS box partition. Software: RSC has robustified CU8 for non-surveys, and has worked on enhanced broken file logging for ease of operations; NJC has worked a little on galaxy photometry issues for his forthcoming paper. Survey Data Release: Updated schedule following on from recent work over the last week: Weeks What: 1.0 QC1 0.0 CU5 (diff images for the GPS - after QC, but in parallel with the following and it's very quick anyway) 1.0 Quality bit flagging 3.0 CU7 (source merging from scratch again, unfortunately) 0.0 CU13/14 for DXS/UDS (in parallel with shallow survey CU7s) 2.0 Final CUs. NCH noted that the following bits of fettling need to be done in addition: 28/29 Dec pv2_3 keyword and database updates (JB); NIGHTZPT for Y frames need 0.08 added (NJC & RSC) and final cross-semester QC1 (NCH next week) should be applied following 06B/07A QC1. Further actions: ACTION: NJC to contact Seb Foucaud to check up on UDS preparations Non-survey Data Release: MAR noted that more non-survey registrations have arrived for 07B ACTION: MAR to verify latest registrations and ensure info is communicated to JB (following recent rearrangement of log folders on disk01) MAR reported that JAC have been in touch over general access to non-survey data once this has passed it's proprietary period. The team agreed the best way forward is to enable flat-file access after a suitably conservative period from observation date (18 months?) to allow for the usual 12 month proprietary period plus data flow lag. A mirrored metadata database will be set up and maintained on a public server to provide a single database of all proprietary-lapsed non-survey data. AA will be communicating with the Board over these issues to get their formal approval. NCJ and JB asked about the status of the 07B WFCAM campaings as regards registration. Turns out only one (GABARIT) is currently registered. ACTION: NCH to nudge WFCAM Campaing PIs to remind them to register if they want flat-file access through the WSA. Astrogrid deployment: No news this week. Miscellaneous: NCH reported that the referee's report has come in for the WSA paper, and is most positive and constructive. Revision/resubmission will be done asap...