From nch@roe.ac.uk Sun Oct 28 12:05:30 2007 Date: Fri, 26 Oct 2007 15:31:05 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 26/10/07 Minutes of WFAU VDFS Science Archive meeting: 26th October 2007 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RSC, NJC, MAR, PMW, JB, ETWS, LR (JAC) Apologies: BCW, JPE, AL, RGM, JMS, LGR, MSH NB: DONM: 10am, Friday 2nd November 2007 in the Plate Library Actions discharged this week: ----------------------------- ACTION: JB to restore BestDR1 to thutmose from LTO tape at a convenient time (but no rush!). Discharged; tapes are unreadable so alternative arrangements have been made with US colleagues/institutions. ACTION: ETWS to look into the examples of unavailable jpegs. Discharged as the consensus is that this was a problem to do with tempwsa during the fire-fight in the first couple of weeks of Oct. ACTION: NCH to ask MSH to spec out a tape winding machine for LTO. Discharged; see Hardware below ACTION: JB to get IT Support to redirect the disk01 mount point to the relevant NAS box partition. Discharged; this is now done and djoser crossmounts have been removed; next in line is sneferu. ACTION: NJC to contact Seb Foucaud to check up on UDS preparations Discharged; preparations continue apace; some concern that some reprocessing is still being requested at this late stage... ACTION: MAR to verify latest registrations and ensure info is communicated to JB (following recent rearrangement of log folders on disk01) Discharged; see Non-survey Data Release below ACTION: NCH to nudge WFCAM Campaing PIs to remind them to register if they want flat-file access through the WSA. Discharged; three out of the five are presently registered. ACTION: NCH to bug site services concerning unacceptable infrastructure problems. Discharged by stealth; power trip in C1 is being replaced to prevent C1 going offline outside office hours when no-one is around to pull the big reset lever; IT Support, JB and MSH are planning rearrangement of things in C1 to allow better air circulation and cooling etc. Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; will get sorted (honestly!) as part of the general ingest DB schema revamp to facilitate rapid generalised photometric/astrometric recalibration after DR3. ACTION: RSC to progress implementation of automated replication of the ingest DB file metadata on a public server. Continues; RSC has located and fettled the appropriate scripts, and will now test against the restored ingest DB. A similar solution is proposed for proprietary-lapsed non-survey data access (see Non-survey Data Release below). Some discussion took place as to the names of these databases: FlatFiles? WFCAMfiles? ...? Actions carried forward from 19/10/07 meeting: ---------------------------------------------- ACTION: JB & ETWS to restore SegueDR6 on thutmose at their convenience (i.e. no hurry given the present circumstances) NCH noted that it'll almost certainly have to be restored under SQL2005 rather than 2000, so new server hatshepsut will be needed rather than thutmose... or a new installation of SQL2005 on thutmose of course. NCH noted that hatshepsut is currently groaning under the weight of the full-blown SSA load anyway. Specific points and new actions: -------------------------------- Project management: NCH welcomed Luca Rizzi who has been visiting over the last day or so to get to know the troops and for general discussions on WFCAM archiving ops. NCH and PMW noted that yesterday's VDMT has been rescheduled to 8th Nov. WFCAM & VISTA updates: LR gave a summary of the latest JAC tests on the channel "edge" (and related) features, noting that the source of the problems has been identified tentatively with an electronic module in camera #1 and further work is planned to investigate/fix. PMW quoted Eli Atad as noting that Vista M1 should be polished by the end of November. Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: JB noted that UKLight appears to be fixed again as transfers are running fine with CU1, three days of 07B done already. Restore of BestDR1 from tape broke due to a corrupt tape, alternate methods of recovery are being investigated by MSH and RGM. WSA Operations: UKIDSS DR3 preparations continue; mostly QC this week. JB notes that CU1 is up and running again, CU8 has been finished (both files and DB), CU2 has been run to update the DB with the 20070101 fixes. Hardware JB noted some battery maintenance is needed on RAID controllers on the public server; the team agreed they should all be done as a matter of preventative maintenance. Original and venerable file server djoser has been retired from active service for processed file storage; it's degraded RAID array will be fixed at some convenient point to provide development space. As regards the unreadable LTO tape business, MJI has reported no problems at CASU, noting that they have reread tapes a few times over the last few years which may have helped. LR noted that JAC have not tried to reread archive LTO tapes, and have no winding policy. JB noted that the Veritas BackupExec SW and the HP hardware can't and/or won't just wind a tape (may not be this simple with LTO anyway?), so he is going to investigate some alternatives (e.g. spooling them on the LTO-3 drive on the good ol' SuperCOS HP-Unix system cosaxp6). JB noted that the scheduled November 10th power downtime in server room C1 is now cancelled, and that the date has yet to be rescheduled, probably the first or second weekend in December. There was concern amongst the team that this is rather soon after DR3 release and close to the ESO Workshop, and that any fall-out could impact any live demos etc. and hence be rather embarrassing. NCH has requested that Premises keep him in the loop over the arrangements so that as much advanced notice as possible can be given to external users as to when the downtime is likely to be; NCH also thanked JB/MSH for being willing to work outside normal hours to make sure the archive systems are looked after as necessary. Software: NJC reported: "I have been working with Ross to finish off the recalibration for DR3. This was mainly modifying the log files provided by Cambridge to do the Y-band nightly zeropoint and testing. I have also been working on my ADASS conference proceedings, including producing a new ERM for the recalibration design. This will be useful for future documentation. I have done a little bit more towards my galaxy photometry paper." RSC reported: "I identified a problem in the SVN repository that the SVN keywords, such as Id, Author, weren't being automatically updated for new files that entered the repository after the switch from CVS. Turns out that, by default, the SVN client doesn't set these properties when adding new files, and I've provided an alternate client config file for everyone to use that fixes this problem. Notes are provided in the SoftwareIntroduction and UsingSVN TWiki pages. I also went through the SVN history to correct the properties on all the files that were added to the repository after we switched from CVS. I've given CU7 a new -p/--prepare option to just prepare the framesets and ingest them into the mergelog without source merging to help prepare the LAS release. Also I've made sure the -n/--new option can handle the case whereby the source table doesn't exist to begin with. CU16 has been tested for the new udsDetectionXSource table, and CU19 has been modified to release all non-deprecated UDS detections, now that the UDS intermediate stacks will be quality controlled for variable source studies. I've tidied up the CU8 script a bit following our recent recalibration experiences. Also I've finished implementing the broken/missing file list logging for CU8 and ProvenanceFiller (trac ticket:96). Work on improving the UDS cross-talk flagging and the modified SyncDb script for metadata-only releases continues." Survey Data Release: Updated schedule following on from recent work over the last week: Weeks What: 0.2 QC1 0.0 CU5 (diff images for the GPS - after QC, but in parallel with the following and it's very quick anyway) 1.0 Quality bit flagging 3.0 CU7 (source merging from scratch again, unfortunately) 0.0 CU13/14 for DXS/UDS (in parallel with shallow survey CU7s) 2.0 Final CUs. NCH noted that an enquiry about science exploitation of the intermediate stack catalogues from the UDS has been received via UKIDSS UDS Survey Head Omar Almaini. This has been addressed by NJC adding in a cross-neighbour table definition for the survey for DR3; MAR is synchronising the QC with that being applied by the Nottingham folks independently, so at least some of the intermediate stack catalogues will be released in DR3 to service this kind of science usage (although there does remain the question of bringing all the UDS reprocessed intermediate stacks/catalogues into the archive...) NCH also noted that an issue has arisen over small LAS retiled areas; apparently some reobserving has been done to sort it out, but exactly what has been done is a little unclear. RSC has kindly added a new option to the source merger to make framesets only (without full source merging and seaming) for test purposes; NCH noted that it will probably take some fiddling around next week to sort out this problem as part of the overall wide-shallow all-semester QC that has to be applied to remove any repeat frames left after normal QC1. Blimey... Other than that, MAR noted that QC1 should close out early next week, and JB noted that a few final quick tweaks (fix Dec 28/29 astrometry, update provenance) need to be applied before forging ahead with CU5 (GPS difference images), DXS CUs 13 & 14 (stacks/cats), and then quality-error bit flagging followed by the biggy of running CU7 (source merging). The team discussed some housekeeping options for the coming weekend, and agreed it might be a good idea to shrink the bloated G: detection file group. ACTION: JB to set dfg_1 shrinking in the WSA this weekend. Non-survey Data Release: JB and MAR have brought the latest 15 non-survey registrations, including some 07B projects/campaigns up-to-date in the WSA ingest DB to enable flat-file access. JB reported: "CU21 has been run so another fifteen (all registered in the last couple of weeks) non-surveys are now in the DB, I am to pass MAR info so that they can have their accounts activated to give them flat file access. CU4 has also been run for both these and some previous non-surveys that need data ingested. Investigations continue as to the current status of u/05b/j3 data, thanks to Luca for help with this." ACTION: JB to ngest CU4 non-survey data before running detection table shrink on G:\ Astrogrid deployment: No news this week. Miscellaneous: WSA paper revision continues; ETWS and MAR are setting up the requested shift of NCH's personal web page links to the design docs onto the main web site; AL has provided some bon mots concerning Astrogrid DSA.