From nch@roe.ac.uk Thu Jun 8 17:30:54 2006 Date: Mon, 22 May 2006 11:52:17 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU WSA weekly meeting minutes, 19th May 2006 Minutes of WFCAM Science Archive meeting: 19th May 2006 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: JB, MAR, ETWS, NCH, NJC, PMW, RSC, MSH, JMS Apologies: AL, JPE, RGM DONM: 10am, Friday 26th May 2006 Actions discharged this week: ----------------------------- Actions partly discharged but continuing: ----------------------------------------- None this week Actions carried forward from 12/05/06 meeting: ---------------------------------------------- ACTION: JB to review Hardware/OS/DBMS design doc to note areas that need updating and itemise new sections required - CONTINUES ACTION: ETWS and JB to put a TWiki note on the intranet detailing the outcome of the UKLight meeting at NeSC. - CONTINUES Note that these are on hold until after DR1. Specific points and new actions: -------------------------------- Project management: NCH asked JMS about the putative UK VDFS review in October; JMS agreed that he and JPE need to progress this. ACTION: JMS to contact JPE regarding progress on October UK VDFS review organisation MAR will be attending the UKIRT Board meeting (Mon/Tue) on Tues PM to fly the WFAU flag. NCH provided WSA update notes for AA's paper, and also provided input and read contributions from AL and SJW. WFCAM update: Nothing new to report this week Comments and issues arising from CASU fortnightly minutes: The team noted the minutes of the meeting of 10th May. Only substantive point is a broken file update from the archive end for 05B, provided by JB (acknowledging that we hadn't fed back some of this to CASU in time for their last meeting): The number of broken files for 05B were in total 36 with: 2 files that broke on download (power cut related?) 4 test files erroneously included in the upload sequence 1 engineering dark sneaking into the pipeline 29 catalogues that included NAN values two of the test files were also included in two sky (Y and Z band) files that require reprocessing and reingestion of the Y and Z band data for that night comprising: 2 flat frames 4 dark frames 27 sky frames 123 confidence frames 257 science frames 121 stack frames 121 catalogues (a total of 655 files) yielding a grand total of 691 files required to be retransfered/reingested etc. Somewhat larger than the number reported in the CASU mintutes, but still impressively small considering the large numbers of files handled with no problems. NCH noted the comments from RGM concerning passband merging at the archive end, and noted that a spate of recent bug-fixes and enhancements should have taken care of these issues for DR1. Networking: UKLight things continue to progress slowly, an application to submit has been drafted and will be circulated for comments asap. WSA Operations: JB reported: "The system backups went as normal - thutmose is now also being backed up. I believe the passwords for the sa user in SQL have now all been aligned. DR3 is now on thutmose awaiting SQL restoration. Issues with test files being included in the Y and Z band for the 26th October 2005 has been notified to CASU and reprocessed, these files are now awaiting reupload and ingestion. Apart from the Y and Z band for the 26th Oct. 2005 05B has now been ingested in it's entirety. Ingest for version three 05A has started - we have got to 10th April 2005. Download of version three 05A has reached 23rd April 2005." With reference to the required reingestion of ZY for 26th October: ACTION: NCH to deprecated the buggy versions, providing a helper SQL script for any future requirement to do similar fixes ACTION: JB to retransfer and reingest the bug-fixed data. For issues regarding software development affecting operations, see Software below. Hardware: JB noted that a helpdesk ticket has been put in for khafre to be installed, and that there was a minor RAID array glitch on the venerable file store node djoser at the beginning of the week; if the same disk continues to give problems then it will be swapped out. NCH notes (after the meeting!) that Eclipse have provided a 300GB U320 SCSI disk spare for the new server thutmose in case of RAID degradation. Software: A major part of this weeks meeting was taken up with a general discussion concerning the (negative) affect of development software check-ins to the CVS on operations, especially when approaching release. NCH asked RSC (as the chief culprit!) to investigate branching in the CVS to prevent future punch-ups between the operations and development sides, suggesting a chat with JDT as Astrogrid CVS guru. RSC summarised the pros and cons of two different modes of employing branching in CVS, and the team agreed to adopt the second mode whereby the main branch is the operations branch, a development branch being created as and when, with any merging to be controlled by RSC. RSC has put a TWiki note up dealing with some of this at http://apache.roe.ac.uk/twiki/bin/view/WFAU/CVSBranching. In any case, NCH asked that no more unnecessary changes should be made on the main CVS branch: only essential bug fixes are allowed from now on. NCH reported completion of the bug fixes and enhancements to CU7, and has checked in the changes (to the main CVS branch!) for DR1. The only outstanding issue that has not been resolved is the problem of different tilings of UKIDSS LAS YJ and HK in a small number of areas; the situation will need to be looked at closely when CU7 has been run in anger for 05B/DR1. ETWS reported mainly doing bugfixes and also installation of cfitsio with large file support on djoser, testing different visualisation software to display large fits files. RSC reported: "Updated catalogue metadata for the 11th December 2005 following an ingest bug using an old script that's now also been updated and made more accessible. Updated all codes to ignore filenames in the database that contain "PixelFileNoLongerAvailable:". Committed various code documentation and design upgrades to CVS and dealt with subsequent bugs (sorry for any inconvenience). Investigated splitting code development into two CVS branches, one for operations and one purely for development. Helped Johann investigate the missing provenance information leading to the discovery of another dodgy sky frame. Investigated the extent of the effect of a CFITSIO upgrade to our code base but haven't upgraded yet." MAR reported working on adding the UKIDSS PTS into the various UI forms which involved some refactoring of code to make any further additions easier. NJC has been involved in WFMOS meetings and discussions for most of this week. Survey Data Release: Reiterating the DR1 preparation schedule as it currently stands: DR1 release at: 14/7 (NB: databases to be DR1 & DR1PLUS !!) Copying/transfering/ 7/7 backups etc. start UKIDSS pre-release 1/7 checks start Final CUs: 16 (incl. 23/6 SDSS DR3 hopefully), 18,19 start CUs 2,3,4,7 for DXS 20/6 and UDS start Archive QC2 starts 14/6 CU7 for wide/shallow 7/6 surveys starts Things beginning to get a little tight as regards QCing in time to do allt hat is necessary... Non-survey Data Release: MAR reported 4 non-survey projects released on Monday to users. Four more new registrations this week. Astrogrid deployment: MSH reported that the new WSA forum was set up and nearly ready to go, after some DNS issues. JB (delegated as moderator) will be emailed when things are ready for a try-out. Miscellaneous: RGM (taking some time off from nappy duties) suggested we may want to look into MS SQL Server 2005 to investigate some of it's new features (e.g. integrated HTM indexing). The team agreed that if MSH has time to investigate installation, and provided it wasn't done on one of the main catalogue servers, this could be useful. MSH accepted a "scrap heap challenge" to pull together a motherboard and a few bits and pieces (possible an old grendel node?) to get an instance of SQL 2005 up and running for some preliminary tests. ACTION: MSH to set up a Windows box and an instance of MS SQL 2005.