From nch@roe.ac.uk Fri Jul 13 09:16:07 2007 Date: Fri, 6 Jul 2007 16:02:47 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , John Taylor , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU VDFS science archive weekly project meeting minutes, 6/07/07 Minutes of WFAU VDFS Science Archive meeting: 6th July 2007 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RSC, PMW, ETWS, MSH, LGR, RGM Apologies: JDT, BCW, JPE, AL, JMS, NJC, MAR, JB DONM: 10am, Friday 13th July 2007 in the Plate Library Actions discharged this week: ----------------------------- ACTION: JB to update the twiki with the locations of the disks on servers Discharged; see http://apache.roe.ac.uk/twiki/bin/view/WFAU/ServerNames (action was not minuted in last week's minutes - apologies) Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; will get sorted (honestly!) as part of the general ingest DB schema revamp to facilitate rapid generalised photometric/astrometric recalibration. Actions carried forward from 29/06/07 meeting: ---------------------------------------------- The following from last time continue: ACTION: NCH to discuss and review AstroWISE interfacing with MAR and JDT. ACTION: NCH to start some hard-nosed negotiations with the UKIDSS PI and CSS concerning the contents and timing of DR3. - CONTINUES; NCH is getting increasingly worried about this now, so has sent an email back up the dataflow system to get some idea of timescales (CCd to UKIDSS PI and Consortium Survey Scientist). MJI has noted that CASU will probably require at least a month to sort out and process the data from 25th Oct to 19th Dec; responses from the other parties concerned are eagerly awaited. ACTION: AL to check on the exact wording of the survey release policy with reference to world release of proprietary survey data. - CONTINUES Specific points and new actions: -------------------------------- Project management: The VDFS extension FECed JeS has been prepared, approved by ERI and HoS, and will now be submitted. PMW noted that progress tables for June need to be filled in (this was done during the meeting), and the forward plan for Q3 prepared. ACTION: NCH & PMW to meet on Mon 9th at 2pm to prepare the Q3 plan. WFCAM & VISTA updates: Stop press news from VISTA project leader JPE: "Regrettably it now seems that M1 polishing will not be complete until the beginning of October. The current schedule implies that Science Verification probably won't be able to start until late May/early June, and so public surveys probably won't be able to start before the 2nd half of 2008. On a more positive note we currently have the camera mounted and running cold on the telescope for electromagnetic interference tests, and there are no obvious effects (from quick look - we only took the frames yesterday). Of course the project continues to do what it can to pull the schedule forward, but there are many things that just can't begin to be done without the M1." Comments and issues arising from CASU fortnightly minutes: No new minutes as of 6/07/07 am. Networking: Transfers continue, we are now halfway through 07A transfers, still waiting on the first half of 06B. ETWS noted that transfer rates are running at 2.8 to 3.0 MB/s. PSB, NCH and JB have been communicating on how best to progress some investigations/experiments aimed at increasing this somewhat sluggish performance. NCH is at the IoA on Tuesday for a GAIA meeting and will take the opportunity to chat with PSB then. WSA Operations: JB reported: "CU1, CU2 and CU3 have all been running this week and the variety has allowed me to discover more about how they interact when running together. Progress can be seen on the monitoring page but the summary to date is; CU1 is approximately half way through 07A, CU2 is two weeks in and CU3 is a week in. Backups went as normal and following a request from MAR I checked that SQLAgent was running on all the SQL servers." Hardware: JB reported: "A new large (42U) rack has arrived and has been found somewhere to be stored until we find time to swap it in for one of the smaller racks - though the space isn't urgently needed at the moment (it arrived faster than expected too!) The monthly reboots went as planned with no issues arising. Eclipse has been called about a failed fan on the new NAS box, nothing critical as the machine is barely in use at the moment, it has been completely setup though and it's volumns mounted. We had a meeting with the guys from IS who run the ECDF and both we and they came out with more questions than answers with an intention to remedy this, however from the point of view of the VDFS it looks like we will not be able to make use of there services in the near future as they are not really set up to deal with our volume or external access requirements. Tests are now being run by Mark on the SAS and SCSI arrays on the new SQL server which we should have results from next week when we will look at putting SQL through it's paces too." MSH gave advance warning of some rearrangement of the middle rack in server room C1. This will involve downtime on the public server amenhotep (hopefully the WSA interface can be pointed at the batch server thutmose during this work to minimise the impact on users). MSH reported also that he and JB are benchmarking the new database server with some low-level file dump tests. Software: NCH noted that he has discovered big bugs in the seaming algorithm used to create the GPS (and recreate the LAS) in the final released version of UKIDSS DR2. This has all been noted on the release history web pages and an email sent to the user wsa_announce list; NCH has also contacted some LAS users personally. The seaming flags need to be fixed asap in the LAS and GPS source tables; NCH will do this as specific test cases should be checked to make sure all is well. ETWS reported: "Enhanced CU1 so that it distributes date directories over all available disks to improve IO performance of all ingest CUs. Updated the monitoring tool to take into account 07A data observed in 06B. Included more functionality into the DataDeamon. This helps other users than scos to determine the machine where to run code with least interference with normal production." RSC reported: "I've initiated some development work to overcome some problems with the software architecture as it currently stands that were identified in our software meetings. The first stage only affects code that I'm familiar with testing so will be committed to the SVN trunk, the second stage has a wider scope so will be committed to a separate branch and I will work together with Eckhard to develop and test these changes before merging into the trunk. None of these developments should be merged into the current release branch, instead they will be released when a new release branch is created. This work has been delayed though by me having to reinstall my laptop's OS due to system files on the hard disc becoming corrupt. I've also taken the opportunity to test out a new Python IDE (integrated development environment), called Eric, that offers good SVN integration." ETWS, NCH and RSC all noted that the new SVN system seems to be working out well. RSC has put extensive notes and links on a TWiki topic at http://apache.roe.ac.uk/twiki/bin/view/WFAU/UsingSVN JB noted that from an operational point of view he now has a working SVN checkout with some new fixes in it and is getting the hang of the new environment, should be au fait with it by Monday. Survey Data Release: Now word as yet on early 06B NDR data delivery to CASU; it now seems inevitable that DR3 is going to be late, in whatever form UKIDSS decide they want it ... Non-survey Data Release: Nothing new to report this week. Astrogrid deployment: MSH asked about the status of the XMM and 6dF catalogue databases with regard to WFAU's DSA, which seems to be getting widely used amongst the AG community. NCH noted that he was expecting an email from Mike Watson of Univ Leicester concerning XMM, while ETWS noted that if there are any parsable schema scripts for 6dF, metadoc preparation can be done easily. NCH will forward info on XMM to MSH when it arrives, and suggested we speak to MAR about the status of 6dF on Tuesday. Miscellaneous: JB, ETWS and NJC have all registered contributions for ADASS XVII (that piccy of Stonehenge on the homepage looks suspiciously like one of the standard Windows XP desktop backgrounds...)