7th January 2005 Hoyle committee meeting room 10.30 - 12:50 Present: PSB, JRL, MR, RGM, MJI, DWE, STH Apologies: WJS, EGS, RWA Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Report on recent VDMT meetings 4. WFCAM commissioning update 5. Status of ESO documents for FDR 6. Status of Data Centre Archives 7. INT WFS products release 8. RAID arrays update 9. APM building news 10. AOB Minutes ------- Marco Riello, who started on Tuesday, was welcomed to the group. 1. Actions from last meeting MJI has made no further progress in comparing the FIRES project reduced data with ours. He said that it was sensible to carry on with this since it was the best data available - ongoing <<<< STH JAC have taken over the task of generating the MSBs for the time being. However, STH will still monitor the MSB status. JRL has not had time to work on the ZPN WCS problem in IRAF due to the commissioning. There was much discussion to give this task a higher priority. It was agreed that after the urgent stuff is out of the way he would work on this (after next week) - ongoing <<<< STH Sandy Leggett has acquired NIR spectrophotometry to help with the photometric calibration. This will be a work package in the future. WJS has got hold of a copy of the latest SDT document WJS has asked if the positions of the two guide stars could be combined to produce a single x,y correction for VISTA. The outcome from this is ... STH has generated modified versions of JRL's diagrams EGS has released the revamped INT WFC web pages. RGM said that there were a few things missing. He will contact EGS regarding this. <<<< (There was another comment regarding JMI's database which I didn't quite catch.) MJI raised the issue of ESO workstations with JMS. PSB has now acquired the necessary information to obtain the software needed to set up an ESO workstation. PSB said that he might start off by trying this on an existent Linux system and if no success is to be had then he will try on a dedicated machine. MJI found out from a VDMT meeting that the UK VISTA URD has been signed off and released. JRL has obtained dome flats by flying out to Hawai'i and doing them himself WJS provided STH with the latest overhead figures (eg. time to change filter etc.) STH raised the issue at a VDMT meeting about the level of detail, for example, the pseudo maths code, that needed to go into the ETC document. It was decided that you might as well put in the proper maths since that would take about the same amount of time. PSB has redone the templates section of the Calibration Plan document STH has reviewed the cascade diagrams PSB did a short-term review of all the documents MJI has proof-read Section 2 of the DRLD PSB added the photometric calibration equations (~1 page) from the Calibration Plan document to the DRLD STH has added the diagrams needed for Section 3 (Functional Description) JRL has written Section 5 (DRL Data Structures) PSB has written a skeleton for Section 7 (Data Reduction CPL Plugins) MJI formulated the maths for the various quality control parameters for inclusion into Section 2. JRL has written Section 11 (Data-Processing Table) MJI raised the issue of Gavin Dalton's inclusion in the FDR at the VDMT meeting ALL provided reports to STH RGM supplied comments for the grant bid. MJI said that he would make sure NAW that bound copies of the grant bid would get produced and distributed <<<< 2. Comments on WFAU minutes It was noted that the many of the hardware problems at WFAU had been sorted out. MJI said that they had been shown around during the last VDMT meeting. PSB noted that the network problems had been solved. (Not 100% sure what Pete was on about here.) It was also noted that the VDFS web pages action had been discharged. While on the subject, MJI suggested that DWE show MR around our web site. <<<< 3. Report on recent VDMT meetings There have been 2 VDMT meetings since the last CASU meeting. The previous one was a telecon so won't be reported. MJI reported on the last VDMT meeting which was held yesterday in Edinburgh. The ESO FDR will be held on 31 January at Garching. The confirmed attendees will be MJI, PSB, JPE, JMS and Alistair McPherson. It was also suggested that WJS, Steven Beard and Martin Folger also attend in person. STH and JRL should attend via telecon. There was agreement regarding the ETC document. An updated version would be distributed today and MJI (?) would collate the comments over the weekend. This would then be submitted on Monday along with the ETC perl cgi script. <<<< Although the DID was not officially part of the FDR it was regarded as a useful subset. It was suggested that PSB send the data dictionary files (apart from VISTA TCS data dictionary) to JMS so that he can submit them to ESO by Monday. <<<< It was mentioned that our grant review would be held around the end of February. There was discussion regarding when the UK VDFS would be reviewed and what should be within the submission. JMS said that it should be at the functional specification level without too much detail. The submission of the document should be around 15 April, preceded by a VDUC meeting in February to sign off the document. The review would then follow in May in a manner similar to that of ESO. There was some confusion regarding what the summit pipeline actually did. This has now been sorted out. However, it does raise the point that we should be very clear when explaining what the summit pipeline does. The PPARC WFCAM press release was noted. There was a suggestion that we do a similar picture by the end of February (or sooner). STH was very keen and volunteered to do this. <<<< MJI said that we will send the data up to Edinburgh when we have all the data and have sorted out the header problems. The quarterly job list still has not been made. MJI and STH will draw up a list and circulate it. <<<< 4. WFCAM commissioning update JRL reported on his two week trip to Hawai'i in November to help with the WFCAM commissioning. Although there was bad weather for most of the nights, there were a few good nights and lots of data was obtained. Even during periods of bad weather, many calibration data files could be generated. At the moment there are many features in the data that are not understood. We have two months to sort them out before WFCAM goes back on the telescope. Investigations are only being carried out on data from the 8th November onwards after which the electronics were reasonably stable. The data is on 16 LTO tapes containing a total of about 2 Tbytes. The data is being placed on disc and can be found at /data/apm6_b/wfcam/MEFs/ and is further sorted into directories for each night. You have to log onto cass36 if you wish to access this data. All the data will be there by the middle of next week. There followed some discussion regarding various web pages: STH should contact Paul Hirst (& the Edinburgh web-master?) in order to change the CASU link to go to our VDFS web page rather than our group home page. <<<< DWE should add a WFCAM logo and link to the bottom of our VDFS web pages. <<<< It was also suggested that these pages be revamped (I'm not sure what exactly was wanted here. What needs changing?). <<<< MJI needs to update the CASU group web pages. <<<< 5. Status of ESO documents for FDR All bar one of the required documents have been submitted. The Data Reduction Library Design (DRLD) document version 1.0 was completed and had the most effort spent on it over the last 3 months. This document was submitted for FDR on the 17th December. The other FDR documents (URD, DRS and CAL) have been updated with all the required responses to PDR RIXs and updated as far as possible to incorporate the changes in template design and other design changes necessitated by the switch from deliverable specification 1618 issue 1.0 to 1618 issue 2.0. The other work package that has taken rather more effort than planned this quarter is the ETC document. Although an interim ETC document version 0.9 was submitted on December 23rd, Version 1.0 of this, required for FDR, is still awaiting final discussion and iteration. Many of the problems with the design of the ETC are related to the overlap of the ETC requirements with the design and requirements of the Survey Definition Tool. In addition a live working ETC based on a perl cgi script was developed, accessible via a web form interface (http://www.ast.cam.ac.uk/vdfs/etc.html). 6. Status of Data Centre Archives RWA was unable to attend the meeting and submitted the following report: ING archive We now have all the DVDs from La Palma up to and including early September 2004. There are 1950 for the INT, 450 for the WHT and the 76 from the JKT which stopped coming in 2003. All have been checked. The dumping of the INT CDs awaits the allocation of a suitably large disk space. With 7300 CDs each of 650 Mbyte, a fill factor of 0.9 and a Rice compression factor of say 2.5 then we'll need 1.7 TByte. The cost of disk space will be compensated for by the much quicker response to requests - jobs that used to take hours to fill will take a few minutes. Out of 1494 CDs loaded in 2004, 1196 (80%) were from the INT. There were 527 requests for 47460 files in 2004 compared with 526 and 41996 in 2003. These are from the `official' archive. Requests in 2004 for data later than the last available archive epochs were received from 15 astronomers for 5964 files i.e. about 12% of the `official' total. AAT archive Another 56 DVDs have arrived, bringing the AAT archive up to 2003 Sept 2. The first batch of 152 DVDs have been loaded into the DVD tower and the latest batch will follow shortly. 7. INT WFS products release MJI said that he had received an email from Danny Lennon regarding making the WFS data publically available. (I'm not 100% sure this is correct) The main task is to calibrate the large areas involved. The individual data frames are already available. It is hoped that this task will be completed by the issue of the spring INT newsletter. EGS was unable to attend the meeting and submitted the following report: A quick update on progress. I have almost finished the calibration of all the WFS frames in the ELAIS regions taking as reference the calibrated released ones. I need ~1 week to check the results, see why some files do not have header keywords (e.g. airmass) and write a small report so you can have a look before applying the corrected zero points and zero errors. 8. RAID arrays update MJI reported on the various RAID arrays that we have available. Recently we have obtained 17 x 300 Gbyte IDE discs which have been distributed to the RAID arrays (with some as spares). This means that there are a number of 80 and 160 Gbyte IDE discs spare. (I think you were offering them to a good home, but not too sure.) There is a lot of data sitting on various machines and the intention is to migrate this all to the RAID array on apm3. The details are as follows: /data/apm2_a 2 Tbyte general use /data/apm3_a 2 Tbyte general use /data/cass36n 1 Tbyte optical processed data mainly INT /data/cass36o 2 Tbyte optical processed data mainly INT /data/apm6_b 2 Tbyte wfcam raw MEF data /data/apm6_a 1 Tbyte wfcam ingest buffer /data/apm7_a 1 Tbyte wfcam products /data/apm8_a 1 Tbyte data processing buffer /data/apm9_a 1 Tbyte data processing buffer /data/apm10_a 1 Tbyte data processing buffer /data/apm11_a 1 Tbyte data processing buffer /data/cass38g 1 Tbyte sybase/ING data store /data/apm12_a 1 Tbyte sybase/ING data store /data/cass36a-m INT WFC processed data /data/cass03a-k INT WFC processed data /data/cass38a-k sybase store and ING archive buffers STH was volunteered to make an internal web page to make sense of all of this. <<<< 9. APM building news The last two building liaison meetings were attended by PSB. He said that the timescale for the roof construction had slipped a little, but by now this has been completed. He pointed out that the main network cables for the IoA will be placed into proper cable runs rather that in a tangle under the floor of the APM building. The currently projected completion date is now mid-February. STH officially conceded the bet, blaming his youth for his optimistic folly. PSB suggested that the payment (lunch) be bought after completion. 10. AOB MJI reminded everyone to fill in their .away forms to note their expected absences. This is a very useful tool for planning meetings. <<<< MJI said that the group has given a presentation at the staff meeting and he has had some feedback. MJI asked about the data on cass38's discs. JRL said that he hadn't done anything with them yet. He said that RWA uses them as a staging area so would like to keep them. PSB suggested that cass36 and cass38 need replacing. It was also noted that the DVD tower would be decommissioned once it is out of maintenance. PSB also said that Acrobat 7.0 was out and that we get a licence discount on 10 or more (£36 each). Since the secretaries are getting a batch, we should join in for one or two. PSB will organize this. <<<< RGM said that he had access to a free PDF writer. He said that he would email any interested parties. <<<< (It is called pdf995 and is Windows based only.) Continuing Actions ------------------ MJI continue with comparing FIRES project reduced data by testing a CASU seeing-weighted stacking procedure JRL finish solving ZPN WCS problem in IRAF New Actions ----------- RGM contact EGS regarding missing items in the revamped INT WFC web pages MJI produce and distribute bound copies of the grant bid DWE show MR around our web site STH update and submit ETC document and perl cgi script MJI PSB send the data dictionary files to JMS so that he can submit them to ESO by Monday STH generate a pretty picture from the WFCAM data so that we can use it in a press release (by the end of February or sooner) MJI draw up a quarterly job list and circulate it for comments STH STH contact Paul Hirst (& the Edinburgh web-master?) so that the CASU link goes to our VDFS web page rather than our group home page DWE add a WFCAM logo and link to the bottom of our VDFS web pages DWE alter the VDFS web pages ???? What needs changing? MJI update the CASU group web pages STH make an internal web page describing our disc resources ALL fill in .away forms to note expected absences PSB sort out Acrobat 7.0 purchase RGM email interested parties about free PDF writer