12th July 2005 Hoyle committee room 10.00 - 12:55 Present: MJI, RGM, PSB, DWE, STH, MR, JRL Apologies: WJS, EGS Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Report on VST meeting 4. VDMT report - Q3 plans 5. Data archives update - UKIRT, WFCAM, AAO, ING 6. Optical processing update 7. WFCAM JAC telecons and processing update 8. VDFS update 9. AstroGrid liaison update 10. AOB Minutes ------- 1. Actions from last meeting JRL has still to include access to "observing" log index files in the raw WFCAM archive. There was some discussion if this was required since they are now available in the processing status web page but the general opinion was that it was worth having in more than one place - ongoing <<<< MJI are still in the process of writing a new report on the detector JRL properties as they currently are and the processing strategy, MR ongoing and needed in time for the SV meeting <<<< MJI collated the comments on the WSA Photometric Recalibration Design EGS properties and passed them on (this was acknowledged in the WFAU minutes) ALL sent in our quarterly reports to STH JRL has encountered problems while working on the download side of the AAO archive scripts due to connectivity issues to AAO - ongoing <<<< WJS the latest SADT Definition Document may have been acquired, unfortunately it is the Nov 2004 version. There was some discussion regarding whether there was a new version available at all - ongoing <<<< DWE wrote an observing efficiency script. The general result was that all surveys were about as efficient as each other ie. there was not much difference between the deep and shallow surveys. For 5s exposures the telescope is usefully integrating on sky for about 55% of the time (for 10s it is 65% and for 20s 83%). The observations for the University of Hawaii and other non-UKIDSS observations followed a similar pattern. MJI will raise this at the JAC telecon on Friday. <<<< JRL has removed the remnants of the old UKIRT data archive from the data discs on cass36. Some scripts remain, but the data has all been removed. DWE updated the hardware web page to reflect the latest changes MJI see item 3 for the report on the VST meeting MJI raised the issue of flatfields (twilight and dome) at the WFCAM telecon. He noted that in June many more successful flats were taken. MJI asked about the WFCAM schedule at the WFCAM telecon. WFCAM will return to the telescope on 26 August and will start science observing again ~a few days later. We should therefore start receiving data again for processing ~mid-September. RGM distributed the email from Steve Warren regarding the next UKIDSS meeting. The dates of the next WFCAM related meetings were then discussed: UKIDSS Survey Heads 27th July to discuss MSB design Science Verification meeting 3-4 weeks after data release from WFAU - the pencilled in date is 18 August in Edinburgh Workshop on 'Panoramic near-Infrared Astronomy' in Edinburgh 9-10th November, followed by a UKIDSS consortium meeting 11th November The next VDUC meeting is pencilled in for the first week of September. MR took over development of the WFCAM progress DQC database (see item 5) MJI discussed date of next VDUC/UKIDSS meeting at the VDMT (see above) RGM has not yet found out more detail about the AstroGrid hardware money. However, he assured us that it was still there - ongoing <<<< RGM has not yet sent PSB's comments on the document covering the service support for VO to Tony Linde - ongoing <<<< 2. Comments on WFAU minutes The air-conditioning failure and resulting dampness was noted. Good luck with the recovery and hardware move. The transfer of data between Cambridge and Edinburgh is working well. Around 2 Tbytes of data was transfered over one weekend with an average transfer rate of ~10 Mbytes/s, to the apoplexy of the University network monitoring service. Noting the comment regarding transfer of data via LTO tape, MJI pointed out that this was discussed at the VDMT meeting, which recommended that offering this service was probably not a good idea and that network transfer should be the goal. 3. Report on VST meeting MJI reported on the VST meeting held at Garching on 22 & 23 June. This was a very useful and productive meeting discussing the 3 core and 4 secondary public survey proposals. Also discussed was what data products would be produced and their verification. It was noted that ESO will serve the data products of VST directly to the public and that these have to be delivered to VO compliant standards (whatever that will mean). There was some reorganization of the proposals: a) the two Galactic plane proposals - VPHAS (Drew et al.) and UVEX (Groot et al.) are going to be merged; b) KIDS (Kuijken et al.) should absorb the the remaining UKIDSS LAS/GCS programme (Warren et al.) and Dalton et al. proposal. c) z<2 census (Oliver et al.) trimmed but to be resubmitted d) ATLAS (Shanks et al.) to be resubmitted in its entirety The end result is that these 4 proposals will be resubmitted late September for consideration by the PSP who will then forward them with recommendations to the OPC for final deliberation. For a number of the above proposals the originally requested Z observations will probably be dropped with the intention of using VISTA to obtain these. The time allocation for the first 10 years is 40% GTO time (ie. to the consortia who paid for and built the camera and telescope, 10% to Chile and 50% to ESO (this probably being split ~40% to public surveys and ~10% to private proposals). Considering the proposals that have been chosen it looks like CASU will be processing around ~20% of the data from VST, which is about what we expected. It is likely that a similar model to the above will be used for VISTA and there will be a VISTA call for proposals in the autumn. First light for VST is planned for early 2006 and public surveys may start around autumn 2006. (Note that ESO Messenger#120 contains a status report on VST and a list of the GTO proposals.) 4. VDMT report - Q3 plans MJI and STH reported on the VDMT held on 8 July. The UK URD was signed off and there is a copy (without all the signatures) in the usual place. The issue regarding keywords in the VISTA FITS files was discussed and will be brought to a controlled conclusion. For the next VDUC meeting what is required for the VDFS UK URD review is an itemised list of work packages/tasks with dates and some priority ranking, similar to the development plan in the VEGA proposal. This will be submitted to the next VDMT meeting for review and discussion prior to further release. In parallel with this the VDMT felt that it was now timely to reassess the dates and timings for all the work packages and version releases in light of the delays to WFCAM commissioning since the original plan, drawn up in 2003. STH pointed out that there is a test version of the SADT available for both VISTA and VST and that he would be investigating it. <<<< For the Q3 plan it was noted that most of this would be continuing what was current in Q2. JRL said that an additional item would be the preparation for the ADASS in Spain. This is during the first week in October and the abstract deadline is 1 August. All were asked to send in proposed plans for Q3 for review. <<<< 5. Data archives update - UKIRT, WFCAM, AAO, ING UKIRT: another tape of non-WFCAM data has arrived. JRL noted that we have had 44 requests submitted and met by the online system since it was re-instated. WFCAM: raw data archive now has full authentication with individual password password and the running and maintenance of this has been handed over to MR. MR was asked to smarten up the front page. <<<< JRL pointed out that there was a (known) problem with all calibration frames in the period 5th to 23rd May whereby the FITS PHU contains the wrong jitter and microstep information (in fact inherited from the previous frame. This has also affected some science frames - those taken without interleaving. JRL was asked if the summit comments (OMP) database was easily accessible. MJI reminded him that he had produced a simple perl script for local use that he had made available to Simon Dye via the web. He will put links on our web pages to this. <<<< AAO: at the moment there is a connectivity problem with the AAO DB which is slowing up progress but it is being sorted out. ING: RWA and MJI are carrying out Internet transfer tests with Don Carlos Abrams and Luis Hernandez at LPO to find an alternative to transferring data to Cambridge via DVD, which has been found to be extremely unreliable. These tests showed that reliable rates of ~3 Mbits/s are possible - no prizes for speed but gets here in the end. This is sufficiently promising in that network transfers will currently just about keep up with the average data taking rate (and the bottleneck may lessen in the future). The need for a local backup of this data was raised and will probably be on LTO. RGM said that WFCAM raw data can now be obtained direct from the ESO archive via one of two routes (he has tested this). MJI pointed out that it is now ESO policy to make public survey raw data world accessible immediately it is available in the ESO archive (WFCAM is a special case). WFCAM processed DQC archive: MR noted that this was not a simple matter of cloning the INT WFC DQC (for various reasons). MR said that there was a need to make it more homogeneous since this will be used for VST and VISTA as well. MR said that he would have a chat with JMI in order to progress this. It was noted that some of the design features from Aladin would be useful and using this could take some of the strain out of maintenance. This will be investigated further after MR has finished the current processing. <<<< 6. Optical processing update EGS has carried out some processing of XMM INT WFC data following a request from RGM. MJI and Nicolas Martin are processing several hundred frames of MegaCam data. 7. WFCAM JAC telecons and processing update Two telecons have been held since the last meeting (24 June and 6 July). The four main issues were speed of the summit pipeline, problems with the FITS headers, twilight flats and observing strategy. Speed: JRL has speeded up the summit pipeline and carried out tests to see how much of a logjam builds up. Using a night of data taken in June which previously gave problems, he noted that using a simple script to mimic the incoming data rate, the delay is now on average 38s with a maximum of 117s. This compares favourably with the up to 1+ hour delay previously incurred. A few bits of fine tuning are still in hand to improve the speed even further. STH asked if these tests were carried out with Galactic plane data, since this is another survey where the data processing has to work hard. JRL said he would investigate alternative nights <<<< One of the problems for the summit pipeline is that JAC also want to use it to generate catalogues for each science frame (e-star project for transient alert science). Fulfilling real-time DQC and this simultaneously is tricky. There was some discussion about hardware infrastructure at the summit and how difficult it may be to test possibly network-dependent aspects outside of a real observing night. Twilight flats & FITS header problem: these are carried out at dawn and have tended to either be overexposed or have a dither problem. However, although this has limited the number of independent flats obtained it appears to have been sorted, and there are 7 nights in June with reasonable flats. MR will continue his investigation of these. <<<< Observing strategy: currently only one strategy is being employed (shock and awe - take as many independent frames as possible) making it difficult to assess alternatives (eg. coaveraging with fewer dithers). MJI will attempt some tests on the existing data to mimic this latter strategy and cf. science achieved. MR has made a web page displaying the progress of the WFCAM processing (http://apm15.ast.cam.ac.uk/wfcam/). This is currently linked off the diary page and provides links to the DQC summary info plots and index files. DWE was tasked to link off the main web page and also include a link to the rwa WFCAM archive. <<<< MR was asked to put some brief explanatory text in the web page on the various information available (with MJI's help if needed) <<<< 8. VDFS update PSB said that good progress has been made recently. The issue of what goes into the FITS headers appears to have been resolved at the last VDMT meeting eg. status of the flat fielding lamps. We have also had feedback from ESO regarding the VDFS FDR documents. Michele Peron has informed PSB that the re-submitted FDR documents are now ok. The remaining FDR documents are the SADT and the ETC. There are a few minor changes to be made to the textual part of the ETC and the C modules need inserting in place of the perl modules in the appendix. STH will complete these changes and the updated ETC will be sent off. <<<< MJI asked PSB to forward the email from Michele Peron to JPE and JMS. <<<< PSB said that he would check the status of the various VISTA documents to make sure that the version that we have is the final version containing all the signatures. It was thought that this could be done at the same time as sorting out the ETC document. <<<< A number of relevant issues have been decided at a recent DICB meeting. It was agreed that the number of jitters can additionally go into the primary header using a non-ESO hierarchical keyword. Sometime in future the astrometric reference system used will include the ICRS, for now it remains FK5. PSB raised the issue of whether VISTA calibration frames (domeflat,dark,...) need to have WCS keywords in their headers. The general opinion was that their absence (or presence) would not cause the pipeline software to barff. It was thought that further detailed checking of SMB's template definitions would be a good idea and that WJS would be the best person to check that these are sensible. <<<< PSB said that it was still unclear whether calibration frames would be classified as science or calibration. This issue needs resolving by project management. MJI will raise issue at next VDMT. <<<< We were reminded that some DRL software will be required to be delivered by the end of the year. However, this is dependent on realistic simulated data being available in good time. PSB noted that if the RIX responses are now agreed and the Board disposition disposed, signing off and releasing the RR document would be good news. However, there still may be outstanding issues relating to the SADT. He will contact JPE about its status. <<<< 9. AstroGrid liaison update It was reported that Astrogrid 1.1 is still about to come out. RGM said that he had had discussions with other Astrogrid sites and all have said that their installations have had to be carried out by Astrogrid staff. 10. AOB None Continuing Actions ------------------ JRL include access to "observing" log index files in raw WFCAM archive MJI finish off report on the detector properties as they currently are JRL include section on processing strategy and illustrate with MR processing examples JRL finish off download side of AAO archive scripts WJS find out if there is a new version of the SADT Definition Document RGM find out more detail about the AstroGrid hardware money RGM send PSB's comments on the document covering the service support for VO to Tony Linde New Actions ----------- MJI raise issue of observing efficiency at JAC telecon on Friday ALL send in our proposed plans for Q3 MR improve look of front page of WFCAM raw data archive JRL inform where scripts are for access to the summit comments data base MR contact CDS regarding Aladin for CASU WFCAM processed DQC archive MR investigate properties of all flats taken during 05A DWE link processing progress web page off main VDMT web page (also a link to the raw data archive) MR put some explanatory text in the processing progress web page STH finish the 3 remaining RIXs in the ETC document MJI raise calibration frame issue at next VDMT STH test test version of the SADT available for both VISTA and VST PSB forward the Michele Peron email to JPE and JMS PSB check the status of the various VISTA documents and ensure they are up-to-date WJS check that VISTA template definitions are sensible