6th April 2006 Hoyle committee room 10.30 - 12:30 Present: DWE, MJI, PSB, STH, MR, RGM, WJS Apologies: NAW, JRL, EGS Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. VDMT Q1 reporting and WPs 4. Data archives update 5. Optical processing 6. WFCAM processing 7. VDFS VISTA update 8. AstroGrid liaison update 9. AOB Minutes ------- 1. Actions from last meeting JRL has sorted out the access problems with the OMP database mirror, however there is still an outstanding issue regarding the MSB quality flag and propogataing this information to the run number level. JRL is thinking about how to implement it since it apparently non-trivial - ongoing <<<< MJI has made no further progress investigating the merging of classification statistics - ongoing <<<< JRL has finished modifying the AAT archive interface including the appropriate 2 year proprietary period MJI has not contacted JPE regarding the CASU preferred method of data transfer from ESO, but will raise it at the VDMT meeting - ongoing <<<< MJI circulated the ASCII version of the 06Q1 work packages ALL provided comments regarding the 06Q1 work packages (see item 3) WJS has not been in touch with JPE regarding acquiring the numeric (rather than graphic) versions of the data tables required by the ETC. PSB listed what data was missing. WJS said that some of this may be in the hands of commercial companies and it may not be possible to obtain, but that he would see if he could get hold of the data. If this is not possible then the best we can do is deliver digitized versions of the graphics - ongoing <<<< It was noted that the reason that the Z filter response was not available was that the Z filter hasn't been ordered yet! RGM is working on quantifying the foul-up causes in LAS outliers list. This will be finished by the end of April - ongoing <<<< STH has not found out what happened at the AstroGrid Implementation meeting. RGM suggested that he email to find out what was happening regarding the minutes. DWE suggested that he talk to FvL - ongoing <<<< PSB has set up MRTG on the network switch to monitor network traffic ALL not all reports were in yet covering Q1-06. This is needed by the end of the week to provide a report in time for the VDMT meeting on Tuesday (11 April) - ongoing <<<< JRL have had a meeting with RWA and made progress with the ING archive EGS issues DWE has got the persistence report ready for release http://www.ast.cam.ac.uk/~wfcam/docs/reports/persistence/ This is also announced in the dairy. DWE has written a web page containing a few examples of persistence. This is linked at the bottom of the persistence report http://www.ast.cam.ac.uk/~wfcam/docs/reports/persistence/ \ persistence_examples.html It should also be added to the known features Plone page. RGM suggested that a link to the relevant 2MASS page should be added to the report <<<< JRL and MJI have investigated the deblending problems identified from the LAS outliers list. The development software had no problem with this data, but sadly the production pipeline version had a low-level bug that caused very occasional problems. This bug was fixed prior to commencing 05A reprocessing. Although multiple comparisons are routinely carried out between the development toolkit software and the pipeline version, this bug had slipped through due to its unusual nature. Weird science outliers are, of course, a good way of finding pathological problems. MJI emailed Nigel's list of bad sky frames to JRL, again! JRL implemented the bad frame list in the sky determination procedure JRL fixes have been carried out for most of the reported header problems MJI for the raw 05A data. A small number linked to the jitter sequence keywords still need fixing - ongoing <<<< MR checked the reprocessed data from 8/9 April and identified a few MJI issues. The reprocessed data is definitely better but more checks are needed - ongoing <<<< DWE wrote a script to check for newly added files to the internal file system. New files are checked to see they are in a sensible location and then copied over to the Plone system. DWE reorganized the internal file system on the same lines as the Plone internal system EGS is not giving a talk at NAM (nor presenting a poster) MJI distributed the details of the Croatia workshop. DWE said that he had signed up for it. 2. Comments on WFAU minutes Surprise was expressed that WFAU appear to be going ahead with the UKLight option already. This has resource implications, hardware and recurrent costs, and staff effort for CASU. We felt that it was still not clear that this will solve the bandwidth limitation problem. At this stage we still need to ascertain where the bottleneck in network transfer speed is and explore other aspects (eg. cost) of the UKLight option. This issue will be raised at the VDMT meeting. 3. VDMT Q1 reporting and WPs MJI said that he has received comments regarding the Q1/Q2 plan. He will send out the updated plan to assist with the writing of the monthly reports. <<<< Summaries for the VDMT are being written eg PSB on the ETC and network issues. These all have to be finished by the end of the week. STH and MJI discussed the predicted VISTA transfer rate and came to an agreement based on the expected mean daily data rates for VISTA. MJI is considering the possibility of a script to push the processed WFCAM data to WFAU rather than the current system. The data server having control of the process is proving extremely effective and trouble-free for transfers to the ESO data archive and from transfers from ING to the archive in Cambridge. Both are much lower end-to-end bandwidth network systems, yet have no problem maintaining an acceptable data transfer rate using simple transfer protocols. This possibility will be raised at the VDMT meeting. RGM raised the issue of public release of VDMT products. He was of the opinion that the pipeline products (images and catalogue products) should be released with next to no delay once they had been pipeline processed and released by CASU to the WSA. He recalled a requirement from JAC for this. This will also be raised at the VDMT. MJI summarised the issues that will be raised on Tuesday and asked if there were any others. The alternative options for dealing with persistence were discussed and it was felt that this also should be an item for discussion at the VDMT. All agreed that Tuesday would be a fun meeting. RGM will decide if he is going to participate via telecon for some of the items. <<<< 4. Data archives update AAT: has been upgraded (see earlier) ING: the interface is being upgraded (see earlier) WFCAM: no new data UKIRT: Paul Hirst has reported that some files have got missing header information JRL will look at this when he returns <<<< 5. Optical processing VST survey management plans: MJI reported that the VPHAS+ VST survey management plan has been submitted and that he forwarded a copy of this to Tom Shanks to help him draft the equivalent plan for the VST ATLAS survey. It was noted that the committee for deciding the VISTA Public Surveys will meet on 19 June after the initial pruning has been carried out in May. The number of submitted surveys was 15 - the Dunlop UDS survey was the one missing from our list. MJI said he would get a copy. <<<< 6. WFCAM processing MR is currently leading the reprocessing 05A. The processed data will be dumped in the same place as the earlier version due to pressure on disk space. The previous version catalogues have been saved for direct science comparison. The new sky processing strategy creates a lot (~500/night) of intermediate step sky frames (xskys). Although these are currently being saved for diagnostics purposes they should not be required for long term archiving. MJI will double check this at the VDMT meeting. STH reported on the ZY photometric calibrations. Comparisons have been carried out with respect to Sloan for one globular cluster field (M3) and a much larger scatter than expected has been found, although the zero-point was ok. A WFCAM (night-to-night) comparison for M3 also showed a large scatter. MJI wondered if this was simply due to crowding and suggested inter-comparing other WFCAM Z-band data. This investigation is ongoing and a summary report will be given to the photometry working group by the end of the week. The persistence problem was discussed. MJI said that we had a number of options: ignore it; correct each frame (probably not viable or worth doing); or flag suspect catalogue entries using 2MASS as the basis for the predictor. The latter is the favoured option (if persistence is shown to cause science problems) but would involve a fair bit of coding. The main advantage is that it can be added on post-processing as a flag bit in the catalogues. However, rather than rushing into this it was felt that this should be science requirement-driven ie. has persistence been identified as a problem in doing the science? MJI noted that the main wfcam data discs were full. By the time we start processing 06A we will need more disc storage. New options are being considered - either 7TB or 11TB RAID6 (2 parity discs) systems that hang off our existing disc servers. This will cost about 1k per TB (excl. VAT) and about 50TB will be needed over the next year (not just for WFCAM). STH mentioned a request from Sandy Leggett regarding linearity. JRL had analysed series of dome flats to characterize this. MJI said that he would contact JRL and compile a response. <<<< RGM raised the issue of stacking problems. MJI said there were two main aspects to this arising when doing deep stacking. The usual CASU pipeline output uses 32-bit integer FITS and given enough files in a stack quantisation noise can become an issue. The external (to the pipeline) stacking software has an option to produce Real*4 output if required. This version has been available for over a year. The other aspect was caused by a software limit in the supplied pipeline catalogue software. A simple workaround for this was suggested at the time and a more robust solution is under development. 7. VDFS VISTA update WJS reported that the telescope has passed the acceptance test, including (nearly all) the mirror supports. This will now be shipped from Houston over Easter before the Hurricane season. WJS then reported on the schedule for delivery of the mirrors and coating plant. The camera is being tested as we speak. PSB said that we have not received any AIT data yet [Gavin must have overheard him because some appeared the next day]. It is expected that the full all-up testing for the camera will take place in May. PSB stressed that we want all the data from each exposure, not cutouts around the exposed spots etc.. WJS said that he would pass this requirement on. <<<< WJS also added that the optical test camera has arrived in Edinburgh. RGM said that ESO now has an optical sky monitor at Paranal but was unsure how accessible the data from this was. PSB said that the number of pending VDFS documents has gone up by one. The newly found document is called "VDFS Requirements for VISTA Data Information" and is a QMU-led document. Nothing else has happened on the documentation front. The DRL has got bigger still and is making steady progress. The use of the ETC has quietened down now that the survey deadline has passed. ESO has started work on the production version of the ETC. This means that it is becoming more important to supply them with the correct calibration tables. MJI reported that the FITS extension reordering request (an SPR - Software Problem Report) had gone in and will be implemented before commissioning, but not before AITing. PSB will write a script to order any AIT FITS files before letting them loose here. <<<< 8. AstroGrid liaison update Nothing to report 9. AOB None Continuing Actions ------------------ JRL sort out how to transfer MSB quality information to processed files MJI finish investigating merged classification statistics MJI raise issue of CASU preferred method of data transfer from ESO at VDMT meeting WJS contact JPE regarding acquiring the numeric (rather than graphic) versions of the data that is required by the ETC RGM finish quantifying foul-up causes in LAS outliers list (by end April) STH find out what happened at the AstroGrid Implementation meeting ALL send reports to STH covering the entire quarter (by Friday 7th) JRL check and if possible fix any remaining FU'd headers (JITTER keywords) MR check reprocessed data from 05A for quality control MJI New Actions ----------- DWE place link in persistence report to the relevant 2MASS persistence web page MJI distribute updated Q1/Q2 plan RGM decide if telecon presence at VDMT meeting wanted JRL investigate and fix problem with UKIRT archive MJI acquire copy of Dunlop UDS proposal MJI send email to Sandy Leggett regarding linearity WJS pass on information to RAL that CASU want whole data frames not just cutouts PSB write a script to order the extensions sensibly for AIT FITS files