8th March 2006 Hoyle committee room 10.30 - 13:35 Present: DWE, MJI, PSB, STH, MR, JRL, WJS, (RGM after 12:15) Apologies: NAW, EGS Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. VDFS VISTA update 4. WFCAM processing 5. Prototype Plone website 6. Data archives update 7. Optical processing update 8. AstroGrid liaison update 9. AOB Minutes ------- 1. Actions from last meeting JRL has not provided an MSB quality script which MJI will use to update the FITS header during pipeline operations. This is waiting on the OMP database mirror being fixed - ongoing <<<< MJI has carried out further investigations into the merged classification statistics - ongoing <<<< WJS AIT3a has been and gone. A visit is likely for the next set of tests AIT3b (see item 3) MR has made the science plots of IC4665 and M17 and the LAS tile visible from the Plone reduction status web pages PSB have discussed a suitable strategy for jittering VISTA twilight flats JRL a summary of this has been emailed to SMB MJI JRL has not finished the mods to AAT archive but will by the end of month <<<< JRL has asked if there is any more UKIRT (non-WFCAM) data that needs ingesting (but has not received a reply) STH has not raised various VDFS issues at the VDMT meeting since there hasn't been one since this year. He will initiate discussions. <<<< DWE has carried out further investigations into persistence (see item 4) MJI forwarded the EDR paper CASU coauthor list to Simon Dye JRL processed the publicity frames for the BBC, but found that that the data quality was very poor and that no sensible reduction was possible MJI has not had any feedback from the UKIRT board meeting so everything must be fine ALL sent in reports to STH, however there hasn't been a VDMT meeting since then. The lack of VDMT meeting and format of our reports were discussed. MJI said that he and STH needed to revisit the work packages for 06Q1 and reassess progress <<<< RGM investigated the CASU presence at NAM 2006. EGS submitted an abstract MJI emailed Andy Lawrence (CC Andy Adamson) asking for clarification regarding CASU direct access data rights on the processed data, but has not received a reply. 2. Comments on WFAU minutes MJI noted that there had been some activity regarding UKLight. PSB reported that some UKLight hardware has been installed by the University Computing Service. Many emails have been exchanged over the last week and much of it has been centred on the issue of transfer speed. PSB pointed out that this is not the central issue for CASU. The current worry is over the charged costs of data transfer roughly, 1k per TByte. Thus the benefit of UKLight should be judged largely by cost and not data transfer speed. PSB said that Andy Batey has looked at what proportion of the IoA's network traffic is caused by CASU. The split seems to be 80:20 between CASU and SEGUE. Note that the CASU part includes ESO as well as WFAU transfers. STH noted that there had been a comment regarding the lack of dynamic range within deep stacked images and catalogue generation due to the way data was handled internally. MJI said that he had replied to this query (while he was away on holiday) suggesting simple interim fixes. 3. VDFS VISTA update - Telescope status: WJS reported on the current status of the telescope and mirror schedule. - VIRCAM Status - AIT: The next set of camera tests to occur will be AIT3b, which will occur after AIT4b! AIT3b will be in ~2 weeks time and a CASU visit during this to carry out and acquire some test data is being negotiated (PSB and WJS to sort out details). Real VIRCAM test data will be useful for creating a test harness for later ESO deliverables relating to the summit and Garching pipelines. <<<< PSB sent a request to RAL that the TCS simulator be available and that generally the current software be installed to ensure up-to-date data format; there is some difficulty with the use of a very old version of the DCS subsystem to control IRACE to work around the random FITS ordering problem. PSB noted that flat-fields would be one of the things required. These do not have to be perfect so an even illumination is not critical. MJI asked PSB and WJS to make a list of the desired data and forward to RAL. <<<< - VDFS docs status: PSB reported on the status of these documents which is summarised below. Documents completing FDR: RIX: v1.0 2005-05-05 ?? DFS Impact: v1.2 2005-05-09 awaiting release Calibration Plan: v1.2 2005-08-12 awaiting release DRL Design: v1.2 2005-08-12 awaiting signing ETC: v1.1 2005-09-01 released to ESO? SADT: ?? ?? Documents complementing DRL release v0.1: DFS Impact: v1.3 2005-12-16 awaiting approval Calibration Plan: v1.3 2005-12-19 awaiting approval DRL Design: v1.3 2005-12-22 released Documents for UK review: UK URD v3.0 needs updating?, signing - DRL status: JRL said that excellent progress was being made on the DRL plus documentation and noted that a few more recipes would be needed before a 0.5 version is ready. JRL,PSB and MJI will have a telecon with ESO tomorrow to discuss the status of the DRL and schedule for delivering incremental releases. - Workstation for summit pipeline: There was some discussion regarding where this would come from. The equipment would first go to ESO Garching for software installation, before being shipped to the summit. Both PSB and JRL stressed the advantage of being able to test the pipeline within the ESO software environment before delivering the VDFS summit and Garching DRL modules to ESO. - Commissioning: It is unclear what requirements there are on CASU presence and use of CASU VDFS pipeline software during commissioning. This needs to be discussed at the next VDMT meeting. <<<< - ETC status: PSB reported that the most recent issues were regarding overheads. There has also been some recent discussion as to whether the calibration files should be made available to users. MJI's opinion was that this could be done most easily via the help pages but PSB pointed out that this would still involve the production of more user-friendly calibration files. <<<< It was noted that there had been 2804 hits on the ETC (presumably by people designing surveys). There followed some discussion regarding whether slew and dump-to-disc time should be included in the calculations - the general feeling was that they should be, but there was still some uncertainty regarding parallelising the latter with moving to the next position on sky. PSB noted that he had removed the slew overhead from the reported overhead time on request from JPE. PSB should check that the ETC uses the updated numbers that Will suggested.<<<< - News on VISTA survey proposal front: WJS said that he knew of 9 surveys that had been proposed. STH went to the twiki page http://wiki.astrogrid.org/bin/view/VISTA (there were 11 there). These were discussed along with what CASU's contribution might be to these. MJI suggested that a table summarizing these surveys would be useful. We were all asked to read the proposals to see if there were any we would like to be involved with over and above the standard processing. <<<< MJI volunteered to update JPE's data management plan template suggesting that for standard VDFS processing tasks and data quality control, we should have a single CASU entry. More project-specific processing tasks would depend on CASU individuals wanting to be directly involved in the science and have true Co-I status. In this case the relevant individual's names should be used. <<<< Higher level deliverables within these surveys need to be clearly flagged and the resources to carry out any additional work need to be identified. CASU have been asked about our preferred method of data transfer between ESO and Cambridge. MJI said that, given Internet transfer is not an option with ESO, then LTO tape would be preferable to discs. MJI said that he would contact JPE regarding this. <<<< 4. WFCAM processing - photometric calibration status STH reported on a single Cohen spectrophotometric calibrator star (G type) test that had been carried out. In JHK the photometry was off by 2-4% and in Y and Z it was different by 10-15%. Internally the scatter between different WFCAM measurements of the same stars was at the level of 1-2%. It is unclear whether this reflects a real problem or single star shot-noise. STH noted that further investigation of the Z-band calibation were planned using SDSS comparison and both Y and Z using updated Z and Y pseudo-standards from Sandy Leggett. - persistence investigation DWE described his current work on persistence. A practical robust scheme to correct the image data does not seem possible. The error on the correction would be of order the size of the correction. The source of the scatter may be systematic in some way rather than random eg. dependent on spatial location, but is proving difficult to measure and hence quantify. The impact on the UKIDSS surveys will probably be small because most frames are interleaved or are part of deeper stacks. Due to moving the frame between each interleave exposure, persistence images get even further smeared out and they are thus not detected as a source by the cataloguing software. For non-interleaved data, eg. LAS, it is estimated that around 0.1% of catalogued images may be caused by persistence. Flagging options were discussed as an alternative. This could be via the confidence maps (a lot of extra computation) or more practically via the catalogues, using 2MASS stars and a simple persistence model as a predictor of possibly suspicious images. There is still a need for assessing how serious a problem it is and RGM said that he will use a list of 1000 known spurious outliers from 20 sq deg of LAS data to assess this. These have already been inspected visually but will now be characterised into different causes (including cross-talk and persistence) to assess the need for possible remedial action. <<<< - progress on PSF fitting DWE said that no progress has been made on PSF fitting since the last meeting, due to investigating persistence taking precedence. - UKIDSS EDR Sci and CASU papers It was noted that we were still waiting to see a new version of the science paper. However, a new version of the EDR paper had been received and all named CASU authors were encouraged to read and comment on it. <<<< 5. Prototype Plone website Due to time considerations and the lack of EGS, the demonstration of the new Plone web site was postploned. Meanwhile, we were all asked to look at the new site and send in comments to DWE and EGS. <<<< 6. Data archives update - AAT, ING, WFCAM, UKIRT: All archives are up-to-date. ING is now all on-line bar a gap of 6 months between the end of DVD ingest and start of internet transfers. This is planned to be filled by internet transfers. MR pointed out that several nights of the last of the WFCAM data did not have any data from chip 3 due to a problem with the controller. JRL needs to upgrade the pipeline to cope with this. This has caused a slight delay in finishing off the last week or so of 05B data processing. <<<< 7. Optical processing update - VST, IPHAS MJI noted that the VST public survey updated management plan deadline is at the end of the month and that some input is required from him. <<<< 8. AstroGrid liaison update STH said that there had been an AstroGrid Implementation meeting in Liverpool on 1 March. Only NAW and FvL from CASU were able to attend. STH said that he would find out what happened and report at the next meeting. <<<< 9. AOB EGS and MR are preparing a data reduction paper from the various reports and papers that have already been written. MJI will review progress after the VISTA PS deadline passes. STH is writing a photometry paper. Continuing Actions ------------------ JRL once the OMP database mirror has been fixed, provide a script to add JAC MSB quality flag to the FITS headers MJI finish investigating merged classification statistics JRL finish mods to AAT archive interface by end of month STH liaise with rest of VDMT team to organise next meeting and make sure assorted issues are on the agenda New Actions ----------- STH reassess the 06Q1 work packages and progress in advance of VDMT meeting MJI WJS sort out CASU visit to RAL during AIT3b PSB PSB make a list of desired data from the AIT3b visit and send this to RAL WJS MJI raise issue of requirements on CASU during VISTA commissioning at next VDMT MJI include pointers to tables within the help pages PSB PSB check that the ETC uses updated overhead numbers ALL read VISTA survey proposals to check for possible science involvement MJI update JPE's data management plan template MJI contact JPE regarding CASU preferred method of data transfer from ESO RGM quantify foul-up causes in LAS outliers list ALL read and comment on new version of the EDR paper ALL look at the new Plone site and send comments to DWE and EGS JRL upgrade WFCAM pipeline so that it will cope with completely missing chips MJI provide input for VST PS management plans STH find out what happened at the AstroGrid Implementation meeting