17th November 2005 Hoyle committee room 10.00 - 13:00 Present: DWE, MJI, PSB, STH, MR, JRL, RGM, WJS Apologies: NAW, EGS Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Report on VDMT and UKIDSS meetings - data release policy discussions (RGM) 4. Status of WFCAM processing - photom calibration (STH) - recent changes to pipeline (JRL) - progress on PSF fitting (DWE) 5. VDFS VISTA update - DRL v0.1, test data 6. Data archives update 7. Optical processing update 8. AstroGrid liaison update 9. AOB. Minutes ------- 1. Actions from last meeting WJS has not yet passed on the SADT comments to Martin Folger - ongoing <<<< JRL has not finished off the OMP database access scripts - ongoing <<<< MJI raised the issue of adding an MSB quality flag to the FITS header. JRL said that a 1-7 flag is set by JAC and can be added post-pipeline <<<< MJI suggested combining with the script that adds the WSA keywords MJI has still not finalised feedback comments to MR on flatfield and sky properties draft report - ongoing <<<< MJI has given more URD feedback to WFAU after the Edinburgh VDMT meeting MJI continues to investigate the merged classification statistics - ongoing <<<< MJI has investigated the blended object complaints MJI contacted Steve Warren to see who needed to attend the UKIDSS meeting. MJI, STH and RGM attended (see item 3). WJS said that visiting RAL during AIT3a is a possiblity. This should be in the second half of December. WJS said that he would check up on this <<<< MR have finished setting up the test dataset harness. This is on apm24_b MJI and consists of about half of the data from 8 April. RGM asked if there was any repeat data within this - there is as it was part of the mesostep test data MR has finished setting up the pipeline version log on the web page PSB contacted the Computer Service regarding UKLight. We deferred this item until later, but then postponed discussion - ongoing <<<< MJI have contacted Paul Aslin about possible lobbying of the IT Syndicate PSB about the Internet charging model. They will keep us posted if there is any news. MJI arranged a meeting to discuss design of DQC archive DWE reported on the progress of PSF fitting of 05A data (item 4) WJS has not provided updated measurements of the VISTA filter characteristics, needed for ETC V1.0 - ongoing <<<< ALL sent in monthly reports EGS have not yet finalised a set of options for group web pages DWE deferred until EGS gets back - ongoing <<<< 2. Comments on WFAU minutes RGM said that he had noticed that there was a draft outline of an archive paper and asked if there was a need for an equivalent pipeline one. MJI said that he was accumulating the guts of a pipeline paper since this was an agreed part of the UKIDSS set of introductory papers. 3. Report on VDMT and UKIDSS meetings MJI said that at the UKIDSS meeting presentations were given describing the current pipeline, archive and the SV1 results. Each of the survey heads then reported on the science assessments being carried out. RGM's impression was that the pipeline is well regarded since papers are being prepared already. He also pointed out that the results from SV1 were being used for PATT proposals. No new major issues were raised, but there was much discussion at the meeting regarding cross-talk and whether some kind of flagging of suspect images can be carried out. MJI said that most of these comments referred to the old data and that no-one has yet had a chance to comment on the data from the latest version of the pipeline which attempts to remove cross-talk. MJI agreed that he would generate a diagram to explain the way cross-talk affects the data across the channels in case people wanted to produce their own algorithms for flagging potentially affected objects. <<<< RGM said that the data release policy was in disarray. There was a lot of discussion at Edinburgh, but no real consensus. The DXS and UDS groups not surprisingly want faster access to the data. Other individuals want to polish the cannon balls some more and, as part of Science Verification, only release data after it has been formally passed through the SV process. This is still being discussed by extended email exchanges. MJI reiterated that from the CASU point of view, the products are ready for release and serious science use. STH said that the issue of star/galaxy separation was also raised. MJI commented that classification is a statistical process and hence subject to the usual vagaries of CFU's. WJS asked about the general problem of resolved -v- unresolved blended objects causing classification problems. MJI noted that for roughly equal magnitude objects an equivalent Rayleigh criterion applied whereby objects separated by ~FWHM defined the boundary between resolved and unresolved and hence between between correctly and incorrectly classified. MJI reported that there were no major concerns arising from the VDMT meeting held on 8th November. RGM then raised the issue of communication between UKIDSS and CASU and was concerned that if there were problems with the data, was the information coming back to CASU ? Specifically, there was a recent UKIDSS Survey Heads telecon - was this minuted ? MJI noted that the latter was minuted and that CASU were generally represented at all of these meetings. He further commented that the agreement with UKIDSS was that any problems found with the data during the SV process were supposed to be channelled back to CASU through the CSV. Various aspect of autoguiding for WFCAM observations and the number of trailed images were also discussed. 4. Status of WFCAM processing - recent changes to pipeline JRL said that the main changes that have been carried out to the pipeline recently are in the sky subtraction process. This has now been reorganized so that it is more robust, with the skies now partitioned on exposure time. as well. A complicated mix of additive and multiplicative components has been identified and the best way to correct for this is to have the skys based on both time sequence, filter and exposure time to minimise the "scaling" required for the sky correction. MJI said that one manifestation of this problem were illumination-dependent pedestals offsets seen in stacked dark skies. The other main change is to the preprocessing scripts: the preview and preproc scripts now code share and are used to analyse and then write the processing sequence information to the FITS headers. The processing script no longer needs to repeat this analysis and takes the required information from the updated FITS headers. Currently the 05B slaves are: JRL, MJI, MR and RWA. The processing culprit is now being recorded such that any subsequent problems spotted with the data will be forwarded to the aforesaid individual. - photom calibration STH said that there is pressure to change the observing strategy for the UKIRT faint standards. It is felt that too many standard fields are being observed in too many passbands. This currently takes between 6-12 minutes every 1-2 hours. Various groups are unhappy with this strategy particularly as the 2MASS calibrations look so promising. Current investigations show that using 2MASS for the calibrations may meet the calibration specification, reducing the necessity of frequent observations of standards in all 5 broadband passbands. Ironically, the main external reference and benchmarking system, the UKIRT faint standards, may not be as error-free as previously assumed. These issues are being further investigated by the calibration working group using 14 nights of data identified as being taken in very good photometric conditions. An urgent recommendation is needed for the observing strategy for standards, preferably by the end of this month. <<<< MJI commented that at one of the Edinburgh workshop talks Roc Cutri had convincingly described the 2MASS photometry calibration and reported that because of the intense effort made for the 2MASS calibration it is highly likely to be good to better than 2% over the entire sky and hence potentially good enough to meet the calibration requirements for both WFCAM and VISTA. STH pointed out that for many nights the photometric zero point varies noticeably and on timescales shorter than the frequency at which the standards are being observed. Thus there is a clear need for 2MASS to be used for the individual frame calibrations, as is currently being done. Systematic x-y and radial terms are also under investigation using stacked 2MASS-WFCAM residuals data from many nights. Although the JHK calibration looks very promising, the Y and Z calibration is slightly more problematic, depending more sensitively on the dwarf-giant ratio and on the degree of reddening variation from field-to-field. MJI raised the issue of generating a set of secondary photometric standards centred on the UKIRT faint standards. All felt that this was still required and that these secondary standards would be useful for both WFCAM and VISTA. STH said that one of the limitations he had found was that there were not enough stars in the fields that he had used in the mesostep tests in 05A to fully characterize the systematics and that he will have to select a more crowded field in which to carry out these tests and forward the MSBs to JAC. <<<< - progress on PSF fitting DWE reported on the recent work on PSF fitting. The current model is based on the adaptive kernel method of Alard and Lupton. Due to having 36 coefficients there are robustness issues, but recent modifications may have dealt with most of these. MJI suggested that a circular aperture (rather than the current postage stamp) be tested to see if that helped with the <<<< robustness. DWE said that this would make next to no difference to the fit itself due to the weighting used but might affect the derivation of the kernel. Tests have been carried out to automatically process all the catalogue files for complete nights (8 & 9 April), taking about 1.5 hours per night to carry out the PSF measuring and 2.5 hours for the PSF fitting. These tests were carried out on apm3 a twin 2GHz processor AMD machine. Tests show that the astrometric accuracy relative to the standard pipeline output in the mid-magnitude range is improved by PSF fitting, however at the bright and faint ends there is no improvement, the bright end being limited by differential atmospheric effects and the faint end by signal:to:noise ?? For the photometry, PSF fitting shows no measurable improvement over what is produced by the standard pipeline. MJI suggested that this required further checks, by for example, comparing colour-colour diagrams of a crowded field (eg. M17). DWE said he would look into this. <<<< A limited amount of UDS test data was taken in early September and this shows that there is no significant difference in results derived from 2x2 and 3x3 interleaving in 0.6-0.7 arcsec seeing. MR reported on WFCAM data reduction progress. The survey status web pages now have date information added to various columns. 8 new tapes arrived last Friday that cover up to 5th November. These have all been ingested. Processing has been completed or is in progress up to mid-October. So far only 3 imcomplete images have been found in 05B, so the 05A missing files problem has been much reduced. All the data for 05B so far have been read from tape with no problems (as had 05A) showing that this method of data transfer (25 Tbytes and counting .......) is very reliable (cf. IDE disks shipped from ING with 0/2 successful). 5. VDFS VISTA update - DRL v0.1, test data PSB showed an image taken by the VISTA test camera. Obtaining such data was very useful since problems get identified early on. For example the negative numbers of the image, due to it being a naked reset frame, were a bit of a surprise. Properties of this test frame were discussed with Steven Beard and Naidu Bezawada and fed back to Guy Woodhouse. Work on the TCS is making rapid progress and last week PSB held useful discussions with Steven Beard, David Terret and Alastair Borrowman. WJS reported the latest VISTA news. The AIT2 tests went well. Cooling and vacuum performed well. There did seem to be a problem with the filter wheel being very noisy. WJS then gave his impression of the noise. Although very good, it is unlikely he will get very far on talent shows with this. JRL said that the DRL tests are ongoing. The main work consists of using code checking utilities such as lint and splint and PSB. There were some comments regarding problems with Doyxgen reading (or not) JRL's amazingly well documented code. JRL reminded us that the code written so far can be used in a parallel or sequential processing manner since the extensions can be processed individually or in one go. MJI noted that it is up to the people at Paranal whether they use parallel or other alternatives for the summit pipeline. PSB made several comments regarding deliverables and recipes. There was an expressed desire to get early feedback on these recipes in advance of the formal delivery date. 6. Data archives update Nothing new to report for ING, AAO, UKIRT. The WFCAM raw data archive keeps growing and internet transfers of UKIDSS raw WFCAM data to ESO continues. 7. Optical processing update A lot of optical processing is happening (INT WFC, Subaru and MegaCam). MJI noted that it is a relief to process optical data after battling with NIR data foibles. 8. AstroGrid liaison update STH said that he had been invited to join the AstroGrid Deployment Advisory Group. Floor is also part of this group representing Planck. RGM said that the next AstroGrid Consortium meeting will be held at Jodrell Bank on 5/6 December. 9. AOB. JRL raised the issue of secretarial support, which will be needed for helping to organize ADASS 2007. We were reminded that CASU have 0.8 of a secretary and that Di Sword was now the designated person. Continuing Actions ------------------ WJS pass on SADT comments to Martin Folger JRL finish off the OMP database access scripts MJI feedback comments to MR on flatfield and sky properties draft report MJI finish investigating merged classification statistics WJS advance the proposed CASU visit to RAL, in particular check that having it coincide with AIT3a is OK PSB report on his findings regarding UKLight WJS provide updated measurements of the VISTA filter characteristics EGS produce set of options for group web pages and circulate DWE New Actions ----------- JRL provide a script to add JAC MSB quality flag to the FITS headers MJI produce a diagram to explain the cross-talk pattern STH in conjunction with the calibration working group recommend a new strategy for observing standards STH select a more crowded field in which to carry out photometric tests and produce and forward (to JAC) MSBs to enable it to be observed DWE test circular apertures for PSF work DWE generate some colour-colour plots from a crowded field to test the PSF-v-pipeline photometry