WFCAM update from CASU ====================== Summary: Over 25 Tbytes of raw WFCAM data has been successfully transferred to CASU and some 50 Tbytes of pipeline processed data products made available to WFAU (ie. roughly two million 2kx2k science images!). Without Rice compression, which gives a factor 3-4 saving on volume, storage costs would be a serious problem since all of this currently has to be kept on-line in Cambridge (raw and processed) and in Edinburgh (processed plus extra products). Pipeline products are flux calibrated and astrometrically calibrated with respect to 2MASS. No further calibration is currently done since the pipeline output has been shown to be within the overall requirements specification of 2% for JHK photometry and 100mas for astrometry. An assortment of quality control parameters (seeing, sky properties, limiting magnitude etc.) are generated by the pipeline and are used to assess data quality and the astronomical utility of the data. ----------------------------------------------------------------------------- WFCAM processing status: An up-to-date online summary of the processing status together with assorted quality control (QC) plots etc. is available via http://apm15.ast.cam.ac.uk/wfcam/. After processing, the derived QC and other useful header information are ingested into our internal WFCAM DQC database which, among other things, is used to generate UKIDSS survey progress web pages. Processing of 05B data was finished at the beginning of March. Steady state processing was attained mid-way through the semester with data processed, QC checked and released within 4 weeks of it being taken (and processing waiting for tapes to arrive). There was a minor hiccup with 3 nights at the end of the semester when one of the detector controllers (#3) and the spare went walkabout. The complete absence of ANY #3 data required a workaround. We have received no community feedback at all on the processed 05B data, even though the first tranche of the processed data was made available in October, well over 6 months ago. All of the processed (Version 2) 05A data and (Version 1) 05B data (see appended table for details) has been transferred to WFAU. Reprocessing of 05A data (Version 3) is well underway and should be finished mid-May. The main changes are an improved sky correction strategy and application of the cross-talk correction algorithm that was developed and used for processing 05B data. PSF fitting: the split PSF measurer is working well and a baseline PSF fitting programme using this has been written. However, this latter software still does not given optimum results. For example, data covering M17 was analysed to compare the performance of the PSF and standard pipeline photometry. For frames with good seeing, there was, as expected, negligible difference between the two methods. However, for poorer seeing the aperture fluxes were better. An improved PSF fitting algorithm is being developed in which the microstep interleave components are analysed in parallel to fix this problem. Work on this is still in progress and has delayed incorporating PSF measures into the catalogues. Assorted PSF reports are available at http://www.ast.cam.ac.uk/vdfs/docs/reports UKIDSS EDR Sci and CASU papers: CASU contributed several parts of the EDR and Sci papers. A first draft of the CASU technical paper has been produced based on an updated SV report and the earlier WFCAM commissioning report. The SV report was udpated to include a diagram to "explain" the cross-talk pattern. In addition colour-colour and colour-magnitude diagrams of IC4665, M17 and an LAS tile were produced and included, along with a brief howto, as examples of the quality of data that the pipeline is generating. The SV report can be found at http://www.ast.cam.ac.uk/vdfs/docs/reports/sv EGS generated assorted stacked DXS data and catalogues as part of the Early Data Release (EDR). Though there was some confusion about how this fitted in with what WFAU were doing. Raw WFCAM archives: All of the relevant raw WFCAM 05A and 05B (ie. UKIDSS + calibration data) internet transfers to ESO are complete. The raw WFCAM data archive in Cambridge is up to date and typically lags about 2-3 weeks behind the date the observations are taken. Transfers of UKIDSS and calibration data to ESO are automated, but due to the more limited bandwidth connection to Garching, are typically a few months behind the data being taken. However, all the 05B UKIDSS raw data (and 05A) was available from the ESO archive by the last week of March. WFCAM photometric calibration: We have recommended in conjunction with the calibration working group a new strategy for observing standards (every ~2 hours but only on photometric nights). We have demonstrated that 2MASS can be used to calibrate WFCAM JHK to better than 2% (tested against UKIRT Faint Standards). Testing the Y- and Z-band calibration is still underway using SDSS comparisons and both Y and Z measures will be further compared with updated Z and Y pseudo-standards from Sandy Leggett when available. The report is available at http://www.ast.cam.ac.uk/vdfs/docs/reports/2masscal.pdf The updated WFCAM calibration uses a restricted 2MASS J-Ks colour range of 0 < J-Ks < 1 to help exclude late-type giants, weird objects and heavily reddened stars. This should produce a more stable calibration for the Y and Z bands (but generally have little impact on the JHK calibration). The new calibration is being applied to the (Version 3) latest 05A processing and will be applied retrospectively to all data held here. In addition, errors on the individual 2MASS-derived zero-points are now included together with measures summarising the photometric quality of the night and the overall nightly average zero-point. Systematic magnitude residuals as a function of position have also been investigated and quantified using stacked 2MASS-WFCAM residuals data from all suitable 05A and 05B data. The rms variation over the arrays is within 2% but there are patches where the residuals are noticeably worse than this. This still requires some further investigation but we anticipate releasing correction tables to go with the upcoming 05A and 05B data releases. WFCAM persistence investigation: a practical robust scheme to directly correct the image data does not seem possible. The error on the correction is similar to the size of the correction. The source of the scatter may be systematic in some way rather than random but is proving difficult to measure and hence quantify. For interleaved and deeper stacked surveys the impact of persistence is minimal - persistence images are not generally detected as objects by the cataloging software. Post-pipeline processing flagging options are a viable alternative eg. using 2MASS stars and a simple persistence model as a predictor of possibly suspicious images. However, it is felt that this issue should be science requirement-driven ie. is persistence causing problems in doing the science? The persistence report, including some examples, is available at http://www.ast.cam.ac.uk/vdfs/docs/reports/persistence/ WFCAM pipeline issues: RGM reported some deblending problems identified from outliers in the LAS. Some were caused by seeing varying and are a fact of life. A few others were tracked down to a low-level bug in the pipeline catalogue software that was not present in the development version. This bug was fixed prior to commencing 05A reprocessing. Although multiple comparisons are routinely carried out between the development toolkit software and the pipeline version, this bug had slipped through due to its unusual nature. Weird science outliers are, of course, a good way of finding pathological problems. There have also been some reported problems related to deep stacking and cataloguing UDS data. The usual CASU pipeline output produces 32-bit integer FITS. Given enough files in a deep stack quantisation noise can become an issue. The post-pipeline processing stacking software has an option to produce real*4 output if required and should resolve this problem already. The other quantisation problem was caused by a software limit in the supplied pipeline catalogue software. A simple workaround for this was suggested at the time and a more robust solution is under development. The UDS data is also causing a problem with catalogue generation related to a known limitation of the object detection algorithm with over-sampled data. A fix for this will be supplied in advance of the next data release. Summary of Cambridge processing =============================== Version Semester Images Catalogues Calibration Available process|pipeline #1 #1 05A stacking bug no errors 2MASS old ce May 2005 or Petrosian -Jul 2005 fluxes #2 #2 05A fixed stacks full standard 2MASS new ce Aug 2005 better skies parameter set -Sep 2005 #3 #4 05A +cross-talk +minor catalog +J-Ks limits Apr 2006 removed bug fix -May 2006 -------------------------------------------------------------------------- #1 #3 05B +cross-talk full standard 2MASS new ce Oct 2005 removed parameter set -Feb 2006 #2 #4 05B (as above) +minor catalog +J-Ks limits May 2006 bug fix -Jun 2006 ===========================================================================