Overview of the CASU Pipeline Processing ---------------------------------------- Email: mike@ast.cam.ac.uk Web documents: http://www.ast.cam.ac.uk/vdfs/ Pipelines summary ----------------- Summit pipeline: real time DQC and verified raw product to Cambridge proceeding on schedule with CASU software modules installed by JRL at JAC and tested on UFTI data under ORAC-DR Standard pipeline: instrumental signature removal, catalogue production astrometric and photometric calibration will run on a nightly basis in Cambridge, proceding as planned apart from re-scheduling certain tasks because of delay in commissioning Further processing pipeline: year 2 deliverable for PSF and galaxy profile fitting some aspects of this have been started early due to delays in commissioning Advanced pipeline: database driven from WFAU Science Archive, deals with stacking and mosaicing, merging catalogues, list-driven photometry, difference imaging, overlap calibration again several aspects of this have been started earlier than originally planned Data Transfers -------------- Raw data will appear from DAS in 32 bit integer NDF format. A copy of this raw data will be stored at JAC on LTO-I tapes. The summit pipeline will convert the NDF to single extension FITS using lossless Rice tile compression, which gives a factor of ~4 saving. Each detector has its own independent DAS PC and summit data processing PC + LTO-I tape drives on each. The pipeline will write and verify the raw compressed single extension FITS files to 4 tapes in parallel for export to Cambridge, with an expected ~weekly duty cycle. Each tape holds 100Gb native ie. equivalent to ~400 Gb of raw uncompressed FITS files. In Cambridge the 4 tape streams will be ingested, verified and converted to Multi-Extension FITS (MEFs) and then stored (in compressed form) using online RAID arrays. The raw data archive will be pushed to ESO in Garching via the internet. Technical Challenges -------------------- non-linearity measurement and calibration of arrays (at sector level?) [nearly all systems we have dealt with are non-linear] cross-talk between channels and detectors, potentially 128x128 matrix [several WFIs have cross-talk] image persistence from frame-to-frame, adjacency and temporal effects [two possible solutions developed but not used in anger yet] sky level variations on short timescales and/or spatially require several possible sky subtraction strategies [see later] rapid variablity of OH fringing requires specific defringing algorithms in addition to basic sky subtraction [a common problem in all imagers] seeing variations on short timescales [interleaving will lead to some interesting effects] possible time varying detector characteristics [impacts design of MSBs and flatfielding strategies] calibration and extinction monitoring [essential to get well-defined and regular monitoring - see later] data volume and throughput, particularly if much reprocessing is needed [lack of pipeline tuning lead time potentially exacerbates this] generating robust Data Quality Control (DQC) measures [can only visually inspect a small subset of images] Progress -------- Interface Control Documents (ICDs) with JAC and WFAU agreed FITS header definition (crucial for pipeline automation) finalised and test MEF data processed automated DQC monitoring tools developed using images and catalogues provisional pipeline progress monitoring tools with web interface programming completed for all basic modules ORAC-DR test routines for UFTI delivered and installed software module development for standard and summit pipelines on target automated astrometric and photometric calibration developed practical method for dealing with exposure maps, noise variance weighting and bad pixel masks developed V1 stacking and mosaicing and V1+2 difference imaging sofware completed series of simulation tests, and other reports and manuals finished and available on web page tookits/pipelines developed and run on optical data from: INT WFC (4x 2kx4k); ESO WFI (8x 2kx4k); KPNO & CTIO Mosaic1,2 (8x 2kx4k); AAT WFI (8x 2kx4k); Palomar WFI (6x 2kx4k); CFHT 12k imager (12x 2kx4k) and on NIR array data from: CIRSI mosaic on INT/DuPont (4x 1kx1k); INGRID on WHT (1kx1k); UFTI on UKIRT (1kx1k); ISAAC on VLT (1kx1k) Astrometric Calibration ----------------------- Expected WCS is ZPN with simple polynomial radial distortion of form r' = k1 r + k3 r**3 with possibly higher order r**5 term needed. Encoded in FITS using RA---ZPN CTYPE1 etc. keywords/values and using PV2_1, PV2_3 etc. to denote the radial polynomial coefficients. Distortion is wavelength dependent with K > H > J eg. at 0.4 deg radius non-linear distortion is roughly 0.32% in J, 0.34% in H and 0.38% in K As an example: in K distortion at corners of detectors amounts to ~10 arcsec and leads to a differential distortion of a maximum of 0.2 arcsec when stacking detectors taken with an offset of 10,10 arcsec. The scale distortion also leads to a photometric calibration effect since outer pixels project differently on sky compared to inner pixels. This leads with conventional processing, if uncorrected, to photometric systematics up to +/-1% over the array. These can be corrected for using the planned "mesostep" calibration squences, which will also correct for problems caused by scattered light or other non-uniform sky illumination in the flatfields. ========================================================================== NIR array tests with UFTI and ISAAC ----------------------------------- Email: jrl@ast.cam.ac.uk There is a good deal of concern regarding the removal of 2d instrumental and sky signatures from infrared imaging data. The ability to model these effects out properly will determine to a large degree the observing strategies with WFCAM. Members at CASU have spent a lot of time working with data from three infrared imagers (CIRSI, UFTI, ISAAC) to get a feel for the stability of the various instrumental signatures that can occur in such detectors. Fringing is a severe problem with UFTI in the H-band, as it occurs with an amplitude of +/- 5% of sky. This can be removed during the sky subtraction phase using a coaverage of the frames taken over the course of an hour or so. Although the mean fringe intensity varies over this timescale, it is still possible to remove the fringing to a level of less than 0.1% with suitable scaling. With the current service dataset there doesn't appear to be any variation between individual fringes. A sky frame from one part of the night can be fringe corrected with a similar sky frame from another part of the night. Based on our experience with the three imagers, the fringing on UFTI seems to be particularly bad. Similar H-band sky frames with ISAAC show no trace of fringing. The stability of dark frames, flat fields and reset anomaly are all crucial. Dark frames on all three imagers show mainly reset anomaly. The latter appears to be a function of exposure time and incident flux, so it is important that dark frames be taken for each exposure time used during the course of a night as they can be used to remove the reset anomaly for the corresponding target images. With ISAAC the dark frames appear to be stable from night to night. UFTI seems to suffer from a peculiar form of persistence which is visible in the dark frames after exposure to twilight sky. This is what makes using twilight flats on UFTI impossible. It has been agreed that the best way to flat field data is to use twilight flats. As there is only a small window of opportunity for taking twilight flats during the course of a night, it is essential that the instrumental flat field be stable from night to night or even over the course of weeks, so that library flats can be used when none are available during a night. Tests with ISAAC appear to suggest that flats are stable for that instrument at least over the course of a week and even on timescales of a month. ========================================================================== WFCAM Photometric Calibration ----------------------------- Email: sth@ast.cam.ac.uk Draft document: http://www.ast.cam.ac.uk/vdfs/docs/photom.pdf Introduction ------------ The WFCAM photometric calibration concerns the conversion from detector counts to Vega magnitudes in the MKO-NIR system and transformations into other photometric systems. The goal is to measure photometry to 2% accuracy (UKIDSS science requirement). Looking to produce two papers: (i) modeling of the end-end system (before WFCAM goes on the telescope) - synthetic colours of representative astronomical objects. (ii) empirical measurment of the WFCAM photometric system (after commissioning/on-sky characterisation). WFCAM characterisation plan --------------------------- Once commissioning is completed, we'll complete the WFCAM on-sky characterisation. See http://www.jach.hawaii.edu/JACpublic/UKIRT/instruments/wfcam/commissioning/on_sky_tests.html for details. The tests include the measurement of the following: (i) detector noise properties (ii) microstepping test (iii) sky emission (iv) fringing (v) sensitivity (vi) background limit (vii) cosmic rays (viii) persistence (ix) flatfield (x) scattered light: meso-step field, effect of bright stars (xi) area calibration: chip-chip/channel-channel (xii) astrometry (xiii) guiding (xiv) photometry: set up secondary standard fields Primary Standards ----------------- JAC are observing standards (the FS and Persson stars) with the Mauna Kea consortium filter set in UFTI (Simons and Tokunaga 2002, Tokunaga et al. 2002). WFCAM will use the same JHK filter system. There are more than 100 UKIRT standards with (JHK)MKO-NIR which will not saturate a 1s WFCAM exposure (about 50 for a 5s exp). List of standards at http://www.jach.hawaii.edu/JACpublic/UKIRT/astronomy/calib/fs_izjhklm.dat Preliminary results show persistence effects are small (2e-4 after 20s). The UKIRT standards therefore make excellent primary standards for WFCAM. Y,Z and narrow-band filters require extra work Secondary Standards ------------------- I have chosen fields: (i) Evenly spaced in RA (ii) Equatorial (good for VISTA) (iii) dec +20 degrees (X=1.0) - go overhead (iv) >=100 stars on a chip 1. Around UKIRT FS - 100s of stars - primary standard measured simultaneously 2. Near Galactic Plane (5h45+18, 7h15+00, 17h50+00, 20h30+18) 1000s of stars, avoid worst crowding 3. Globular clusters (NGC5053, M3) - Horizontal branch for blue stars - but need to avoid small dense cores. A total of around 100 fields selected - mostly equatorial with around 7 in north and 7 in south - need further refinement to make sure there is nothing big and bright on the WFCAM/VISTA footprint. Nearly all these fields are centred on a UKIRT FS. Further Questions ----------------- How photometrically stable is MKO, e.g. with humidity? How does extinction vary with time on a wet vs dry night? Using UFTI data spanning two years I've found that both photometric zeropoint and extinction are independent of humidity to first order. Nightly Calibration ------------------- We must measure enough standards every night to ensure that the photometricity of a (fraction of a) night can be derived from the data, i.e. of the order of hourly. It will not be under the observers control - this will be under the control of JAC. Overheads E.g. ( (3x5s) +20s ) x 5 filters + slew + acq 5s S/N=100 J=15 (5 min total) So it will take about 5 mins to observe a standard field in 5 filters with WFCAM. We need to think about how the standards MSBs should follow the target MSBs. Should we do standards in all filters even if we spend a night observing in only K ?