WFCAM report from CASU ====================== Data volumes ------------ The current total of raw WFCAM data received for semesters 05A, 05B, 06A 06B, 07A, and the beginning of 07B, is now around 70 Tbytes. This translates to about 20 Tbytes of actual on-line disk storage using lossless Rice tile compression. Processed data volumes are about twice this at around 140 Tbytes due to keeping individual processed science and calibration files, interleaved and stacked products, confidence maps and and catalogue products all online. This translates to around 36 Tbytes of actual on-line disk storage. Much of the processed and raw data is now stored using RAID6 technology rather than the earlier RAID5 standard. We have been investigating in collaboration with JAC personnel the feasibility of transmitting raw WFCAM data to Cambridge via the Internet. An outline plan has been discussed and we are iterating on a procedure. If we can make good progess on this we should be able to reduce manpower overheads, increase data integrity and improve lead time to processing and thence science delivery. WFCAM processing status ----------------------- The remaining NDR data from early 06B was transferred from JAC to Cambridge over the Internet in about a week, and has now been processed and transferred by WFAU. The quality of the October/November data up to the 20th was in general quite poor due to: bad weather, lack of enough good calibration frames (darks and twilight flats), many missing frames (caused mainly by camera problems), and a lot of interrupted incomplete group MSBs. All 2007A data has been processed and transferred to WFAU. Various bits of 06B, 07A service data (Mattila SNe followup; Watson XMM followup) required urgent access to the processed WFCAM data and were dealt with by directly transferring processed data to the PIs. Raw data tapes for 2007B (1 August --> ) have started to arrive in Cambridge. However, at the time of writing (28 September) only two sets of tapes have been received with only 10 complete nights on them. Fortunately these contained sufficient calibration data to compute good quality twilight flats and in nearly all cases correct nightly darks. [The main exception here was for the project U/07B/20 (20070804,20070806,20070813) for which no darks from 07B of the correct type are available, darks from 06B have been used instead.] To access the current state-of-play of data transfers and pipeline processing status, plus quality control diagnostics, see http://casu.ast.cam.ac.uk. WFCAM reprocessing status ------------------------- A number of reprocessing requests for 06A and 06B data prior to DR3 have been received and dealt with: Some earlier GPS frames from 06A required reprocessing to fix problems introduced by a known feature of the then extant cross-talk correction algorithm in regions of bright spatially varying nebulosity. Several sets of frames with poor sky subtraction were also reprocessed but as most of these were caused by poor weather curtailing observing, not too much could be done about them. We have developed and tested a new, and more CPU-intensive, sky estimation algorithm involving iterative object masking to attempt to handle some of the low level transient "features" visible in some UDS stacks. This is designed to work at the pawprint level, when suitable tiles have not been observed. This has now been thoroughly tested and included as part of the standard processing for all 07B data. A few problems with aberrant master dark frames turned up during jpeg visual inspection for DR3. The first of these was traced to a whole night of bad darks used to make the dark frame corrections. Unfortunately this affected two processed nights of data which required complete reprocessing. A further night had similar problems and also required (partial) reprocessing. This has lead to a further tightening up of procedures for checking master dark frames. [These problems were all related to the so-called channel-edge problem which is mainly obvious on detector #3 but also randomly affects all the detectors.] WFCAM recalibration ------------------- We have written a robust MLM estimator to counter assorted problems with the curve fitting to better constrain the zero-point offset for the Y and Z-band data. Analysis of the Z and Y zero-point offsets is now complete and shows no correction is required for Z and a 0.08 offset is needed for Y to bring it onto the Vega system. No changes are required to the colour equations. Previous analysis of the WFCAM illumination correction variation for semesters O5A, 05B and 06A concluded that a factor of two improvement of the systematics in the photometry could be achieved by the simple expedient of using 2MASS residual-derived corrections to update the individual detector zero-points. Photometric recalibration of all processed data (05A,05B,06A,06B,07A) has just been completed and made available to WFAU. This includes correcting the Y-band zero-point and providing individual detector zero-point corrections (typically ~1-2%) based on monthly analysis (following the flatfield frames used) of photometry residual systematics as a function of spatial location in the array. Monthly tables and figures detailing the remaining residual systematics, ie. after applying the detector-level corrections, have also been computed and will be made available from the CASU web pages shortly. These remaining effects are generally low level ~1% but in certain regions and/or passbands can be several times larger. There are some interesting effects visible when data from all five WFCAM semesters is examined (see attached plot). These imply that some of the detector level offsets are most likely caused by flat pedestal level effects (these can also be seen in the sky corrections but in this case they are generally additive and do not impact object photometry), but there is also clear evidence that the dominant effects are caused by small-scale QE variations between the detectors. This manifests itself at the ~+/-1% level as the more constant trends seen in the offsets. The problem is apparent because the twilight sky used in the flats is bluer in colour than the majority of objects and hence any colour equation differences between the detectors manifests itself as a small zero-point offset. Details of the complete WFCAM photometric calibration are being incorporated in a draft MNRAS paper.