WFCAM report from CASU ====================== Data transfers -------------- 08B and 09A internet data transfers have been proceeding routinely. The raw WFCAM data arrives in Cambridge and is checked, MEF'd, OMP'd and ingested within 24 hours of it being taken. The nightly processing is usually started shortly thereafter. Checking the products, and bundling up the monthly photometry updates takes the rest of the time. The raw WFCAM data archive in Cambridge is also up to date, as are the transfers of all raw UKIDSS and calibration data to the ESO archive. The current state-of-play including transfers and processing status can be found at http://casu.ast.cam.ac.uk/surveys-projects/wfcam/data-processing/ The total of raw WFCAM data received for semesters 05A-08B and the first two months of 09A stands at around 130 Tbytes (~32 Tbytes of actual RAID disk storage with Rice-tile compression, equivalent to ~6 million 2kx2k science images). Processed data amounts to around 225 Tbytes (~56 Tbytes compressed on RAID disks). By spending all the remaining (old) rolling grant equipment budget we now have enough disk storage in place for both raw and processed WFCAM data for the next 3-6 months, after that nada. WFCAM processing ---------------- 08B processing was finished by the end of December and the assorted monthly photometric updates completed in early January - though see later item on reprocessing to fix a bug. 09A processing was held up for about a month while waiting for a full set of sky flats to be taken. The long wait was mainly caused by uncooperative weather which had the side effect of not that much data to process either. Processing is now up to date and can be tracked at the processing status page for 09A (http://casu.ast.cam.ac.uk/surveys-projects/wfcam/data-processing/ report_night_reduction_status_casu?semester=09A&SUBMIT=Submit+Query). There are occassional requests for fast track processing for urgent data access, though most are dealt with by simply making fast track direct access to the flat file data products as they come out of the pipeline. The extra load imposed by this is currently marginal, but is monitored and asssessed in lieu of other operational requirements. Pipeline updates ---------------- Correcting for the variable level channel "offset" problem first noted during 07B has now been incorporated directly into the pipeline, obviating the need for subsequent reprocessing of selected data and improving overall throughput and required effort. Monthly photometric detector level calibration and generation of illumination correction tables remains the limiting factor in flagging data for transfer to WFAU. The MBSTid log that monitors MSB quality has been running routinely for all of 08B and 09A as the data is pipeline processed. These are made available for download on the processing status web pages. The various offset sky options have now been fully incorporated into the pipeline and together with the various standard sky subtraction strategies already on offer cover all common science use cases. WFCAM reprocessing for bugs --------------------------- Assorted reprocessing issues have cropped up or been slowly progressed since the previous Board report. Three years downstream a long standing bug (and feature of the method) with estimated Petrosian radii and fluxes, also affecting Kron and Hall measures, for deblended objects was reported to us. This affects all processed WFCAM data. Fortunately, due to the rather large number of flux measures in the catalogues, a fix has been found for this that can be run on the existing catalogues. Although the update can potentially be applied quite rapdily, progressing it all the way through the various data products from 05A-08A in the middle of DR5 preparations did not seem like a good idea. This update will therefore be rolled out for DR6. All of 08B and 09A catalogues are already generated using the modified software. A different but rather subtle bug to the catalogue generator was introduced on 15th April 2008 when some very minor fix to deal better with heavily saturated stars (sic!) was introduced. Unfortunately due to its unpredictable and hard to spot consequences (it was only actually found by Mike Read during band-merging tests for DR5 earlier this year) this required us to regenerate all the catalogues from 20080312 onward (i.e. the rest of 08A and all of 08B) and lead to some further delay in the release of DR5. It also required recalibration of the photometry, detector zero-point offsets and illumination corrections, though the only changes to the images were updated photometry coefficients in the headers (typically only at the 1-2% level). All the catalogues for that period were regenerated within a week and the reprocessed data was transferred (again) by WFAU and incorporated into DR5. WFCAM reprocessing for users ---------------------------- The UDS team came in with an unexpected request to reprocess all the UDS K-band data for 05B, as this semester currently has a high attrition rate due to background variations. A few tests on typical nights unfortunately confirmed that the latest "pawprint" sky estimation strategy, with iterative object masking, does appear to do a much better job. This reprocessing was completed over the Christmas period after 08B processing was finished, and made available to the UDS folk for assessment on 2nd Jan 2009. Sacre bleu the reprocessed 05B K-band data, using the pawprint sky method developed for UltraVISTA, enthused them so much (0.3 mags deeper) they wanted to reprocess ALL the UDS data the same way. With the help of the appalling 09A weather all the UDS K-band from 05B-07B was reprocessed by mid-April. The plan is that this will be stacked by the UDSer's, by ~mid-May and make it into the UDS DR5 release. UDS J- and H-band reprocessing are on hold until the GPS nebulosity filtering exercise, required for GPS DR5, is sorted. This is to improve both detection and photometry in nebulous regions. It is next in the queue - honest! Phil. A much shorter request for reprocessing a limited amount of Orion data taken for U/05a/100 also came in over the same period. The original sky estimation and subtraction for this was so bad that this was pushed through quickly. This data also provided an interesting test case for the nebulosity filter developed for GPS photometry in regions of high nebulosity. This was duly applied to give an alternative sets of nebulosity-filtered catalogues. Ricky Smart contacted us at the UKIDSS meeting held at the RAS in December about the parallax programme. Although satisifed with the results he was curious that his own detailed attempt at PSF-based astrometry actually gave worse results than the pipeline cunning maximum likelihood-based astrometry. MJI suggested trying "dribbling" the 2x2 interleaved stacked data to improve the PSF characterstics and provided him with the appropriately dribbled frames and corresponding catalogues. Although the PSFs were now well-behaved (i.e. not "spiky") RS's PSF fitting still gave worse results than the pipeline astrometry! Dribbling on demand ------------------- The "dribbling" algorithm mentioned above has now been used quite frequently on request and works from the final pipeline stacks. If interleaving has been used this provides a simple robust solution to correct the PSF mismatches that occur in all interleaved data. It makes use of the fact that the real pixels are 0.4 arcsec in size and overlap the neighbouring pixels in a 2x2 (or 3x3 if used) interleaved 0.2 (or 0.133) arcsec grid. This runs directly on the pipeline output interleaved stacks and gives much better behaved PSFs and, of course, correlated pixel noise. It is particularly beneficial though when mosaicing, or doing even deeper stacking, since this usually involves resampling/interpolation anyway. Funding ------- In spite of the best efforts of several interested parties there has still been no further progress on the rolling grant application submitted almost 2 years ago (in 1 years time we should be submitting the next one). Current rolling grant funding for several CASU activities, including WFCAM processing, has basically run out. We have already had to hire out key personnel to other (funded) projects to pay the staff costs. In addition, another key team member died at the end of last year and we have not been able to replace him, even though we have carried out a recruitment exercise and identified a suitable candidate. We have minimised the impact on 08B and 09A WFCAM processing by giving them the highest priority. However, due to pressures from other projects and a serious lack of personnel a series of reprocessing requests and software enhancements, which would normally have been carried out in parallel with processing, have been delayed. At the time of writing the major outstanding reprocessing requests include: updating 05A-08A catalogues to fix Petrosian radii and magnitudes for deblended objects; although we have reprocessed UDS K-band data from 05B-07B, reprocessing J and H are still outstanding; and of course re-analysing GPS data in regions of high nebulosity to improve source detection and photometric accuracy. Delayed software enhancements include a series of minor ones, such as computing average confidence map scores over the default apertures to aid in spurion rejection/edge detection mainly for users of flat file catalogues; to major ones such as implementing optional full iterative PSF model fitting and Sersic profile fitting. Progress on all of these, and much else, is going to grind to a halt unless the rolling grant issue is resolved.