WFCAM report from CASU
======================
Data transfers
--------------
The current total of raw WFCAM data received for semesters 05A, 05B, 06A,
06B, 07A, 07B and the first two months of 08A has reached the 100 Tbyte
mark (around 25 Tbytes of RAID disk storage with Rice-tile compression).
Processed data amounts to some 180 Tbytes (45 Tbytes compressed on RAID
disks).
The pre-08A plan was to transmit all data via FTP from a summit server.
However, there were some problems with setting up the summit server which
delayed the start of these transfers and the first few weeks of 08A data
were later written to tape, arriving here March 20th. In parallel with this
the FTP transfers began in earnest toward the end of March. One consequence
of this is that we could not start processing 08A data until March 31st
and of course by then a rather large backlog had built up.
After further iterations and debugging, automatic transfer of WFCAM data
is now working routinely. Each night of data is placed on a JAC summit server
as 1-chip Rice-compressed FITS files and a flag set; a cron job in Cambridge
scans the flag and mirrors the data. Transfer speed is around 8 MByte/s with
a night of data taking typically two hours to transfer. This currently uses
a 32 thread transfer protocol, not ideal since it clobbers the server end
more than necessary, but will do for now in lieu of further server system
tuning. As a result of this, observations are now generally sitting on disk
here within 24 hours of being taken (thanks to Brad and Pete for sorting
most of this out).
Data still arrives on a per detector basis and is checked and factored into
MEFs as usual. Since all WFCAM data from now on should arrive via ftp we have
instigated writing our own archive tapes (currently using LTO-IIs) from the
MEF'd raw 08A data as an off-line backup. Previously we just saved the
incoming tapes.
All raw UKIDSS and calibration data available at CASU has been transferred
to ESO for 07B and 08A is just starting.
WFCAM processing 07B
--------------------
We had to wait quite a while for the last week of WFCAM data from 07B to
arrive which delayed signing off on the 07B processing until near the
end of February. (The last night of 07B WFCAM data was the 3rd December.)
All of the processed 07B data was transferred to the WSA by the end of
February.
A new feature of WFCAM data (the parquet floor effect) has become a serious
nuisance since our previous Board report was written. The effect mainly
manifests itself as a variable channel level "offset" on detectors #2 and #3
and has both a repeatable and a random component, in addition to being
strongly, but not completely, anti-correlated between the two detectors.
An example of the repeatible part of the effect is available at
http://casu.ast.cam.ac.uk/surveys-projects/wfcam/data-processing/
2007b-known-features. This shows a sky frame formed during the original
processing of one of the September nights. Unfortunately, the random
component is also about the same level (~1% of sky) and even after sky
subtraction affects some science product frames. A possibly related problem
also occassionally affects detectors #1 and #4.
The parquet floor pattern was first seen seriously in 07B data from
mid-September onwards and is still present in 08A data. If affects virtually
all nights at some level and consequently we have had to devise some extra
processing options to deal with it.
After some investigation it was clear that this also seriously affects most
of the twilight flats (which would further contribute to the problem if used
blind in processing). After a series of tests to ensure that other
flat field characteristics had not changed during the camera engineering work
in September, we decided to use the August flats for all 07B processing.
We have also found that the pipeline-created sky frames are a good
diagnostic of the problem being present and use these to investigate which
nights are affected. To date we have not seen this problem in master dark
frames produced by the pipeline.
With some misgivings about its general use, we developed a channel-level
software fix which if all else fails we use on affected data in an attempt
to mitigate the problem. Unfortunately, this has to be run at the individual
frame level, ie. post-sky subtraction but pre-interleaving and stacking,
due to the random nature of some of the effect. This involves selective
partial reprocessing of affected MSBs and significantly adds to the overheads
of running the pipeline. We currently use the following recipe:
a. for each night we run the pipeline as normal
b. check the skies for banding - which seems a reliable diagnostic
c. if banding is present it seems to affect most (CDS and NDR data)
frames in a night.
d. banding in UDS, DXS, GCS, GPS products has so far been negligible
due to stacking a lot of frames or crowding hiding any sign of it
e. all Z, Y, 1-0S, nbJ and LAS J individual science frame runs from the
pipeline output for nights with banding present are then corrected for
any channel banding for all detectors (since it potentially can,
and has, occured on all detectors at various times)
f. we then rerun the pipeline for these files from the interleaving/
stacking and catalogue generation phase.
This and the monthly detector zero-point updates (see later) are delaying
normal release of data by about an extra 2-3 weeks. So far we have had no
complaints about this extra delay but will continue to monitor the situation.
WFCAM processing status
-----------------------
The parquet floor problem is still present in 08A data and in particular
is in the twilights flats. This time because the instrument had been off
the telescope we had to compute new flats in a slightly more creative
way than normal. Briefly, proto-master flats are created as usual and are
then flatfielded using previous good master flats as a reference. The
parquet floor pattern on #2 and #3 then shows up clearly and is removed with
the usual bespoke software. Flatfielding^{-1) the output of this process
in a similar way with the reference flat restores the parquet floor-free new
master twilight flats. 08A master flats for the February-March period for
Z,Y,J,H,K,1-0S and BGamma have been produced this circuitous way and we will
create further monthly flatfield updates using this approach.
A new, but fortunately minor, problem related to scaling of coadded exposures
was noticed whilst processing the first few nights of 08A data. The cause
of this was traced to the FITS compression conversion due to a software
upgrade of the external cfitsio library, which we had supplied prior to 08A
data taking. A patch has been issued to JAC to fix it. Already transferred
data is fixed here in situ prior to ingestion to the raw WFCAM archive and
subsequent transmission (UKIDSS and CAL data) to ESO.
In spite of these problems and a few others relating to missing darks and
missing RECIPE keywords, processing is rapidly catching up with data
taking, though final checks and monthly zero-point updates remain to be
done. The status pages for 08A show the current state of play
http://casu.ast.cam.ac.uk/surveys-projects/wfcam/data-processing/
and derived quality control diagnostics.
As usual requests for urgent early access to various bits of processed
service or PI data for 08A have been received (20080401 Patel U/07B/18;
20080331 Tedds U/SERV/1779; 20080331 McMahon U/SERV/1783; 20080310,20080320
Haines U/08A/32) and were dealt with by directly transferring processed
data to the PIs.
WFCAM reprocessing status
-------------------------
We are up to date with reprocessing requests, though PI checking of UKIDSS
07B processed data is still ongoing.
WFCAM calibration
-----------------
We have tested the spatial illumination correction maps
(http://casu/surveys-projects/wfcam/data-processing/illumination-corrections),
by comparing the before and after overlap regions in LAS data.
The rms distributions for repeat measurements of stellar sources in the overlap
zones (ie. dominated by sources near the edges of the detectors) improves
markedly and becomes 0.015-0.016 mags in JHK and 0.019 mags in Y. The
improvment occurs at all magnitude levels and similar improvements occur
for non-stellar sources. This strongly suggests that the illumination
correction is indeed as its name implies.
This analysis has been added to the photometry paper and a close to final
draft of the photometric calibration paper has been generated (available
at http://www.ast.cam.ac.uk/~sth/wfcam/photom_draft.pdf).
Individual detector magnitude zero-point corrections are computed monthly and
are adjusted in all FITS headers before they are flagged as read to copy to
the WSA. Post-processing application of the illumination correction will be
applied at the science archive end at ingest to maintain our data processing
philosophy of leaving data within FITS tables in natural units. External
calibration information either resides in the FITS headers or in external
correction tables.
UKDISS wishlist
---------------
An informal discussion took place during the UKIDSS workshop in Garching
(17-19 December) regarding a wishlist of improvements to WFCAM processing.
We list them here but note that progress on some of these awaits the
outcome of the STFC spending review and the current CASU rolling grant
application which was reviewed by a special panel January 22nd.
1. Poor background subtraction (or residual artefacts) in a few
(nebulous) regions. After investigating a few representative examples
it transpires that most of the problem (apart from the
known ancient cross-talk bug issue) was caused by the decurtaining
stage introducing a "Maltese Cross" pattern in the background. This
is a known feature of the decurtaining algorithm that occurs in rare
cases where the strongly varying background interferes with the
curtain fixing estimator. An optional workaround has been tested and
implemented.
2. The current sky estimation strategy generally works well but there
are a few specific cases where problems occur. After discussions with
JAC staff prior to, during and after the meeting an outline method of
alerting the pipeline to such cases was agreed on. As a result of this
as part of 08A processing we are currently, in liaison with JAC, trialling
alternative sky estimation strategies involving flipping target/sky between
detectors. This requires some modifications at both ends to observing
recipes and to header keywords. This will hopefully get around the majority
of the cases where the pipeline is trying to second guess the best strategy
without full prior knowledge.
3. The desirability of elliptical apertures for galaxy photometry was
raised. The obvious compromise would be to stick with the current
fixed circular apertures and solely use elliptical apertures for
Petrosian, Kron and Hall measures. We await the outcome of the
prioritised UKIDSS wishlist (and grant review).
4. The GPS team want better photometry in nebulous regions, both from a
"point source" detection and flux estimation point-of-view. At the
meeting we noted that using a smaller scale background tracker was the
simplest thing to try out. Tests after the meeting showed that although
an improvement this was not the answer and an alternative background
tracking method has been trialled. This potentially provides a solution
for the point-source detection and parameterisation issue in regions of
complex nebulosity and has been implemented and tested, and is currently
awaiting feedback from Phil Lucas.
5. The desirability of full iterative PSF model fitting and Sersic profile
fitting (presumably in both cases only for objects/fields worth
bothering with since this will be seriously CPU intensive) was raised
This has always been in the long term development plans but has not been
completed yet. It also depends on the outcome of the grant review.
Other
=====
An MOU between JAC, CASU and WFAU for dealing with WFCAM data was finally
agreed and signed by all parties.
As noted earlier, the CASU rolling grant was reviewed by a special panel
on 22nd January. This grant bid covers operation of the WFCAM processing
pipeline and further VDFS software development. The notional start date
for the new rolling grant was 1st April 2008. We are using the rollover
from the previous grant to cover current WFCAM and VDFS activities.