WFCAM update from CASU ====================== Data volumes ------------ The current total of raw WFCAM data received for semesters 05A, 05B, 06A and 06B is now well over the 50 Tbyte mark. This translates to around 16 Tbytes of actual on-line disk storage as the lossless Rice tile compression used is achieving an factor of 3.5 saving in disk space, whilst still maintaining the FITS file format and rapid access. Processed data stored on-line is nearly 150% of the raw data in native volume, due to saving both the individual processed files and the interleaved products, intermediate and stacked. Here the average saving on disk space required from using Rice tile compression is a factor of 4 and hence requires about 24 Tbytes of on-line disk storage space. All of the ingested raw and processed is stored on-line to enable rapid reprocessing and recalibration, and rapid access to the raw and processed data for science verification and fault diagnosis. This also provides an off-site backup of the processed data products stored in Edinburgh. For access to the current state-of-play and the various quality control information generated by the pipeline processing see http://casu.ast.cam.ac.uk. WFCAM archives and processing ----------------------------- There have been a series of problems with raw WFCAM data transfer that have delayed the processing of the first 8 weeks of semester 06B. After a fruitful discussion during a recent visit from Andy Adamson and Gary Davis everything now seems to be back under control. A simple but effective protocol to share information on data transfers with JAC has been put in place using collaborative Google spreadsheets. Now that we have several weeks of complete WFCAM data, processing of 06B data has begun in earnest and data transfer of the relevant raw WFCAM files to ESO for 06B has started. There are still problems with data prior to 20th December, but WFCAM data from 20th December is ok and is being ingested, verified and archived within in a few days of arrival of data for all 4 detectors. Pipeline processing takes roughly up to 1-2 weeks from this point and so far is generally well ahead of the arrival of new data. The raw WFCAM data archive in Cambridge is up to date (in terms of received data) and the occassional user has requested access to raw data, however, this is not very common. Several users have requested direct access to their processed PI data without waiting for transfer to Edinburgh and we have supplied them directly with the processed data products. We have made a few improvements to the cross-talk correction to make it more robust in regions of complex nebulosity, where it had previously introduced some extra (-ve) artefacts. This was in response to a problem noted previously by Phil Lucas in areas of bright spatially varying background nebulosity. This affects very few regions and we have left it up to end users to contact us if any of their data requires reprocessing because of it. A series of new intermittent faults mainly on detector#3, but on other detectors also, are causing a few problems. These are in hand but have lead to the occassional hiatus in processing mainly due to their impact on the sky estimation phase. Some examples are available under "known features" on the processing status web pages. WFCAM photometry ---------------- The current calibration applied to all WFCAM data for 05A, 05B, 06A, and 06B as it is processed, is still awaiting the final offsets from the photometry working group for the Y- and Z-band calibration. This is to to ensure that A0 stars have zero colour in all WFCAM broadbands. For the current status of this see http://www.ast.cam.ac.uk/~sth/wfcam/wfcamcal WFCAM astrometry ---------------- From 06B all WFCAM astrometric calibration is using the waveband-specific distortion terms noted in the last report. We will retrospectively apply these to all processed WFCAM data and update all the astrometry of the processed data for 05A, 05B, and 06A. Although this level of improvement of the systematic errors (~<25 mas rather than ~<75mas) is not needed for many applications it will lead to improved proper motion estimates particularly when different bandpasses are used for the different epochs. As noted previously we have also developed and implemented, a simple robust variant on Maximum Likelihood centroiding. This adds minimal overhead to the processing but yields up to a 50% improvement in individual object rms astrometry, yielding close to optimal centroiding for both stars and galaxies. This is now being routinely run on all processed data and will again enable better astrometric precision, in particular for proper motion measurements. PSF fitting ----------- The GPS tests in sample crowded fields and areas of varying nebulosity have been completed. These tests were carried out in conjunction with Phil Lucas (University of Hertfordshire) who used DAOPHOT as a control comparison. The tests compared standard pipeline aperture and prototype pipeline PSF photometry with DAOPHOT PSF photometry. Results are mixed. In general the standard pipeline gives marginally better results but in specific cases, eg. in the middle of globular clusters, DAOPHOT is better. The conclusion is that unless we can implement full iterative crowded field PSF photometry in the CASU pipeline(s), the improvements given by PSF development stages #1 and #2 are extremely marginal and the only real benefit is more reliable error estimates. The current development of the CASU PSF software has several innovative features and is being written up as part of a draft paper which is intended to be submitted to MNRAS. Sky brightness properties ------------------------- MR has finalised an analysis of sky brightness and is writing a paper (some of the work was in presented at an ESO workshop). The main conclusions are: * there is no significant difference in sky brightness between bright and dark time for YJHK, a 0.5 mag effect is seen in Z (brighter in bright time); * there is no significant Moon effect in YJHK when at a reasonable (>20 deg) distance, a safe distance for Z is >30 deg; * sky brightness reduces as a function of time after twilight for J and K; * there is a correlation with temperature for K as expected using a simple Black Body model. He has written a short article on this for the recent UKIRT newsletter. Summary of Cambridge processing =============================== Version Semester Images Catalogues Calibration Available process|pipeline #1 #1 05A stacking bug no errors 2MASS old ce May 2005 or Petrosian -Jul 2005 fluxes #2 #2 05A fixed stacks full standard 2MASS new ce Aug 2005 better skies parameter set -Sep 2005 #3 #4 05A +cross-talk +minor catalog +J-Ks limits Apr 2006 removed bug fix -May 2006 --------------------------------------------------------------------------- #1 #3 05B +cross-talk full standard 2MASS new ce Oct 2005 removed parameter set -Feb 2006 #2 #4 05B || +minor catalog +J-Ks limits May 2006 bug fix -Jun 2006 --------------------------------------------------------------------------- #1 #4 06A || || || May 2006 -Sep 2006 --------------------------------------------------------------------------- #1 #5 06B +more robust astrometry see notes Mar 2007 cross-talk improvement below - removal --------------------------------------------------------------------------- [all version#4 processed data has updated photometric calibration based on a restricted J-Ks range and an empirically-derived Galactic extinction term] [all version#5 processed data has waveband-dependent astrometric distortion coefficients] ===========================================================================