27th November 2008 APM meeting tables 11:30-12:50 Present: MJI, JRL, STH, MR, DWE, EGS, RGM, RWA Apologies: NAW, PSB Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Recent and upcoming meetings 4. Data archives update - AAT, ING, WFCAM, UKIRT 5. Optical/NIR processing - HAWKI, INT WFC, MegaCam, Subaru 6. WFCAM update 7. VISTA update 8. AOB Minutes ------- 1. Actions from last meeting STH has not finished checking that the Plone astrometry technical pages are correct, however, the photometry pages have been updated and the astrometry ones are "mid-edit" - ongoing <<<< STH has included the graphic showing the size of the photometric radial distortion correction in the Plone technical pages JRL has not produced a web page of HawkI publicity for CASU - ongoing <<<< MJI have not finished off the stage-I software release pages - ongoing <<<< EGS JRL attended the Cambridge VOTC meeting MR STH has not made the photometric calibration paper available on the Plone and VDFS sites since he is waiting for the referee's response on the re-submitted paper. Apart from holding off placing this on astro-ph it was not thought necessary to wait further - make it so <<<< MJI has not finished off the technical paper - ongoing <<<< MJI has passed on feedback regarding CASU contentment with proposed directory structure for transferring data on USB disks MJI has informed ESO that CASU would like to acquire the first few nights of SV data as soon as possible after it is taken PSB has made some processed VIRCAM test data available to Edinburgh STH acquired some decent calibration data for VIRCAM MJI made sure JPE/WJS are aware of the Ndit=1 issue for twilight flats 2. Comments on WFAU minutes There were no specifc comments other than to note the plans for the next UKIDSS data release, DR5. 3. Recent and upcoming meetings EGS was asked to check if the CASU presentation from the ESO Public Surveys meeting in September is available on their website. <<<< ESO meetings: MJI was over at ESO for the LP workshop and had a meeting with the ESO members of the VISTA SV teams. Among other things discussed were action items on fast SV data access and the format (apparently Rice-tile compressed) of the 500 Gbyte USB disks which ESO will deliver VISTA data on. He also confirmed that PIs of VISTA surveys will get access to the raw data, if required, direct from ESO. The ESO Garching pipeline will run on all calibration frames to derive QC information but will apparently only sample a subset of the science frames for QC purposes ! QC0 run at Paranal is mainly concerned with conditions of the observations and visual checks. Real time OB grades are only stored locally in logs/DBs which are heavily firewalled => summit OB grade will not make it into the FITS headers at least initially. A processing status web page similar to that used for WFCAM could be used for feedback of OB information derived at CASU, but there is still an open question of how to close the loop on bad OBs, though most should get spotted at Paranal ? We need to emphasise that the onus is on the survey PIs and ESO to access and make use of the information on our web pages. MJI was also over at ESO for the OPC meeting in November (17th-20th) and discussed further VISTA narrow-band NB118 processing and assessment issues (bandpass dependence on radius, sky fringing, calibration etc.) with Palle Moller and Wolfram Freudling. Our default scheme for calibrating the NB118 data, i.e. treat it like any other band and calibrate on 2MASS, may work well enough for their immediate science needs. A simple test is to monitor the stability of the difference between ZPs derived from consecutive J and NB118 data. So far this looks promising. Estimating the likely size of illumination corrections from existing VIRCAM data may be possible but no firm promises have been made since this is still TBD. To ease their worries about NB118 and ZY calibration STH has made the WFCAM calibration paper available to them. We also need to check what the Calibration Plan document currently says about these issues. <<<< One further outcome of these meetings was the need to arrange another meeting ~February with the SV teams to discuss results from first pass processing and their initial assessment of the products. It was agreed that we would try and provide rapid pawprint level access to these data products from the first two nights to facilitate the assessment process. It was agreed that access via a passwd-protected web page would be fine for this, but we also need to ensure that there is easy access to the DP description linked from this access point. We also discussed deep stacking the N4945 SV data, which includes NB118, and examining band-merged catalogues across the Orion SV region to check for lower level artefacts and other gotchas. A further possiblity is to use the many repeat observations in the N4945 SV data to examine time series analysis problems in conjuntion with NJC. Tiling will come second, since it is more important to iterate quickly rather than wait for fully polished products. WFCAM meeting: MJI observed with WFCAM at UKIRT on 3rd-6th October and whilst there had a meeting with Andy and Chris to finalise the format and content of the MSBTid logs. These are now generated after the first stage nightly data processing and provide a more rapid feedback on MSB quality with respect to cloud cover, seeing and sky brightness, if these are specified by the PI. The Cl-range was clarified by defining 3 categories. As the nominal ZP varies by ~+/-2% due to a mixture of measuring and natural variation due to mirror cleaning cycles, these are now: <5% variation in the nominal ZP represents photometric; 5-20% is thin cirrus, and >20% represents thick cirrus. The MSBTid log also flags those MSBs rejected at the summit. The hopefully final iteration on the offset sky strategy was also discussed and a full set of recipes are now in place (JITTER_FLIP12, 13, and JITTER_SKY), and, as data subsequently taken shows, working. ADASS: MJI gave a talk at ADASS and JRL talked. MJI also noted that there is an upcoming UKIRT board meeting and that the submitted report is available in the usual place on the internal web pages. Likely attendance at the December Science with UKIDSS RAS meeting was also discussed. MJI will attempt to provide JRL with results from the new and improved bug-fixed deblended Petrosian et al. measures prior to the meeting to pre-empt expected criticism from the unusual suspects. >>>> 4. Data archives update - AAT, ING, WFCAM, UKIRT RWA reported on an email from a satisfied customer using the ING archives. A taxidermist has been contacted. All archives are up to date though JRL noted that he is still sorting old UKIRT CASS data header problems. 5. Optical/NIR processing - HAWKI, INT WFC, MegaCam, Subaru, ESO WFI EGS reported that he has been processing data from the ESO Spitzer Imaging Survey (P.I. Berta) obtained with WFI (ESO). This is a survey over 5.5 sq.deg. of the Spitzer/SWIRE S1 area obtained in B, V and R from 2002 to 2007 in 155 nights in total. Only the central part has been processed before and the idea is to process everything with the same pipeline used also to process the optical data in the rest of SWIRE (i.e. WFS and Mosaic I and II). MJI noted that it was time to get some mileage out of our processing of HAWKI SV data and asked STH if he could contact Mark McCaughrean and see what the status was. JRL commented that he was going to revisit the sky subtraction before we produce the next iteration of the catalogues. There was also some discussion about the usual radial distortion model that we generated to do the astrometry. Apparently this is newsworthy ! MJI will contact Mark Casali and see if it really is. <<<< RGM noted that there was some HAWKI PI data that he would like processing. <<<< EGS has taken over finishing off the EGAPS UVEX data products from 07B now that the U-band and HeI-band offset calibration has been sorted out. MJI noted that various people including Google Sky and MicroSoft are interested in IPHAS mosaics of the Galactic Plane. This is currently a no work on our part project, which is fine if it stays that way. Assorted new panoramic M31 survey data from MegaCam for 08B has arrived and is being processed and the first data with the new Hamamatsu deep depletion CCDs on SuprimeCam has been processed. These detectors are early production versions of those for HyperSuperimeCam and give a much improved i and z-band response whilst maintaining the good blue passband sensitivity. The cosmic- ray tracks are also much more entertaining. 6. WFCAM update The photometric calibration paper has been updated to incorporate the responses to the referee comments and has been resubmitted to MNRAS. MR has been beavering away running the nightly pipeline for WFCAM data as it comes in and is comfortably keeping up with the data flow. The raw data arrives and is checked, MEF'd and ingested within 24 hours of it being taken. The nightly processing is started shortly thereafter. Checking the products, fixing the parquet floor pattern and bundling up the monthly photometry updates takes the rest of the time. The new data transfer setup from JAC now allows us to offer a (limited) fast track processing for urgent data requests where residual channel offsets are unimportant and for which photometric calibration to 2% accuracy is sufficient. This option has already been trialled and can deliver data products to the PI within 48 hours of the data being observed. The service will be necessarily limited in scope and, after discussion with JAC, will probably only be offered through the PATT form via a tick box requiring additional justification for its use. The extra load imposed by this is expected to be marginal, but will be monitored and re-asssessed as necessary. Assorted reprocessing issues have cropped up or been stalled since our last get together. Three years downstream a long standing bug with estimated Petrosian radii and fluxes (also affecting Kron and Hall measures) for deblended objects has been reported to us. This affects all processed WFCAM data. Fortunately, due to the rather large number of flux measures in the catalogues, a fix has been found for this that can be run on the existing catalogues. Although the update can potentially be generated over a w/e run, progressing it all the way through the various data products in the middle of DR5 preparations did not seem like a good idea. This update will therefore be rolled out for DR6 with the added benefit of allowing much more time for testing. The UDS team came in with an unexpected request to reprocess all the UDS data for 05B as this semester currently has a high attrition rate due to background variations. A few tests on typical nights unfortunately confirmed that the latest "pawprint" sky estimation strategy, with iterative object masking, does appear to do a much better job. This reprocessing is scheduled after completion of 08b sometime mid-December. A much shorter request for reprocessing a limited amount of Orion data taken for U/05a/100 recently came in. The original sky estimation and subtraction for this was so bad that this was pushed through quickly. This data also provides an interesting test case for the nebulosity filter developed for GPS photometry in regions of high nebulosity. This leaves us with the extant GPS reprocessing in nebulous regions to improve both detection and photometry, also on hold due to lack of available effort. MJI commented that there still appeared to be a problem with scaling for nexp > 1 data. Although not irrepairable, this has been noted many time previously and he asked JRL to check with Brad at JAC to find out why this issue still persists. <<<< 7. VISTA update STH gave a brief report on his "observing" trip to VISTA and summarised the data he had taken whilst there. As a result we now have available a series of pointings (pawprints) in Y,J,H,Ks,NB118 on several 2MASS touchstone fields and a few assorted other science fields plus the corresponding series of twilight flats, darks and linearity sequences. Apart from a few problems due a TCS dither offset bug, this data has been processed with a preliminary Cambridge pipeline and has yielded some useful science products. A first pass analysis of these and some earlier data has provided a good first stab at the astrometric distortion, including the need for, and definition of, an r**5 term (PV2_5). It has also yielded estimates of the system zero-points in these passbands and will be used to check the overall system throughput, sky brightness in various bands and provide a first pass estimate of the colour equations with respect to 2MASS. STH was asked to look at this available VISTA data and further investigate this data. <<<< To make the data easier to digest JRL and STH will also sort out the object names of the various OBs according to their RA and Dec (all currently have the same name). <<<< MJI noted that all VIRCAM data products use the ESO HIERARCH system for recording processing information (seeing, sky level, noise etc.) and also use the "legal" table form of the WCS for recording the astrometry transform parameters in the catalogues. This is currently incompatible with all the WFCAM post-processing analysis software operating on the catalogues. As an interim solution this has been sidestepped by running a translating script to convert them to WFCAM style keyword names, but will need updating properly later on. JRL noted that the latest release v0.8.4 of the VIRCAM pipeline software and updated docs were delivered to ESO the other day. Ironically some requested CPL updates related to FITS header problems and efficient MEF file writing that have been incorporated by ESO recently, have caused problems with running the VISTA summit pipeline at Paranal due to them not being incorporated there yet. Apparently a USB disk with all the VISTA data on up to early November is en route to us via the correct channels, and currently is being ingested (slowly) into the ESO archive in Garching. It hasn't arrived here yet but watch this space. PSB reports that he has acquired a VISTA Trac account and also that against the odds network transfers from Paranal are making good progress. So we already have access to the new data on the USB disk before it got here. MJI reported that a sample of narrow band NB118 and J-band data has been made available to the UDS narrow-banders to enable them to check assorted properties. This had been previously cleared with JPE and WJS. 8. AOB MJI gave a brief update on the (lack of) news on the rolling grant application of 18 months ago. Various parties with vested interests in the data processing have been made aware of the issues and asked to press the case for some action. RWA has done a sterling job organising the APM and friends Christmas party, lunchtime 9th December, consequently a good afternoon for inspired software updates. Finally, MJI reported on the recent changes proposed, nay happening, in the IoA computer support group and the impact it will have on CASU. Continuing actions ------------------ STH check that Plone astrometry technical pages are correct JRL produce web page of HawkI publicity for CASU MJI finish off stage-I software release pages EGS STH make photometric calibration paper available on Plone and VDFS sites MJI finish off technical paper New actions ----------- PSB compare Calibration Plan with current expectation for calibration EGS check if the CASU presentation from the ESO Public Surveys meeting in September is available on ESO website. MJI pre-empt expected criticism re: deblended Petrosian fluxes STH contact Mark McCaughrean about HAWKI Eta Carina data status MJI contact Mark Casali about HawkI radial distortion RGM liaise with JRL about processing HAWKI PI data JRL contact Brad at JAC regarding the Nexp > 1 issue STH further investigate photometric features in available VISTA data (throughput, colour equations, sky brightness) JRL sort out project names of VISTA MSBs taken by STH STH