4th October 2007 Observatory Meeting Room 10:30-13:00 Present: JRL, STH, EGS, PSB, MR, MJI, DWE Apologies: NAW, RGM, RWA Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Recent meetings 4. Data archives update - AAT, ING, WFCAM, UKIRT 5. Optical processing update 6. WFCAM news - raw data transfer status - WFCAM network transfers status - processing status 07B - reprocessing status - photometric calibration status 7. VISTA news - general news - cold camera test data report - letter to PIs [and ESO] 8. ADASS 2007 - post mortem 9. AOB - retired cass36 - system management Minutes ------- 1. Actions from last meeting RGM have not completed the work on the classification aspect of cross-talk EGS weeding and classification generally - ongoing <<<< MJI MJI is putting the OMP log comments into the raw directories, though he commented he was still occassionally having problems with the mirrored OMP database, which Jim was in the process of fixing. JRL are still working out a scheme for doing header updates for ING archive EGS - ongoing <<<< MR found out when the first 07B tapes were shipped MR has contacted Jim Geach regarding comparing his processed data with pipeline output (see later) EGS has helped with the assessment of VIRCAM lab data MJI have found out about CASU system management from JMI PSB 2. Comments on WFAU minutes DWE liked the joke. STH asked if the weekly mails from Andy Adamson could be distributed round all of CASU. MJI noted that at the moment he forwarded these emails to JRL and MR and would forward them to anyone else who requested. <<<< 3. Recent meetings There was a VDMT meeting on 13th September. MJI and STH reported on the new timetable for VISTA presented at the VDMT meeting and the subsequent email sent out to PIs by JPE relating the new schedule. On the same day, several members of CASU held a meeting with DES representatives to discuss data processing issues for the new 3 sq deg FOV optical camera for the CTIO 4m. 4. Data archives update It was noted that AAT data was still sputtering in over the network, likewise ING data. Some complete nights of data for 07B had been received and were available in the WFCAM raw data archive. RWA sent in a report noting the following: The ING CD cabinets are being re-arranged to make room for those discs previously stored in SPO20 and including the AAT archive disks which are now all in jewel cases. The DAT tapes in SPO20 will be dumped into the skip next week leaving 3 large empty cupboards with sliding drawers. We need to decide if we have a future use for these for offline tape storage or whether we should dump them as well. Given the delay in starting the Kavli building construction this can be a leisurely decision. The corner cupboard in the Obs meeting room has had the lock changed and now stores backup DLT, LTO and other tapes. It is about half full at present and MR has a large batch of WFCAM tapes to go in later. Keys are held by RWA, MR and Mark Hurn. Requests for the oldest (~20 years old) ING data have almost dried up - 19 over the last 6 months and 4 in the last 4 months but they take longer to fill as the old system no longer works. 5. Optical processing update STH is processing INT WFC data for the Lund group of whom Daniel Aden is currently visiting the IoA. MJI noted that Megacam had an operational problem on June 17th destroying the i' filter and was off telescope and unavailable until July 27th - missing the i' filter, naturally. Several of the programmes MJI is involved in require the i' filter and are now on hold. MJI noted that representatives from Google Sky had contacted the IPHAS consortium and held a telecon about the possibility of using IPHAS data to make colour images of the Galactic Plane to serve through Google Sky, negotiations are ongoing. Gavin Ramsay from Armagh observatory has contacted us about the possibility of collaborating on a Galactic Plane variability project. A recent INT run will be processed and assessed to determine the feasibility of CASU doing the data processing, and possibly light curve generation, using software already developed for the Monitor project. 6. WFCAM news Tapes up to and including complete nights of data from 4th-13th August and 25th-26th August have been received and assembled into MEFS and the processing of these is almost complete. There is a delay with other intermediate nights due to a missing tape. This is hopefully about to be remedied and should enable us to catch up with the backlog. MJI noted that the backlog was alleviated somewhat by the fact that UKIRT was closed for the nights of 15th-17th and 29th August, and 1st-11th September due to weather and cold head problems respectively. He also reported that he had received good news about the channel-edge problem from Andy Adamsom as a result of Derek Ives' recent visit to JAC. MR pointed out that he had added a new column to the processed status web pages which directly linked to the CFH weather pages for each night. This was to encourage PIs to check the weather information before asking for reprocessing of not-so-good data. He and MJI have been comparing the pipeline processed data with some COSMOS fields processed from scratch by Jim Geach (Durham). This involved stacking the pipeline science products of two consecutive deep stack MSBs by us and then directly comparing with the same data (images only) processed "manually". This was done both by visual inspection and by generating catalogues from both sets of products and comparing depths, number of images and so on. Although Geach's processed data has flatter backgrounds the CASU version goes significantly deeper and the shapes of faint images are much "smoother". There is also a curious non-linearity in the difference of derived magnitudes which may be related to the rather "spiky" faint image appearance in the Geach version. Siddhartha Ghosh (Sussex) has been comparing his own methods of deriving bad pixel masks with the bad pixels implicitly flagged in the CASU-derived confidence maps. So far the comparison has revealed no problems. MJI gave a brief summary of the various odds and ends of reprocessing that have been occurring over the last few months. Apart from a UDS query/request that came in a few days ago and is being assessed, all of the reprocessing is now complete. When finished MJI will check the status of all the reprocessed UDS data <<<< STH asked for an update on the status of internet transfer of data from UKIRT. PSB reported that a Gbit/s fibre connection from summit down to sea level is being installed and that when this is completed tests of transferring data directly from the summit will proceed. This will include some procedure refinement and network transfer optimisation. We do not have a definitive timescale for this yet but are quietly optimistic about prospects. This should help mitigate against some of the current data transfer problems we are experiencing with tapes. PSB will contact Brad Cavanagh at JAC for an update. <<<< STH and MJI reported on the status of the WFCAM photometric calibration. Z and Y-band offsets (with respect to Vega) have been measured (using a robust maximum-likelihood fitting technique) from LAS and GCS datasets and found to be consistent with zero (within the errors of measuring it) for the Z-band and 0.08+/-0.01 for the Y-band. MJI has now implemented per-detector zeropoints in the calibration which removes a significant fraction of the apparent spatial systematics. These new calibrations have been applied to all processed WFCAM data (05a,05b,06a,06b,07a) on a monthly basis (following the master flat updates) and the resulting updated FITS header keywords have been made available to WFAU for incorporation in DR3. MJI discussed the systematic chip-to-chip offsets, most noticeable in the Z-band, which seem likely to arise from intrinsic differences in QE between the detectors and possible pedestal offsets in the flatfields. In addition, he has generated monthly tables and plots of the residual photometric spatial systematics for all broad bands. It is not yet clear what is causing these remaining ~1-2% systematics, but possible effects include: contributions from PSF distortions; scattered light illumination corrections; and residual non-linearity in the detector responses. These, and differences between the colour equations for each chip, will be investigated for future releases. STH and MJI continue to work on the accompanying paper describing this in more details and plan to circulate a draft within 2 weeks. MJI asked MR to investigate how to make the correction tables and systematics plots available on the processed status web pages to enable users to assess the remaining size of the problem and, if they want, have the wherewithall to implement a correction. <<<< 7. VISTA news MJI noted an email from JPE commenting on his discussions with ESO about the (successful) bid to get EC funding to lay a 1 Gbps optical fibre link between Paranal and Antofagasta. When installed this opens up the possibility of demonstrating its utility by transferring VISTA data to Garching and then to Cambridge. There was some discussion about the cold camera test data report compiled by JRL and PSB and the related upcoming telecon to discuss this. There was some worry expressed about the problems having two different (and incompatible) readout modes might engender. However, without further tests it is not clear if this will be the case. JRL is still working on further self-flatting tests on this VIRCAM data and will email a summary of points of note from VIRCAM tests to participants of the upcoming telecon <<<< JMS had sent an email around about another cold run at the end of October and asked for input on what data tests should ensue. PSB and JRL will compile a wish list and relay it. <<<< PSB noted that the VISTA QC development project is underway: MR is dealing with the data-ingestion DB (raw & processed) and EGS is working on the interactive visualisation tool. PSB is playing the role of user-in-chief and making the demands. EGS sent out a letter to PI's about the design of the VISTA QC pipeline/survey progress user interace based on the WFCAM version as a starting point. So far he has had two replies: one was positive noting that the WFCAM system was good already, The other was neutral in the sense that the PI planned to design and implement their own survey progress monitoring tool. There was further discussion concerning data quality for current WFCAM and planned VISTA processing, centering on how to identify MSBs/OBs as pass/fail on the basis of DQC criteria, and how to communicate these flags to PIs and survey managers i.e. closing the circle. (PIs will likely have stricter criteria based also upon their ability to do science with the data, whereas our criteria will be on the ability to process and calibrate the data as well as comparing target QC criteria with observed.) PSB reminded us all that we still need the filter transmission data for the NB118 and Z filters in order to finish off the pre-commissioning ETC technical details. 8. ADASS 2007 JRL presented a rundown of lessons learnt from organising the ADASS 2007 meeting. All in all it was a successful meeting and helped raise the profile of CASU in the ADASS community. There are still several loose-ends to tidy up. JRL needs to reacquire the helpdesk laptop to obtain the ADASS presentations and PDF'ify them, and also needs to email other contributors for PDF versions of their talks to post on a web site. <<<< RWA will be editor in chief of the proceedings when the contributions start to roll in, with help from JRL and MR on latex++ details. DWE raved on about the chocolate fountain and wants one permanently installed at the IoA. 9. AOB MJI noted that cass36, and all its disks, had finally been retired and indeed "has left the building". The disk contents (suitably relinked as well) are all still online elsewhere. DWE volunteered to do the system management updates for the windows machine that still gets occassional use. <<<< STH noted the upcoming UKIDSS meeting in December and suggested interested parties should register as soon as possible. MJI, MR and JRL already are. <<<< There is so much muck clogging up the air intakes on the computer kit that MJI and PSB volunteered to sort out buying a handheld vacuum to temporarily reverse the appropriate sucking polarity. <<<< PSB finally retired after 15 years as a starlink manager and now works full time on CASU projects. His first one is to investigate getting wireless installed in the APM building. <<<< Finally, congratulations to Marco and Francesca on the birth of Martino. And really finally, a mysteriously acquired bottle of champagne was consumed in honour of these latter two momentous events. Continuing Actions ------------------ RGM report back on classification aspect of cross-talk weeding EGS MJI JRL work out scheme for doing header updates for ING archive EGS New Actions ----------- MJI forward weekly email JAC reports from AJA to a wider audience MJI check status of the reprocessed UDS data PSB contact Brad Cavanagh at JAC for an update regarding the WFCAM network transfer MR figure out best to way to advertise illumination correction information JRL continue with self-flatting tests on VIRCAM data and email summary of points to note from VIRCAM tests to participants of telecon JRL compile wishlist for late October VIRCAM cold camera tests PSB JRL reacquire the helpdesk laptop and PDF'ify ADASS presentations DWE start duty of updating apm1 Windows machine PSB investigate getting wireless for the APM building MJI buy handheld vacuum for APM building PSB ALL consider attending UKIDSS meeting 17-19 December at ESO