8th August 2007 Virtual Meeting Room Present: STH, EGS, PSB, MR, MJI, RWA Apologies: DWE, JRL, NAW, RGM Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. News from meetings - CWG, VDMT 4. Data archives update - AAT, ING, WFCAM, UKIRT 5. Optical processing update 6. WFCAM news - raw data transfer status - processing status - photometric calibration update 7. VISTA news - pipeline progress - summit processing hardware - cold camera test data from lab and telescope 8. ADASS 2007 - latest 9. AOB Minutes ------- 1. Actions from last meeting RGM have not completed the work on the classification aspect of cross-talk EGS weeding and classification generally - ongoing <<<< MJI MR contacted the Durham group (Jim Geach) about their progress in processing raw WFCAM data and has obtained some example processed data. EGS has investigated the photometric redshifts issue in DXS after conversations with Michael Rowan Robinson and Steve Warren about this. MRR was comparing our WFCAM aperture magnitudes with 2MASS isophotal magnitudes from the extended catalogue and that is why they did not agree. When using the correct magnitudes the WFCAM and 2MASS apertures agree well. There is still the issue that when introducing the WFCAM magnitudes in the photoz code the specz and photoz show a large disagreement, but this is probably due to the fact that MRR has been using Petrosian magnitudes from the WFCAM catalogue. EGS has suggested using aperture magnitudes to measure fluxes with apertures the same size as used in in the optical. As a general rule the default should be specific radius aperture mags, worries about PSF variation corrections are probably 2nd order and are also qualitatively constrainable using PSF aperture correction information. PSB the missing NDR data has been transferred via FTP JRL have successfully carried out tests on a demonstration copy of the PSB Kognitio software using a minimal parallel CPU hardware setup MJI has still not put the OMP log comments into the processed directories - ongoing <<<< JRL are still working out a scheme for doing header updates for ING archive EGS - ongoing <<<< RWA asked Peter Weeks if blinds could be fitted to interconnecting annex but this was considered to be unworkable and an alternative was found (see later) JRL is in discussion with Chris Rammage about how best to proceed with advertising the AAO archive MJI asked Paul Aslin about the Carlsberg extinction payments and found out that it has been paid PSB emailed JPE regarding version 1.5pre1 of Calibration Plan PSB is moving office to APM building and will order another Gbit switch when the dust has settled - ongoing <<<< 2. Comments on WFAU minutes PSB was asked for his thoughts on WFCAM transfer problems (see wfau.minutes. 070622.txt). In response he asked for a (none-privileged) account on djedefre.roe.ac.uk, but there seems to be some difficulty setting this up. So ...... in order to follow up on CASU->ROE networking performance issues, PSB worked closely with Horst Meyerdierks to experiment with a JANET connection between CASU and ATC. HM installed seaforth.roe.ac.uk and carried out some requested software installation and tuning. Results were 25MB/s using hpn-ssh and 45MB/s using ftp (much more lightweight protocol). This compares very favourably with the best ~12MB/s using the multi-threading technique used originally by WFAU and the current rock-steady ~3MB/s over UKLight. Further tests are planned. 3. News from meetings There has been one VDMT meeting (27th July) via telecon since the last CASU meeting. Present were STH, PMW, JMS, NCH. Agenda items included: DR3 release schedule: NCH has not yet fixed a release date for DR3, contingent on the completion of the processing of NDR data from 06B. A release date some 6 weeks after this data is released to ROE was suggested. JPE asked that the Memorandum of Understanding from JAC be discussed with him before responding. JMS reported that agreement had been reached with ESO on the procurement of a single workstation to run the VISTA summit pipeline. This seems to have already been ordered and will consist of a PowerEdge 2950 with 2xCPU Quad-Core and 8GB RAM with 4x300 GB RAID5 of onboard data storage disks, plus a similar user offline workstation. Some concern was expressed over the modest data storage available on this machine, not to mention the limited amount of CPU. JMS was keen to learn how JRL was getting on with the VIRCAM data. Date of next meeting: Fri 7th Sept @ 14:00 by telecon, which MJI noted he cannot make since he is taking part in a thesis defense in Groningen then. 4. Data archives update The store room in the SPO containing our primary backup tapes (UKIRT, WFCAM AAT, ING) and plates is being gradually emptied. The backup tapes will be stored for the duration of the building works in the Observatory meeting room in a large lockable cupboard therein. ING: The duplicate CDs and DAT tapes are being skipped and should make enough room in the CASU storeroom to accommodate the remainder of the CDs still in Room 20 in the SPO. JRL and EGS are discussing moving the archive to the Sybase server on apm20. RWA noted that there are still a few more early tapes to be read and the conversion of those VMS Backup tapes read on the La Palma Alpha recently from STF to FITS awaits discussion with JRL. Another 100 tapes or so remain to be read on LP. AAO: The raw data in the AAO archive is now fully up to date. New nights of data are being transferred as they are made available. JRL is in discussion with Chris Ramage on how best to publicise the new interface. UKIRT: This is up-to-date. WFCAM: All 06B data, bar a few rogue files, have now been received and ingested. A WFCAM calibration meeting was held between MJI, STH, PH and SJW in Cambridge on Mon July 2nd. Discussion focussed on preparation for DR3 and completion of the analysis needed for the paper. A timescale was agreed for the paper (draft to be circulated end of August, with submission aimed for by end September), and STH has since circulated a draft outline for comment. MJI presented a summary of the WFCAM photometry spatial systematics, broken down into their components: varying pixel scale across the FoV (already accounted for); PSF variations as a function of focus, a ~1% effect for AperMag3 from 05B onwards and negligible for larger apertures. This can be readily measured from comparisons between small and large aperture photometry. Correction tables would need to be on a per science product frame basis and unclear if worth correcting for general use; detector-to-detector zeropoint error (to be included in DR3), this comprises about half the size of the current residual spatial systematics; residual residuals. No clear handle on what these relate to, but appear to be reasonably stable over semester-length timescales. Look up tables could be provided for optional use but need not be part of DR3. 5. Optical processing update IPHAS processing for June July almost complete. CASU are also still carrying out the extinction calculation derived from the Carlsberg Meridian Telescope data but as this requires only minimal effort it is ongoing. 6. WFCAM news All the 06B data is now present, except for a few files from 20061031 and 20061122, and has been ingested into the raw data archive. All UKIDSS data for 07A has been transferred to ESO and we are now transferring the 06B nights prior to 20th December. We are aiming to finish off the remaining 06B processing before 07B data arrives in earnest mid-August. NDR: All the NDR data was downloaded over the network in about a week. Some additional time was necessary because for a few nights not all the data had been copied due to some cases where the observing log Brad was using to dig out the NDR data did not match ours, but this was just a minor hiccup. To date, all the December data is processed plus a few nights of November (up to 27th). December nights looked quite reasonable but for the first tranche of November (counting backwards) the quality dropped significantly with loads of incomplete images and observations taken in very poor weather. Additionally we are having to use December flats and confidence maps also for November nights because the flats taken in the earlier period had a very bright star in the center of chip #1 and the dithering was not large enough to get rid of it (this was prior to changes in the flat observing that we discussed with JAC). UDS reprocessing: After reprocessing the UDS data for 20061227 and making it available to Sebastien and Omar, they came back about a week/10 days after saying that, although the re-reduction made the sky background smoother, there were some weird features (especially on chip #2) that Swarp could not cope with. The main feature is a darkish vertical stripe on the left half of chip #2 that extends almost from the bottom to the top of the image. The fact that it does not show any quadrant symmetry or channel aligment suggests that it is not caused by a glitch in the instrument electronics. After looking into this in detail, JRL and MR came to the conclusion that the feature is not introduced by a poor sky correction per se but is actually present in the data itself. It can be clearly seen, although at a lower level, in the sky frame as well. This, together with the fact that the feature is still present after the sky correction, means that the feature was varying in intensity (but not in position) during the night. JRL is investigating if it is possible to get around this problem by using a finer-grained sky correction for cases like this. However, this will require masking the real objects to avoid problems when the dithering size is not large enough to guarantee a proper object reject (like in the case of some of the UDS data) and may lead to problems with larger objects where no pixels are then left for sky estimation. GPS reprocessing: MR has reprocessed all the data in the list provided by Phil Lucas. Most of the problems were due to a known feature of the cross talk correction in regions of highly varying nebulosity that had already been fixed and did not affect any of 06B and 07A data. In some other cases there seemed to be problems with a "patchy sky background" or with "stars being subtracted". The reprocessing fixed the crosstalk problem satisfactorily. In most of the cases where the sky correction was not good the reason was simply due to poor weather conditions. The "subtracted stars" mystery was solved by looking at the 8 images for the 2-point dithering + 2x2 microstep sequences. In all cases at least one of the two microstep sequences had an image with a faulty detector showing all sorts of weird patterns. All the data is available from the WSA so we strongly encourage the PIs to have a harder look at the data when there appear to be problems and also take account the weather conditions (e.g. from http://www.cfht.hawaii.edu/cgi-bin/uncgi/elixir/skyprobe.pl?list) and the nightly summary QC plots before asking for reprocessing. External comparisons: Jim Geach has made available some example processed data for comparison with the pipeline output but this is on hold for now until 06B processed data has been kicked out of the door. 7. VISTA news Reports on the current likely VISTA timescales were received. ESO have settled on the workstation specification for the Paranal QC pipeline (briefly, a 2x4-core system) which, although a top-end workstation, is below the 16-cpu blade system suggested by the Garching pipeline benchmarks. A RAL team with JPE in support carried out a cold run on VIRCAM in July and despite typical commissioning difficulties obtained around 2300 data frames including EMC on-telescope tests and a series of lab tests including linearity sequences. Some near-time analysis was possible back in the UK by using the Paranal (2x2Mb/s) network link within agreed restricted limits (about 4% of the bandwidth), and data was also returned on portable disks by SMB and JPE. All data is now transferred to Cambridge for more detailed analysis and tests. 8. ADASS 2007 The oral programme has now been published on the ADASS website. The poster programme will appear in due course. Financial aid awards have been announced and Elinor is making the arrangements for the hotel rooms for the successful applicants. Ian Bell continues to deal with the credit card details. PSB has convinced Amanda to do the work for the conference photograph. RWA is making arrangements for the registration packs. Following the expected flurry of last-minute registrations and alterations, the numbers are around the break-even 300 mark (phew). The network (4x8Mb BT lines, secure wireless, guaranteed demo wired bandwidth) has been ordered. The initial program is now available and generally, the planning is well on track. 9. AOB MJI reported that the VDFS 6 month extension grant had finally been received. PSB noted that Fibre re-routing in preparation for demolition and subsequent building works is causing a series of short LAN outages. The opportunity is being taken to re-equip with a 10G-capable backbone fibre. Continuing Actions ------------------ RGM report back on classification aspect of cross-talk weeding EGS MJI MJI place OMP log comments into processed directories JRL work out scheme for doing header updates for ING archive EGS PSB order another Gbit switch for APM building