Friday 19th June 2009 APM Meeting tables 11:30-13:00 Present: MJI, STH, SC, RWA, NAW, MR, EGS, RGM, JRL(later) Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Recent and upcoming meetings 4. Data archives update - AAT, ING, WFCAM, UKIRT, VISTA 5. Optical/NIR processing - HAWKI, INT WFC, MegaCam, Subaru, Gemini, VIMOS 6. WFCAM update 7. VISTA update 8. AOB Minutes ------- 1. Actions from the last meeting EGS check PLONE pages for poorly aligned text and other faux pas. Also RWA need to populate the VISTA technical pages using material that JRL has recently generated - ongoing <<<< JRL HAWK-I web page: stalled and dropped from actions. JRL has been sidetracked by processing PI narrow-band photometric data of galaxy clusters for RGM. MJI stage I software release - still ongoing. MJI suggested we put JRL's catalogue software on it and EGS would add his program for converting binary tables as well. <<<< MJI technical paper - still ongoing <<<< MJI utility to convert binary tables plus corrections - ongoing <<<< EGS UVEX support - program written but not yet sent <<<< NAW VIZIER <-> IPHAS, NAW said he thought this might better be done verbally, and in any case it could be safely left until August - removed from actions STH ASTEP web pages is ongoing but will be integrated into EU-funded transient survey position - removed from actions MR missing FITS headers for 20090312 and cf. with original SDF files - a bit long in the tooth to worry about (was worked around at the time and hasn't occurred since) MJI produced VISTA QC summary info for web pages and in the process first pass CEs and Zpts MJI was still waiting for the VISTA I/O disks to be "returned", now waiting for one of the air conditioning units to be fixed <<<< JRL ESO have no intention of using Blu-Ray disk transfer to CASU, just an eclectic selection of hard disks JRL producing dummy tile headers: JRL curved-the-ball and instead "sent" a night of processed VISTA data to Eckhard at WFAU for them to trial indigestion. After a bit of head banging with MJI decided no new header items needed for tile products other than the usual inherited blurb and the usual progeny info. The default tile WCS will be TAN projection since the tiles have to be resampled anyway. SC wall-mounted monitor on lecture theatre wall cost £1800 for 1920x1080 pixels. Back to original plan of large LCD TV then. NAW suggested taking a laptop to Curry's and plugging it in to test, or was that going for a curry with his laptop. MJI has already tested TV/laptop at home and aka laptop aspect ratio being a bit different to TV, it worked fine. 2. Comments on WFAU minutes MJI noted that DR6 visual checking (up to end of 08B) is supposed to be done by 21st June but he (and others) were unsure of the proposed release date. MJI will attempt to find out, though in part this is still awaiting the <<<< Petrosian et al. fix for 05A -> 08A from CASU. RGM pointed out that the WFAU list-driven photometry extinction correction was being applied in the wrong sense but had now been fixed. 3. Recent and upcoming meetings MJI gave a talk at the UKIRT board meeting on May 22 and after the +ve's pointed out the big -ve, viz problems being caused by the constipated progress on the rolling grant leading to personnel shortage and zero funds available for equipment and consumables. The current disk storage space for raw and processed WFCAM data will last another ~two months and after that nada. This was noted by Simon Berry and "they" are fully aware of the situation. Of other note was an accelerated plan for LAS observations presented at the meeting originally aiming for completion by the end of 09B, the poor weather at the start of 09A notwithstanding. STH commented that from his recent observing run he noted that some LAS overheads were due to problems with lack of suitable guide stars. The VISTA IOT telecon for the end of May was rescheduled to the beginning of June and then cancelled. The next one is due June 30th. Minutes of the previous telecons are available on the internal web pages. NAW gave a talk at the Belfast meeting about PLATO data flow including the integration of ground-based follow-up of science data. CASU involvement in this is currently at a low level but may ramp up if the mission clears the many hurdles en route to an (optimistic) launch date at the end of 2017. There is an EGAPS galactic plane consortium meeting in Manchester, September 11th. NAW, MJI, EGS and possibly JRL will attend. The UKIRT workshop at ROE is from September 14-16 with the deadline for submissions in mid-July. STH has been invited to talk about his WFCAM transient survey. It is not clear if any talk on pipeline processing is needed here since the focus for WFCAM will probably be on science results. JRL has registered for ADASS and noted should put together an abstract on VISTA processing, which will be due July 15th. <<<< NAW said there is a GREAT meeting in Nice from November 19-20 and another small WG meeting about ground-based wide-field spectroscopic requirements at the IoA July 7-8. 4. Data archives AAT: data now transferred daily ING: flows apace daily as does WFCAM: see usual web pages UKIRT: JRL is still investigating UKIRT tapes that won't read. AJA has said that we could FTP any missing datasets from UKIRT. A list of missing datasets is being compiled over the next week and will be sent when done. 5. Optical/NIR processing : HAWKI, INT WFC, MegaCam, Subaru, Gemini, VST HAWK-I: JRL has reduced two more datasets for RGM - a PI program of narrow-band photometry of galaxy clusters. Re-photometry of the Carina data has been done by MJI who has received mostly complaints for his pains due to the lack of advanced calibration data (non-linearity, illumination corrections, detector QE diffrences etc....) and adequate sky correction frames. These all contribute to slight (and it is!) positional dependent effects when making rather speculative and optimistic cuts to select subsets of objects. STH reports no further progress with zoomifying. INT WFC: EGS has been processing some INT WFC data for Johan Knapen, with more to come. MJI recently reduced some INT WFC data for Vasily Belokurov as part of their MW substructure search. He also noted that he has done some nebulosity filtering tests of IPHAS galactic plane data, but is still awaiting more extensive feedback from U. of Herts folk. EGS has also reduced a few service nights for Alberto Fernandez Soto - a project to do with UVRHalpha imaging and spectroscopy of elliptical galaxies to identify satellites. EGS will be involved as a co-author on any papers coming out of this. VST: MJI noted that the Halpha filter for the VST GP survey VPHAS+ has been received in Munich and will be tested by Robert Greimel using ESO kit. 6. WFCAM update. MR has further automated the WFCAM raw data ingestion and verification, which, as noted before, is ready and assembled to go roughly 24 hours after the night's observing. Running the pipeline has now been streamlined as much as possible and beginning the processing sequence for each night now occurs a few hours after the automatic email notification of new data arriving. JRL commented that there have been no notable changes to the pipeline - phew! MJI noted that processing of 09A data is bang on track. Two recent nights were wiped out by a top-end problem which was fixed pronto. We are still awaiting non-linearity sequences, particularly at low count levels. STH and MR noted that there was still an issue with illumination of the new dome screen that was causing this delay. WFAU transfers of 09A are up to date. MJI has been in touch with WFAU about reprocessed data. The current status is: a) GPS nebulosity fields - all done and dusted ready for transfer; b) UDS K-band - after much pootling around all done, dusted and ready to go pootling = unique version no. in FITS PHU headers renamed sky files since potential conflict with original updated w* files to reflect these new sky file names added SKYALGO keyword to the PHU of the sky files updated photom calibration with usual detector level updates TBD after WFAU transfer confirmed, replace the original versions of these files and copy over renamed sky files; c) Petrosian et al. fix for 05A - 08A next up when MJI hands back pipeline processing to MR at end of month; d) UDS J,H - same treatment as K-band, eventually. The need for filter flushes was batted back and forth a month or so ago and some specific data taken to test this for GPS and LAS surveys. This is all processed and awaiting someone to eyeball the results. <<<< The dribbling graphic presented at the UKIRT Board meeting has been added to technical web pages to enable users to dribble for themsevles. 7. VISTA update RWA said data is now coming from Garching on blackberry-sized 500 GByte disks, with an eclectic variety of USB connection requirements, and still in oversized boxes. He has contacted Garching to see if the volume (and hence transport cost) can be reduced. STH spent 3 weeks at VISTA contributing to the commissioning effort. Noteworthy measurements made include (but not limited to): + several linearity sequences + flatfields in all bands + very deep J-band field (total of 16 hours) to test stacking -v- depth + a number of system image quality (SIQ) tests + lots of touchstone fields for photometric calibration tests + touchstone field stepped across the array to test illumination correction + the VISTA throughput (which is extremely good) using measured values of zero points derived by MJI from data taken late last year before the Z-filter was available + a few tiles in a variety of regions, to enable us to test tiling software. + use of the VISTA DIMM and ESO DIMM archive to measure the site seeing, but as these figures were often larger than the measured VISTA FWHMs we can only conclude that VISTA seeing sometimes involves imaginary numbers Much of the processing and first pass analysis of these will be done here starting with STH's planning to focus (sic!) initially on the SIQ across the entire array for data taken in the very best conditions. JRL sent the following report and then arrived in time to read it himself: I have continued to make bug fixes to the summit pipeline as I find them. The science pipeline is now almost complete and is being debugged. There are still some notable omissions, (no tiling yet, WCS isn't yet back propagated to lower level data products) but I feel I can get this sorted by the end of June. There are still tickets in the ESO system regarding the summit pipeline that I have not yet addressed. These are mostly comments and don't require any update to the software. Two data discs with approximately a fortnight's data have arrived. The second only arrived yesterday so I haven't had a chance to look at it yet. Most of the data for the first disk concerns verification tests being done at the telescope. I did an initial reduction of the first weeks data as though they were science nights. MJI has discovered a few major problems which have now been rectified and I will re-reduce those nights this weekend. Once that is done MJI can have another look. We have done a great deal of recent work to help with the verification tests for VISTA: Linearity - linearity sequences were taken in all the filters. These were analysed to create linearity curves for each detector for all filters. This turned up a problem with a lack of stability in the dome lights. One of the linearity curves worked out reasonably well and will be used for the reduction of the current data. Read noise and Gain - data from the linearity sequences were reused to work out estimates of gain and readout noise for each of the detectors. The lack of stability of the dome light source does make these estimates a little unstable. In order to test for excess read noise induced by the telescope, dome tracking, autoguider etc, dark observations were done of varying exposure times during the day when most of the telescope systems were shut down and during the evening when everything was running. Noise estimates were done on both sets of data which confirm that these systems have no net effect on the measure read noise. Night sky data are being used to test the background variance as a function of mean flux. This work is ongoing. Offset pointing - tests for the accuracy of offsetting were done by a series of observations with increasingly large jitter offsets. The jitter offsets as reported in the headers were compared with the true offsets determined by the stellar positions. The difference between the expected and true offsets showed that the alignment between the detectors and the autoguider CCD was not well determined. We have subsequently been sent raw autoguider CCD images for which we have worked out high quality world coordinate systems which should cure the problem. LOWFS world coordinates - similar work is being done on LOWFS images to determine the world coordinate system for each of these detectors. Scattered Light - observations at varying distances from the full moon have been done to determine whether science observations are likely to be affected badly by scattered light. The analysis of these data continues, although the effects can easily be seen in the summary QC information. We eagerly await the arrival of more of the data for further tests. Vignetting - a great deal of work has been done to try and directly measure the vignetting profile of the field of view. Unfortunately it has been impossible to decouple the large scale variation in the detector response from any real vignetting profile in the data. JRL also noted that missing CRVALs are still a problem in the recent data and that he has raised the matter again (this was supposedly a closed ticket). Oh .... and in his spare spare time he helped draft part of the VPO input to the June ESO Council meeting. The VISTA pipeline progress web pages, including raw data ingest and verification are now live and operational and available on the CASU plone site. Whilst on this topic MJI reiterated that a lot of VISTA technical information now exists and needs coercing to plonk on plone website. There is also a Paranal QC info web page with plots of everything you could think of, plus some others, derived from the summit pipeline QC output. The VDFS version of the ETC seems to have stopped working sometime in November. This version was a strange hybrid of Perl, C -> perl modules and a mixture of solaris and linux. The software components of this have long been delivered to ESO and maintaining the ETC is now their responsibility. The CASU version from November was painstaking to update and maintain so MJI has revisited the prototype purely Perl (PPP) ETC version and updated it with the relevant data files. This has now replaced the earlier hybrid version and is available from the same web address. This still requires a more thorough checking and an even more thorough comparison with real VISTA data. STH will begin this process by comparing throughputs and other aspects with his hard-earned hand-crafted numbers. <<<< 8. AOB SC talked about Network Area storage options - in particular an Open Solaris system with a Debian front end plus GUI. This involves a licence fee and a maintenance fee with the University possibly paying the latter. Drawbacks - the hardware and software come from different companies and it is too expensive. Apart from that ......... SC has also been looking at computing resources for STH's WTS project and options for next stage GAIA kit. MJI reported on recent (and ongoing) problems with the air conditioning in the large computer room. He could write a Soap based on the saga. At the very least this reminded him about the promised high temperature alarm which will alert via the security system. Let's see if it comes to fruition this time. Sorting out the VISTA i/o disks awaits the restoration of full air conditioning since much of the GAIA and VISTA kit is currently powered down. Continuing actions ------------------ RWA continue checking through the technical pages looking for glitches et al. EGS EGS add catalogue software and binary table conversion software to software MJI release pages MJI finish off technical paper updates MJI write utility to convert binary tables with all corrections EGS send off example code to UVEX team MJI sort out VISTA i/o disks New actions ----------- MJI find out anticipated release date for DR6 JRL put together and submit ADASS abstract on VISTA processing by July 15th. MJI make sure the relevant WFCAM filter flushing test data is made available STH compare fact with ETC fiction MJI