32nd March 2008 Hoyle Committee Room 10:00-12:15 Present: STH, PSB, MJI, DWE, JRL, MR Apologies: RWA, NAW, RGM, EGS Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Recent meetings/visits - Leiden UltraVista 4. Data archives update - AAT, ING, WFCAM, UKIRT - new archive server system 5. Optical processing update- VST update - revised IDR IPHAS paper 6. WFCAM update - photometric calibration - ftp and tape transfers - processing status 08A 7. VISTA update - linearity analysis doc - filters, and filter responses NB118 and NB++ - new SADT release - new calibration plan, lab tests, commissioning - pipeline progress 8. ADASS 2007 9. AOB - STFC programmatic review Minutes ------- 1. Actions from last meeting STH has checked the advertized illumination correction information using LAS for overlap tests and found that the same correction is required for both stars and galaxies whatever the magnitude. MJI reckons this "proves" it is a genuine illumination correction but JRL bet MJI a pint (is that all!) that these were really a reset anomaly problem in disguise. BET< ALL some have provided comments on the photometric calibration paper. A final set of comments were requested before the paper gets submitted - ongoing <<<< JRL is halfway through checking why the pipeline catalogue software and toolkit give different results in a few specific cases - ongoing <<<< MJI checked that the Plone technical pages (WCS howto) were correct. Further discussion led to uncertainty whether the Plone astrometry technical pages were correct. HTML errors were also being reported. http://casu.ast.cam.ac.uk/surveys-projects/wfcam/technical/astrometry eg. for ZPN, is PV2 being used rather than PROJP? STH was tasked with finshing this off. MJI will supply the relevant email <<<< MJI has asked JPE for the transmission curves for VIRCAM Z and N118 filters, but has not received them yet. These are needed to complete the ETC tech spec deliverables and finish off the prototype ETC - ongoing <<<< JRL has backed up the UKIRT cass archive MJI has not checked what SDSS do about elliptical apertures - ongoing <<<< PSB has sent off a wishlist of observations for the next cold VIRCAM run. RGM have acquired most of the versions of the SMPs for the VISTA MJI public surveys but are unsure of the status (ie. are they the final version ?) of some of them - ongoing <<<< PSB has investigated the new colour printer options and will conduct further trials when the IoA has acquired its latest machines. 2. Comments on WFAU minutes No substantive comments though JRL noted the WFAU preparation of a response to the STFC programmatic review. 3. Recent meetings/visits - Leiden UltraVista MJI and JRL part attended this meeting and gave presentations on the plans for processing VISTA data. Several issues came up during discussion not the least of which were an understandable desire on the part of the UltraVista team to have access to VIRCAM bad pixel maps to decide on the detail of their dither offset strategy (a series of "random" offsets within a 30 arcsec x 30 arcsec box). MJI commented that for any reasonable dither pattern within an OB, image resampling during stacking was inevitable and that CASU currently planned to offer three alternative interpolation schemes: nearest-neighbour; "dribble" a variant of drizzling; and an optimal cubic MOMS (eg. Blu, Thevenaz & Unser 2001) - subject to further testing and feedback. This also raises the issue of the correct strategy to adopt during computing accurate dither offsets from the on-sky data. Because of the large radial distortion overall delta_TPA and _TPDs for the entire array are required and will be implemented in the pipeline stacking (the WFCAM pipeline currently uses pixel offsets within a dither stack). One further important concern raised was the overall transfer speed of raw data from Paranal to Cambridge via Garching, which might take up to a month. Improvements at the Paranal end with a dedicated G-bit fibre connection will eventually offer possible alternative transport schemes but not for the first year or so of operation. The usual alternative chestnuts to the 12 x 250Gbyte disk system proposed by ESO were aired but it is unclear at this stage how to progress further discussion of this. MJI noted that all the talks for the UKIDSS public survey meeting held in Garching in December 2007 were now available online at http://wiki.astrogrid.org/bin/view/UKIDSS/Dec07Workshop. STH commented that a planned VDMT meeting in March failed to materialise but that he would no doubt be in touch later for updates for its future reappearance. 4. Data archives update - AAT, ING, WFCAM, UKIRT - new archive server system JRL reported that the entire AAT archive has been backed up and sorted onto one disk system. In the process he discovered some duplication in the AAT data and has resolved that. The UKIRT CASS archive is still backing up even as we speak and has also been rearranged onto one disk system. ING transfers continue as normal and data up to the end of February has been checked. March data is awaiting a similar treat. WFACM raw data updates for 08A are paused awaiting completion of installation of the new archive server system, as are ESO transfers. MJI clarified for himself that apm25/apm26 will now be used solely for the CASU plone web site, QC DB and survey progress pages. The IPHAS IDR DB will be migrated to the new server. PSB and JRL noted that the new archive server system (apm5) has been installed and is now in the shakedown phase. When verification is complete this will become the live interface replacing the current system. 5. Optical processing update- VST update - revised IDR IPHAS paper MJI said that he had been informed that "the VST telescope is being integrated on Paranal and this phase should be completed this April. Primary and secondary mirror support are still under revision, and INAF foresees the final integration in the telescope structure on Paranal starting end 2008 and completion by early 2009". So, we may see some VST survey data in 2009 then. EGS has revised the IPHAS IDR paper in light of the referee comments and this has now been resubmitted to MNRAS. MJI noted that he had seen some impressive on-the-fly demos of cross-matching of other large catalogues with the IPHAS IDR using the Astrogrid toolkit. PSB commented that all you needed was an EGS to construct the hieroglyphs. 6. WFCAM update 07B status: all of the remaining 07B raw data had been transferred across, processed, checked, and copied up by WAFU. The reprocessed UDS data from 06B is also now in its final resting place. Photometric calibration: a close to final draft of the photometric calibration paper has been generated. Part of the changes relate to the tests noted earlier on the illumination correction. This has been circulated around the authors for comment. Acquisition of raw WFCAM data: PSB reported that after some set up and debugging, automatic copying of data from UKIRT is working routinely. Each night's data is placed on a JAC summit server as 1-chip Rice compressed FITS and a flag set; a cron job in Cambridge scans the flag and mirrors the data. Transfer speed is around 8 MByte/s with a night's data taking typically two hours to transfer. This currently uses a 32 thread transfer protocol, not ideal since it clobbers the server end more than necessary, but will do for now in lieu of further server system tuning. At the time of writing (2nd April) WFCAM data up to and including 31st March have been transferred this way bar the first few weeks of February. These eventually arrived on tape mid-March (well almost, data for one detector for 20080221 arrived via ftp last night). Data still arrives on a per detector basis and is checked and factored into MEFs as usual. Since all WFCAM data from now on should arrive via ftp we have instigated writing our own archive tapes (currently using LTO2s) from the MEF'd raw 08A data as an off-line backup. Previously we just saved the incoming tapes. 08A processing: the sudden influx of 08A data was timed perfectly to coincide with the Easter break but processing has now commenced. MJI commented that the parquet floor pattern was still present, at least in the twilight sky flats, necessitating a cunning strategy to produce master flats. Proto- master flats are created in the usual way and are then flatfielded using previous good master flats as a reference but sans gain correction. The parquet floor pattern on #2 and #3 then shows up clearly and is removed with the usual bespoke software. Flatfielding^{-1) the output of this process, also sans gain correction, with the reference flat then restores the parquet floor-free new master twilight flats. 08A master flats for the February-March period for Z,Y,J,H,K,1-0S and BGamma have been produced this circuitous way. Software issues: JRL said that he would revive the changes log for the WFCAM pipeline which seems to have been terminated with the recent changes to the internal plone web pages <<<< MJI asked JRL to close the loop on the discussion related to offset sky strategies and pipeline requirements for keywords, before we encounter 08A data with this feature implemented. <<<< EGS noted that he has been asked by people at the IAC, Tenerife for a copy of all the UKIDSS processed imaging data. He has asked them to contact WFAU with their request since this is the proper access channel. This reminded MJI of a recent email response from JAC which indicated that all processed WFCAM data would be subject to the same proprietary periods as raw data. This includes UK PI as well as survey and campaign data, plus Japanese PI data and possibly (TBC) UofH PI data. This raises an interesting question: while the provenance of the raw data is relatively straightforward, who owns the "copyright" of the processed data ? and what does processed mean in this context, flat files output by the pipeline ? + extras like band-merged products ? As a bit of light relief MR and STH presented some interesting results from their investigations of long term trends (spatial and temporal) in the filter zero-points and the night sky brightness. This is being worked into MR's sky properties paper, a draft of which will be forthcoming later, honest. 7. VISTA update Filters: PSB noted we are still waiting to acquire the NB118 & Z filter transmission data to finish off the ETC technical specification deliverables and complete the prototype ETC. The issue of the on-sky calibration, including illumination correction, of the NB118 filter was raised at the UltraVista meeting in Leiden. This is complicated by the variable bandpass of the filter over the field of view - the wavelength shift is a significant fraction of the filter FWHM. The same worries apply to other, more exotic, narrow-band filters that have been suggested. MJI posed the question of who is responsible for accurate (flux) calibration of specialist NB filter data, since experience with equivalent NB optical calibration shows it is a non-trivial task. The recent release of the SAD Tool and related documents was discussed; unfortunately we did not have a chance to contribute or comment before the deadline. It was noted that SMB had made a number of detailed technical comments. There is still some concern about how an investigator chooses the value for the RECIPE keyword (essential for offset sky reductions and tile/ pawprint stacks resampling/interpolation). We look forward to an end-to-end test from SADT to FITS file to see what comes out. A forgotten action from the last minutes was to download the latest SADT and see what tiling patterns etc. are generated. STH was volunteered to look at this since he reviewed earlier versions. <<<< Oliver Twist: the need for more prep-room data was discussed. There has been no feedback following the previous request which accompanied the updated Calibration Plan in mid-February. Issues to be analysed include filter repeatability and/or tartan patterning, linearity and/or persistence, production of high-fidelity bad-pixel maps ...... usw. Given the amount of discussion the recently released linearity report generated, the sooner we get some more test data the better. Looking further ahead, it is hoped that now M1 is safely in the enclosure, a firm commissioning schedule will become available, this is needed by CASU to plan development and data handling. PSB wondered if it might be possible to beg more bandwidth from Paranal during commissioning, given the minimal impact the 2-hour 2Mb/s window used last summer had. MJI will attempt to find out what the schedule is and in particular what the commissioning plans are <<<< Software: JRL commented that there have been no major software updates just a few bug (surely not) fixes. Having warmed up, JRL then noted that in processing some of the SV data from HawkI on the VLT he had espied OBs taken with offset skies included in them. Naturally, he was curious why this capability was ruled out for VIRCAM OBs. EGS has released a preview (flashy! pas moi) version of the survey progress and QC tool which addresses several comments received. The current version is still active but will be superceded by the new one. 8. ADASS 2007 The finances for the event have been signed, sealed,,and profit delivered. There is a still a small amount of editorial effort required before the tome gets kicked out of the door but at least it should be close to being shifted before the next ADASS announcement goes out. 9. AOB MJI filled in details of what we knew so far about the outcome of the STFC programmatic review and potential meanings of various official statements. He also went through the grant review feedback in an attempt to second guess where things were heading. Construction work on the partition in the APM building main room has begun. In theory this should not be too disruptive since it is basically a carpentry and glazing job, but we will see. It was recently spotted that there is no overtemperature alarm in the APM building (other than the ultimate ones in the ceiling), unlike the rest of the critical computer rooms in the IoA. We will see if collective digits can be removed to expedite installing one or more. JRL volunteered to host a spring barbeque. His prefered date would be a Sunday. He will propose a few alternative dates for us to consider. <<<< Pete will supply (or was that drink) the beer. Continuing actions ------------------ ALL provide comments on latest draft of photometric calibration paper JRL finish off checking why pipeline catalogue software and toolkit give different results in a few specific cases MJI acquire transmission curves for VIRCAM Z and N118 filters MJI check what SDSS do about elliptical apertures MJI finish off checking and acquiring the latest SMPs RGM New actions ----------- STH check that Plone astrometry technical pages are correct JRL revive changes log web page for WFCAM pipeline JRL check that pipeline can cope with offset sky strategy STH download, install and investigate new SADT MJI find out latest VISTA schedule including commissioning plans JRL suggest a few dates for proposed spring barbeque