23rd June 2008 APM comfy chairs 11:30-13:05 Present: RWA, MJI, JRL, PSB, MR, DWE, STH, RGM Apologies: EGS, NAW Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Recent meetings - Video meeting U. Herts - Plato meeting ESTEC - LSST meeting Edinburgh - CACC meeting - UKIRT board meeting 4. Data archives update - AAT, ING, WFCAM, UKIRT 5. Optical/NIR processing - HawkI - IPHAS/UVEX - MegaCam/CFHTLS photometry 6. WFCAM update - illum corrs yet more tests - photometric calibration paper - data transfers - flippin' jitters - processing status 08A - DR4 checking - flagging rejected MSBs 7. VISTA update - general news - SADT + P2PP workshop at ESO - pipeline progress (random jitters) - data connections to ESO; ftp tests - JPE visit and Paranal camera data - survey progress web page requirements 8. AOB - network upgrades - hardware - APM air con etc.. Minutes ------- 1. Actions from last meeting STH has partially checked that the Plone astrometry technical pages are correct - ongoing <<<< MJI acquired the transmission curve for the VIRCAM Z filters. He is still trying to obtain the N118 transmission curve - ongoing <<<< MJI finished off checking and acquiring the latest SMPs RGM JRL postponed reviving the changes log web page for the WFCAM pipeline until the WFCAM data had been reduced - ongoing <<<< STH has dropped the action to investigate the new SADT since there will now be an ESO workshop covering use of this for VISTA/VST public surveys. MJI has obtained a GANTT chart of the latest VISTA integration schedule. He will continue trying to acquire more information about the NIR camera commissioning plans. <<<< STH completed the illumination correction tests in conjunction with WFAU MJI STH has obtained the graphic showing the size of the photometric radial distortion correction for the calibration paper, but has not added this to the Plone technical pages - ongoing <<<< MJI did not directly find out at the CACC meeting what the current network costing model is, but checked after the event and found that it hasn't changed much. MJI after a bit of discussion about location and format suitable meeting tables for the APM building will be shortly ordered DWE found out from Andy Batey that the wall display screen would be far more expensive than the simple expedient of buying a flat-screen LCD TV. Part of this was the "need" to have wall display screen installed on the wall by the Company. Plan B is to buy a TV with PC interface and have it free-standing on the end of the table against the wall. MJI will check compatibility with his recently acquired home equivalent to see if displaying from the laptop works well. He will also measure/check the dimensions of a suitable model. <<<< 2. Comments on WFAU minutes Although there were a lot of WFAU minutes since the last CASU meeting there were no substantive comments apart from further discussion of assorted photometric corrections. 3. Recent meetings Video meeting U. Herts: MJI, EGS and RGM attended. JPE was reminded of our desire to transfer data from Garching to Cambridge via ftp rather than NGAST disks. JPE presented a summary of VISTA progress including a ppt summary slide of the schedule, which would be useful to have. <<<< Video requirements are similar to those of UltraVISTA. MJI noted that EGS had signed up within the consortium to do the deep stacking, starting from OB pipeline output that has passed further stringent QC checking at U. of Herts. Plato meeting ESTEC: STH and NAW attended a one-day meeting entitled 'the PLATO payload consortium: kick-off meeting', held at ESTEC on June 2nd. Both felt that CASU could make a meaningful contribution to the UK bid for involvement in the mission, and NAW has submitted text to the UK SoI, PI Don Pollacco. LSST meeting Edinburgh: RGM attended this one day meeting and reported on the discussions for UK/ European access to LSST data. CACC meeting: Nothing much of note relevant to CASU activities other than news about network charging models gleaned afterwards. UKIRT board meeting: MJI reported that he had given the usual upbeat WFCAM processing talk. Ross Collins gave a similar one for WFAU and Steve Warren for UKIDSS. The interesting snippet gleaned was that the official Board policy is that the proprietary period for ALL processed data is the same as for the raw data, ie. 12 months after end of semester in which data was taken it is World accessible - if M. World wants it. He also noted that before the Board was the proposal to allow UoH access to UKIDSS data in exchange for 50% of UoH time, ~24 nights/yr. 4. Data archives update - AAT, ING, WFCAM, UKIRT The last shipment of Schmidt survey plates was loaded on Friday morning and transported to Germany the same day, along with most of the custom-built wooden shelves. It now remains to organise the 1000+ assorted objective-prism and prime-focus plates still in CDM's office in the SPO. Those plates relevant to Palomar will return there and the remainder will go to the plate archive in the eastern USA. The plate store in the SPO is now empty. AAT: A recently transferred batch of data has been ingested and the archive is up to date. ING: The archive is up to date and slurping in ever more INT and WHT data via ftp transfers. WFCAM: The last batch of 08A data (16th June) was received via ftp and ingested a few days ago making the raw data archive complete for 08A. Transfers of UKIDSS and CAL data to ESO proceed apace with about 20 nights of 08A to go. UKIRT: JRL is fixing a few minor foibles that slipped through the cracks in conversion from SDF to FITS. 5. Optical/NIR processing IPHAS/UVEX: Most of the data up to end of 07B (Jan 2008) has now been processed and first pass calibrated. Two more runs in 07B for UVEX need to be finished off. Final band-merging of UVEX data is awaiting modified bootstrap zero-point offsets for the HeI and u' filters - to be further discussed at the upcoming EGAPS meeting later this week. The proofs of the IPHAS paper have been received and corrections sent off. HAWKI: JRL has processed the HAWKI Carina and GOODS SV fields. He will investigate making a large area PR image, while STH will focus on improving the quality of the photometric calibration by deriving more accurate colour equations. MJI will investigate regenerating the catalogues for the Carina data to take better account of the considerable nebulosity in the region. The results <<<< from this and the GOODS (image, calibrations, catalogues) will be advertised via a web page. <<<< MegaCam/CFHTLS photometry: MJI noted that assorted problems he and Rodrigo Ibata had been having with MegaCam data (mainly centre-to-edge effects at the ~3% level) were apparently due to incorrect pixel-level illumination corrections computed and applied by the CFH Elixir pipeline. This problem probably affects all MegaCam data. To correct properly requires starting from scratch since the problem occurs at the (super)flatfielding stage and varies with each new flat used. A salutory warning against applying a correction-too-far. INT WFC data: RGM has a summer student (Tom Young) working on calibration of old INT WFC survey data. This has required (and will require some further) updating of the pipeline toolkit on the Solaris cluster which, due to lack of much use, had fallen well behind. <<<< 6. WFCAM update Data transfers via ftp have been extremely successful and painless (at least at our end). As noted previously this has generally resulted in observed nights of raw WFCAM data being available on disk here roughly 24 hours after the data is taken. It is checked, ingested and read to go for processing roughly another 24 hours later. Standard processing has been running about a week further back and complete processing to fix parquet floor problems and a brief spell of CFITSIO compression problems (too long a saga to relate here) plus all the QC checks, including monthly photometry updates, is the current bottleneck. Several more tests on the illumination corrections have been completed with good results. These were applied at this end post band-merging of catalogues since the merging software also applies the rest of the calibration, aperture corrections, detector zero-points and radial distortion, on-the-fly. A comparison with a test implementation at the WFAU end, after a few iterations, produced calibrated magnitudes agreeing to better than 1 millimag. RGM suggested that it might be worth producing a standalone catalog-level piece of software that applied all the corrections to the standard catalogue flat files and could be made available to external flat file users. MJI will investigate the best way to deal with this. <<<< JRL has nursed the flippin' jitter nights through the pipeline using a development version. This reminded MJI of the possible need for a telecon with JAC staff to discuss lessons learnt and related issues of requirements on data self-description. <<<< [The possible need of an equivalent observing mode for VISTA was mooted by JPE after our last set of minutes. Though JRL noted that with a 4x4 array of detectors there are few extra flippin' possibilities] Some feedback from DR4 QC eyeballing had been obtained, virtually all was very positive about the quality of the processed data. The only reported UKIDSS problem turned out not to be new DR4 data but a previously missed astrometry problem affecting a small percentage of GPS data (affecting ~200 files over a 3 year period). This was tracked down to several nights (mainly 07A) where one of the CASU nightly QC checks had been omitted by mistake. This problem should normally have been picked up and fixed (via further refinement of the astrometric solution) during these checks. JRL will investigate implementing the flagging of summit-rejected MSBs using the mirrored OMP database tables. If this is feasible a suitable keyword will be inserted in the raw FITS data files prior to ingestion. <<<< On a related note, MJI pointed out that from 08B we would be running an MSBTid script to check if MSBs passed the observing criteria (if set) in the header. A log for each processed night would be made available via the reduction progress web pages for JAC to make use of for faster feedback. <<<< EGS reported via email that he had met with Michael Rowan-Robinson and Seb Oliver regarding photometry of galaxies from 2MASS vs UKIDSS. After some discussion with STH and several tests he found that a comparison of 7" apertures provides a good agreement between both systems (http://apm14.ast.cam.ac.uk/mywiki/wfcam/PhotometryGalaxies). Smaller apertures do not agree so well, and there is an offset probably due to 2MASS aperture corrections. He also reported that on the topic of mirroring of the UKIDSS data at IAC, he had talked to Nigel who reaffirmed that there is no requirement on VDFS for this kind of redistribution of so much processed data. He has suggested to IAC personnel that they start by using the WSA interface to download some data as a test and take it from there. If they still want to do an Oliver Twist then other avenues may need exploring. He estimates the total amount of data they are interested in is around 4Tbytes and has suggested that the interested parties at IAC negotiate directly with WFAU to clarify their science needs. Photometric calibration paper: STH noted that he has updated the paper significantly following input from Paul Hewett, Steve Warren and Sandy Leggett. Steve's comments in particular were very thorough and all the contributors have definitely improved the paper in ways too numerous to mention here. There are a few extra tests from Steve that STH is carrying out at the moment to examine:- 1. the variation in the slope of the colour term as a function of galactic longitude and latitude, rather than season. 2. section 6.1. Steve thinks we can do more to handle the uncertainties within the SDSS photometry. It is not obvious that this is going to add significantly, or change the main result for the Y-band offset, versus the JHK filters. However, STH is still attempting to do the tests suggested to see what they show. These additional checks should be finished within a couple of days. Then the paper will be submitted. (Being a pint or so to the good, MJI declined the bet on this timescale proferred by PSB). 7. VISTA update PSB interacted with Florian Heissenhuber, ESO Network Specialist (according to his signature) to attempt to narrow down the performance problems with WFCAM raw data uploads, which run at an aggregate speed of ~650KBytes/s. Simple ftp tests achieved ~5.5MBytes/s. Florian is not in a position to install "iperf", which was his original suggestion to perform an objective test, on the server. However the interim conclusion is that the standard Internet from ESO headquarters to Cambridge is easily sufficient to carry Rice-compressed raw VISTA data on the final leg to the UK Science Pipeline, provided no throttling software is employed. JPE visited a short while ago prior to leaving for Paranal and more integration testing. There was some discussion of the Calibration plan document, test data to be taken, and a pic of a double star from the on-board test camera. JPE will buy 12x250GByte firewire disks and plans to DHL camera test data daily using them. There was some discussion of the idea of trying to run DRL recipes manually but that was dropped and also further discussion of direct Paranal-Cambridge bandwidth for quicker turnaround. After a bit of iteration the desire of the deep survey folk to be able to use random jitter patterns seems to have made it to the home straight. The pipeline doesn't care what dither patterns are used within reason (ie. within ~10% of a detector width is fine). There is a planned 3 day SADT + P2PP workshop at ESO in September and both STH and EGS are attending on behalf of CASU involvement in VISTA and VST public surveys respectively. Each survey PI + 2 worker drones are invited to attend: Day 1 demo, Day 2 trial Phase II, Day 3 discussion. RGM had sent in some comments on survey progress web page requirements to EGS and cc'd to MJI but further discussion of this was deferred until the next meeting with EGS present. MJI noted that the VMC consortium had emailed him directly about access to VIRCAM filter response tables and detector QE data, and, as directed, he had batted their request to JPE. 8. AOB Network upgrades: As predicted with the previous cack 10G-bit fibre routing to the APM building, Rodents 1 - 0 Networks. After re-routing the 10G-bit fibre down the correct orifice we anticipate fewer bouts of thuggery from local wildlife. Amazingly, and thanks to stunningly good service from our local network engineers, the APM building was only disconnected for a few hours after a quick switch back to the old 1G-bit fibre while a new 10G-bit fibre was installed. PS. The IoA is getting an additional 1G-bit link to CUDN; 1+1 = 2G-bit. ADASS: ADASS proceedings. RWA has made the last list of corrections suggested by ASP and (hopefully) the final version will be sent back in the next few days. Hardware: MJI acquired and installed an additional 14 Tbyte RAID6 fibre channel disk system, bought at a discount rate, and will now investigate the mysterious incommunicative raw WFCAM data disk currently known as "you ****". APM air con: The successful partition installation was followed up by a rearrangement of the main room air conditioning units, by relocating one of them to the store/computer room. This gives a much better balanced system and means we can pack away our winter work gear. MJI pointed out the upcoming Ringberg meeting, "Classification and discovery with large astronomical surveys" to be held 14-17 October. The abstract submission deadline is 30th June so if anyone is interested in attending they need to unplug their dyke (see http://www.mpia.de/class2008/). MJI briefly reported on the latest news in the CASU grant saga and on related grant applications either recently submitted or in the pipeline. Continuing actions ------------------ STH check that Plone astrometry technical pages are correct MJI acquire transmission curves for VIRCAM N118 filter JRL revive changes log web page for WFCAM pipeline MJI continue stirling efforts to find out more re: VIRCAM commissioning plans STH include graphic showing the size of the photometric radial distortion correction in the Plone technical pages. MJI order some suitable tables for the meeting area New actions ----------- MJI check the connections on LCD TV to see if displaying from laptop works and measure external dimensions MJI check with EGS what deep-stacking work has been promised for Video MJI see if JPE is amenable to sending us presentation from Video meeting STH investigate photometry affected by nebulosity in HawkI data MJI JRL produce web page of HawkI publicity for CASU MJI update pipeline toolkit on Solaris cluster MJI investigate ways of implementing photometric corrections for flat file users JRL decide if flipping jitters telecon with JAC is needed JRL implement flagging of summit-rejected MSBs during raw WFCAM data ingest MJI implement MSBTid logs for 08A as a trial MR