11th March 2009 APM meeting tables 11:30-12:52 Present: MJI, JRL, STH, MR, EGS, RGM, RWA, NAW, SC, DWE Apologies: Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Recent and upcoming meetings 4. Data archives update - AAT, ING, WFCAM, UKIRT 5. Optical/NIR processing - HAWKI, INT WFC, MegaCam, Subaru 6. WFCAM update 7. VISTA update 8. AOB - rolling grant status etc.. Minutes ------- 1. Actions from last meeting STH the technical pages are still not up to date - ongoing <<<< MJI JRL has made some progress on producing suitable images for the web page of HawkI publicity for CASU - ongoing <<<< MJI have not finished off the stage-I software release pages - ongoing <<<< EGS RGM reminded us about the request to write an example utility to convert catalogue binary tables to ASCII with all the assorted photometric corrections applied. <<<< STH has put the photometric calibration paper on the Plone and VDFS sites MJI has not finished off the technical paper - ongoing <<<< MJI contacted Mark Casali about the HawkI radial distortion and sent him some example plots illustrating the improvement using correct ZPN model RGM liaise with JRL about processing HAWKI PI data - still ongoing since awaiting previous SV HAWKI data completion <<<< JRL has produced a definitive solution to the pipeline versioning issue and implemented it for 09A processing EGS backed-up the online 2008 ING archive data JRL has not finished the HAWKI NB image data processing to improve sky subtraction - still ongoing <<<< JRL contacted JAC about upgrading to a more recent CFITSIO version in the conversion software JRL have made good progress in sorting out the VISTA documentation set STH and ensuring that the latest copies are available on the internal RWA web pages. JRL has produced a spreadsheet documenting all this. A few documents still remain to be indexed and cross-checked - ongoing <<<< STH has contacted the VISTA PSPIs regarding the need to get permission through JPE for access to reduced VISTA test frames 2. Comments on WFAU minutes The various recent hardware and software (32/64-bit) foibles were noted. JRL commented that the pipeline and particular the current version of the cataloguing software, works fine on both 32-bit and 64-bit operating systems. He was reminded to send the latest version of the cataloguing software to WFAU together with versioning information on library dependencies. <<<< The rediscovery of the CFITSIO 16-bit compression feature was noted. This is one of the reasons we still use an earlier version of CFITSIO to compress the confidence maps. JRL appreciated the SVN suggestion but commented that despite it having several advantages over CVS there was a snag. The problem is that the VIRCAM pipeline software including ESO software deliverables, which shares much of the WFCAM code, sits in a CVS housed at ESO. He felt that running two similar but different versioning systems in parallel could nuke his brain. MJI reported on a brief discussion with NJC regarding deep stacks of ~100s of frames and concommittent memory issues -v- use of intermediate stacks and apparent degradation of image quality. MJI had seen similar effects with other data and thought the problem was most likely caused by seeing degradation due to "astrometric" blurring. The solution we usually adopt is to always update the astrometry of intermediate stacks via the usual catalogue method. 3. Recent and upcoming meetings MJI and JRL commented that they had recently learnt of regular VISTA (Instrument Operation Team) telecons to discuss various issues including data quality and properties. These are held the last week of every month and are organised by Thomas Szeifert at ESO. We thought it was now timely for CASU to be directly represented at these given that we have a.) measured many of the parameters of interest and b.) know at least as much about the properties of VIRCAM as anybody else. The next telecon is apparently scheduled for March 26th and MJI will inquire if someone from CASU should take part. <<<< The upcoming JENAM fete was mentioned and a whip-round for volunteers for talks, though painful, was relatively unyielding. However, NAW raised a paw and will give a CASU-related talk at JENAM (and in the time it takes to write these minutes has already submitted an abstract). <<<< MJI was part of the WFMOS roadshow entertaining the Gemini Panel in Hilo February 22,23 in a supposed shoot-out of the two conceptual design studies. He noted that this was the first ever ppt talk he had written in advance (3 days) of a presentation and was still in shock and awe. Regular Plato meetings continue to occupy NAW and STH but at least STFC have funded, albeit at modest level, our contribution to the design study, unlike everything else we do. Other upcoming events: NAW noted that Jonathan Fay, lead developer of Microsoft's World Wide Telescope will be visiting the IoA after the JENAM, viz 23-24th April 2009. He will be talking at the VO session (t) of the JENAM - 22nd April 2009. 4. Data archives update - AAT, ING, WFCAM, UKIRT AAT: data dribbles in - up to date ING: data trickles in - up to date WFCAM: data sputters in - up to date UKIRT: RWA is reading on all the old DLT tapes prior to backing them all up on LTO4 media as a precaution against degradation/ancient tape readers. (MJI - do we have the 08B CASS data ?) 5. Optical/NIR processing - HAWKI, INT WFC, MegaCam, Subaru There has been a lot of action on the IPHAS/UVEX front with EGS reducing all the 2008 season data taken on the INT WFC. The IPHAS data has been ingested into the DQC DB and also released to the consortium via the usual web pages. UVEX is still being footled with due to assorted discussions regarding band-merging with IPHAS and the variable no. of passbands observed. Cross-matching of these two datasets and band-merging of UVEX data is more of an open issue. The UVEX team seem unable to come to terms with proper use of the band-merged FITS binary tables - which would solve all of their problems. To save time and effort at this end, EGS will find out what coding language they are using, and within reason will produce an example piece of code to illustrate how to go about it. <<<< There was a brief discussion about IPHAS data access through VIZIER. CASU could not see the point of doing this given the existing public access and the problems and extra effort involved in releasing it through VIZIER as well. NAW was volunteered to compose a polite but firm email to the IPHAS consortium regarding not duplicating the data release through VIZIER. <<<< Of course if they want to provide the effort themselves - fine. MegaCam is still awesome but some recent Subaru data on NGC891 that Rodrigo Ibata sent over for diagnosis tests is awesomer^2. In ~10 hour integrations in V,i in 0.5 arcsec seeing last Autumn using the new Hamamatsu detectors this reaches unprecedented ground-based 5-sigma imaging depths of 28th magnitude in i and 29th magnitude in V. The same depth as equivalent HST ACS observations, but over a 30 arcmin field ! 6. WFCAM update 09A processing was held up for about a month while waiting for a full set of sky flats to be taken. The long wait was mainly caused by uncooperative weather which has the side effect of not that much data to process either. This has all now been remedied and as a glance at the processing status page for 09A (http://casu.ast.cam.ac.uk/surveys-projects/wfcam/data-processing/ report_night_reduction_status_casu?semester=09A&SUBMIT=Submit+Query) will reveal, with the help of the weather we are now back on track with processing. MJI asked about the status of duplicating PSB raw data access scripts so we have a completely independent backup when it inevitably goes wrong at the next computer hiccup. MR will chase this up though so far transfer and ingest of 09A data have gone seamlessly. <<<< MJI recalled a request from Derek Ives via AJA for information about possible non-linearity at low light levels for Rockwell detectors. He pointed them to the web pages containing the best direct measures on non-linearity we have, but noted that those measures are long in the tooth and don't really adequately sample the relevant ADU range - though in mitigation there is currently no compelling evidence of any signficant non-lineary effects in WFCAM photometry. As Version 3 of the WFCAM flat field screen should just about be in place by now it would be timely to repeat a set of linearity measures as noted previously. <<<< JRL and MJI reported on a telecon with Seb and Omar about UDS 05B reprocessing results. Unfortunately the reprocessed 05B K-band data, using the pawprint sky method (developed for UltraVISTA), enthused them so much (0.3 mags deeper) they now want to reprocess ALL the UDS data the same way. There are still a few potential gotchas with unfettered use of this method, so Seb and Omar were asked to do a few further tests to make sure this was what they REALLY REALLY want. (MJI noted that the results of these came in just before this meeting. He will forward the UDS test results to JRL.) <<<< This lead to a short discussion of the various sky estimation strategies that the pipeline now has at its disposal: none - not enough data to form a sky so don't bother; leftovers - odds and sods throughout the night combined to make sky frame; pawprint - sweaty iterative stacking with ever deeper object masking; tile - default, usually the best option but needs tile of pawprints How these work and more about them needs to go on the technical plone pages - already noted as an ongoing action. (Note that current offset sky strategy makes use of the pawprint method by default, though MJI had already mused why not tile as alternative since can give better skies in many cases.) Although the information about sky strategy adopted is in the processing SUMMARY file in each date subdir, RGM asked if we could place a keyword in the sky FITS file annotating which method has been used. Since this is easy to do, it is also worth adding to already processed 09A data as it <<<< hasn't been flagged for copying yet. No doubt there will be a request to have it propogated through the WSA shortly hereafter. MJI noted that WFAU transfers of recatalogued 08A data were now complete but that 08B were still awaiting their turn. This presumably means we can now fix the Petrosian etc... fluxes and radii for blended objects for 05A -> 08A inclusive ?? For MAR eyes only - Jim admitted he hadn't put in place the code to compute dither offsets en masse for WFCAM. He is beavering away on it now since we notice similar occassional problems for HAWKI and VIRCAM. RGM asked for a keyword to flag when there was a problem with different computed offsets for the detectors, but subsequent to the meeting we decided to compute the dither offsets in a single robust LS fit in xi,xn space to circumvent too many iff's. [This is a known (admittedly rare) problem. The pipeline computes the observed dither offsets for each detector separately. It is therefore possible for it to get the offsets in a dither sequence correct for say detectors 1,2,3 but only partially correct for detector 4. When they are stacked this means that some of the subcomponent images effectively get rejected causing an effective shorter total exposure and hence the problem with perceived magnitudes on that detector.] A coffee time conversation regarding edge effects on photometry and how to deal with them in a transparent way then sprang to mind. The suggestion of using some of the surplus bits in parameter #55 to encode the average confidence level within aper3 was mooted. This seems like a good idea since it covers, partial exposure in dithers, surplus not so good pixels, edges etc.... The current use is #bad (i.e. 0 confidence pixels) but there is plenty of room to include the average conf too. What do WFAU think ?? 7. VISTA update JRL and STH summarised the current state-of-play. Filter wheel: A lot of the problems with the data have been linked to a problem with the filter wheel which seemed to have a great deal of trouble moving to specified locations. This is crucial if the flat fielding is to work properly. After the camera was warmed and opened up it transpired that a screw in the filter wheel had worked itself loose and was digging into the side of the container causing flakes of paint, undercoat and finally metal to rain down on the filter surfaces. The resulting particles cast shadows and hence caused some funny flat fielding properties, which we now hope will be a thing of the past. During the time VIRCAM was off the telescope the Z filter has now been installed. Data reduction: Data from 1st December through to end of January have all been reduced in a 'standard' way. The tiles were all done using a tilesky algorithm very similar to WFCAM. Those that weren't part of tiles were done using the masked pawsky algorithm (also now used in WFCAM processing). By and large the results are very encouraging although there are still a couple of outstanding issues: -- detector 16 has a worrying 'feature' in its top half which gets very noisy and has a strange cross hatched pattern with blotches that don't subtract out well in most cases. The source of this is being investigated. -- observations in a region with some very bright stars have shown some interesting reflections that can occur when one of these very bright stars is located between two detectors. This is also currently being investigated further. -- preliminary analysis of standard star data shows a problem with the linearity of the results. JRL will return to the linearity sequences that have been observed and hopes that some new ones will come along very soon. The full results of the reduction will be collated in the next week or so and MR will post them in a manner similar to the WFCAM DR pages, although this will not be a public page yet. Offset skies: After being told ad-nauseam that it is not possible to schedule offset sky observations linked to an object observation using the ESO P2PP infrastructure, recent upgrades announced at the September PI workshop beg to differ. We have now been asked to draft a proposal to define how these should be done with VISTA. Schedule: VIRCAM is to be reinstalled on the telescope on 17th March, with observing commencing at the end of the month to revisit the technical verification of the system. STH then showed some preliminary results from the VISTA processing of data taken in Sep, Oct and Dec 2008. In brief: 1. As noted earlier there is a significant residual non-linearity in the photometry (shown in a 2MASS comparison plot). JRL will look at the application of the linearity model to try and understand where this comes about. In particular STH and JRL will test not applying the non-linearity correction to a suitable night of data to see how this affects the observed non-linearity. <<<< 2. The colour terms with respect to 2MASS are fairly small for JHKs, as expected. Further progress here awaits fixing the non-linearity issue. 3. The sky brightness measurements have been compared with WFCAM, with the proviso that: (1) no data have yet been omitted for cirrus and lunar proximity. Early indications are that the sky/background looks similar brightness in YJH and 0.2 mags brighter in Ks. 8. AOB MJI gave an update on the rolling grant status from the previous round (i.e. due to start 1st April 2008) which was stalled yet again despite a huge amount of effort on our part. This is particularly galling given that PDRA support has already been awarded this current grant round for VISTA and VST science exploitation. SC reported on recent hardware additions to supply sufficient disk storage for VISTA and WFCAM to last this current year. MJI noted that it had better be since that is the last of the equipment dosh. Continuing actions ------------------ STH check that Plone technical description pages are up to date MJI JRL produce web page of HawkI publicity for CASU MJI finish off stage-I software release pages EGS MJI finish off technical paper RGM liaise with JRL about processing HAWKI PI data JRL redo the HAWKI NB image data processing to improve sky subtraction JRL sort out remaining VISTA documentation set and make sure latest copies STH are available on internal web pages RWA New actions ----------- MJI write utility to convert binary tables with all corrections applied to ASCII JRL send latest version of cataloguing software to WFAU MJI contact JPE about one of group partaking in VISTA telecon 24th March NAW submit abstract for CASU talk at JENAM JRL place keyword in sky FITS file describing which method has been used EGS inquire what language UVEX team are using for coding and produce example NAW send polite but firm email to IPHAS about VIZIER data release MR chase up wherewithall to produce duplicate WFCAM transfer script MJI inquire about status of new WFCAM linearity measures MJI forward UDS test results to JRL STH test not applying non-linearity correction to see how this might affect JRL observed non-linearity -v- 2mass