12th June 2006 Hoyle committee room 13.30 - 15:30 Present: DWE, MJI, JRL, PSB, STH, MR, EGS, RGM (to 14:30) Apologies: NAW, WJS Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Data archives update 4. Optical processing update 5. WFCAM news 6. VDFS news 7. Plone update 8. AstroGrid liaison update 9. AOB Minutes ------- 1. Actions from last meeting MJI has made no further progress investigating the merging of classification statistics - ongoing <<<< (It was noted that an interim solution had been sent to WFAU some time ago.) WJS it is not known if contact has been made with the VPO to confirm that we will get no more ETC-related digitized data from them (WJS not present) - ongoing <<<< MJI said that he would email WJS to remind him about this. <<<< MJI has emailed Nigel Hambly regarding the non-implementation of the saturation flag in the classifier JRL has not implemented the fix to the problem with the UKIRT archive (missing header information problem) - ongoing <<<< MJI placed the submitted UKIRT Board report on our web site JRL provided MJI with the original data for the non-linearity graphs JRL requested further dome sequences to be taken. These are now in Cambridge (see item 5) and being analysed STH produced 2MASS comparison plots on a per detector basis DWE have not yet helped JRL in setting up an ADASS 2007 web site. This EGS will require copying and cloning various style sheets from previous Plone versions of the ADASS web site JRL has released version 0.2 of the DRL. The current version is 0.3 and is about to be released 2. Comments on WFAU minutes - transfer tests and UKLight update PSB said that Johann Bryant has sent off the application form for UKLight on 1 June. A response is now awaited from UKLight. The timescale of this and the implementation "roadmap" is unclear. PSB also noted that we still wait expectantly for WFAU to carry out a transfer test using single-thread large-buffer zero-encryption high-performance scp. MJI pointed out that the NaNs found in the catalogues that had been noted in the WFAU minutes had now been fixed and the catalogues regenerated. These were a minute fraction of the total detected objects and only affected mainly flux values and errors for the largest radius apertures. 3. Data archives update - AAT, ING, WFCAM, UKIRT WFCAM - MR said that all the data from 28 April to 31 May have been copied to disc, reformatted to MEFs, verified and are awaiting ingestion to the raw archive. The latter is contingent on final tests on the updated verification and flagging procedure (see later). AAT - no change UKIRT - no change ING - EGS said that the restructuring of the archive from a tape/CD/DVD numbered system to a date-based directory structure is progressing well. The period 1996-2006 ie. the present is almost complete. Duplicate data, usually due to several versions of data on multiple CDs/DVDs, is being tracked and corrected at the same time. MJI noted that the internet transfers, already date-based, form part of this and are complete? from 20050101 to 20060531. There is a small gap toward the end of 2004 that requires ingestion from DVDs and that will be taken care of next. After this the next stage is to investigate the pre-1996 data and see how far back it is feasible to push this. In parallel, the re-ingestion of the header information into a new version of the database is progressing well. There is a need to finalise checks for any missing data. The total archive is about 20-25 Tbytes in size and has been backed up onto tape (~100 Ultrium LTO-II tapes). 4. Optical processing update EGS, STH and MJI have been working on data from various optical telescopes (INT WFC, CFH and Subaru). RGM pointed out that it would be useful if EGS could update the INT WFC web pages to contain links to the latest flat-field, fringe and bias frames. <<<< 5. WFCAM news MJI reported on the reprocessing of 05A data (Version 3.0). There are still a couple of nights that are causing problems due to jitter keyword FUs, but these should be sorted as far as is reasonable in a few days. 11 nights with interesting sky file and sky correction problems (pipelines CAN lie) near the start of the 05A reprocessing were found after following up a fault reported by Mike Read. This was traced as to a script update FU in the CVS. The affected nights have now been reprocessed and flagged as Version 3.1 In addition to the broken files discussed in the last minutes, WFAU reported some 05B catalogue files with a few columns with a small number of NANs, mainly in large aperture flux and error estimates columns. The catalogue software bug fix reported in minutes a while back, seems to have cured the problem and the catalogues were reprocessed and made available for copying to WFAU. MJI reported that there was also a mysterious problem with virtually all the sky estimation and subtraction for the night 20051015. This was picked up at UKIDSS QC checking. MJI investigated what could have caused the problem but other than possibly non-optimal choice of sky stratgey by the pipeline there were no obvious faults. However, as a final check, the night was reprocessed using the latest pipeline version which uses an improved sky correction strategy and this seems to have resulted in much better performance for that night - though final confirmation of this is still awaited. MR went through some of the new features that have appeared in the 06A raw data. In the period 28 April to 31 May (33 nights) there have been many files with randomly missing extensions. Of the 50,000 files, 400 are incomplete (0.8%) and half of these are due to chip 2. Additionally, and independently, the chip 2 extension is sometimes populated with null values. Around 3.5% of the MEFs suffer from this. However, it is not quite that bad, since at least 1/2 of these occur on 1 night. JAC suspect that this is a controller problem. About half the nights have at least one null chip 2 file and these are spread evenly throughout the period. These files were causing problems during ingestion and subsequent processing so a more robust and thorough scheme has now been implemented to deal with them and flag them by adding new keywords. Subsequent propagation of this corrupted data (eg. interleaving and stacking) is already handled by the confidence maps and provenance track record in the headers. [AJA reported after the meeting that the problem had now been "fixed", as of early June, such that WFCAM is now dropping less than a frame a night.] STH reported on the non-linearity saga. Further checking of the original dome flats showed that all was OK at the 1% level. However, to recap, comparison with 2MASS showed that there was a 2-3% level departure from linearity over the full ~5 magnitude range of overlap. More dome flat sequence data has now been taken by Sandy Leggett on 26 May to investigate this. In addition exposures of standard stars covering a range of integration times have also been acquired. The processing of these is now underway and we are even considering after a suggestion from Sandy Leggett that this might be a rate-dependent non-linearity (cf. NICMOS). MJI said that the presentation he gave at the UKIRT Board meeting (23 May, University of Hertfordshire) was well received. Mike Reid and Steve Warren were also present at the meeting. The main debating point was the release of WFCAM data with the PI data being the main issue raised. STH said that he had no news regarding acquiring data from the 2MASS touchstone fields. He said that he would chase this up. <<<< MJI noted that further 7 and 10 TB fibre-channel RAID arrays have been added to the system and that we will be continuing with this upgrade route. It was suggested that DWE and MJI update the hardware web page. <<<< MJI and PSB have had a discussion about bulk data storage requirements with a representative from HP who is an expert on Storage Area Networks. He seemed suitably impressed with our homegrown data storage system but, of course, followed up his visit by sending some literature for alternative (more expensive but setup and maintained by HP) bespoke solutions. The planned upgrade for an extra air conditioning system in the "store room" was stalled for a week due to delays in delivery of the condenser unit. This is now expected tomorrow and will take about half a day to install. [Note added on Wednesday: it was installed and is now operational]. This is all part of our proactive infrastructure upgrade programme and should now give us sufficient capacity (and backup) for the foreseeable future. 6. VDFS news There was not much new news on VIRCAM since WJS was away. PSB said that he had received a file from RAL since the last meeting, but it only consisted of headers and no data. PSB welcomed the release of the AIT Requirements document from JPE. This mainly lists the lab tests that CASU asked to be carried out on the VIRCAM system . It was noted again that some Cambridge presence at these tests would be desirable. The DRLD is now up to Version 1.4pre12 and final debuggery tests are ongoing on Version 0.3 of the DRL. These mainly involve the jitter/microstep recipe - the main science recipe. It is anticipated that V0.3 will be released at the end of this week. PSB, JRL and MJI said that they would have a short meeting around coffee time tomorrow (13 June) to finalise changes to the accompanying DRLD. <<<< We were reminded that the community was in the process of discussing the desired observing modes for VISTA. It was stressed that it was important for us to be involved in this discussion so that the decisions made are consistent with template and pipeline processing designs. These will be further discussed at the VISAS meeting in Edinburgh (20/21 June) which JRL will be attending to represent CASU. RGM will also be there in his PS PI role. 7. Plone update DWE described the latest progress on the Plone redesign of the web site. There was a meeting of interested parties earlier today. The overall design seems to have settled down, but most of the technical summaries have still to be written. The plan is to get as much of these written by the end of <<<< this month (when EGS is away) so that the site is almost ready to go live. This would be followed by extensive testing and comments from the whole group with the intention of making the site live sometime in August (2006). There was some discussion regarding the setting up of a common mail account (or other system) for VDFS questions. It was decided given the low current attrition rate that the current "system" seemed to work fine so no change was needed ie. generally MJI gets emailed and the work gets distributed. A short demo was given of the site as it currently stands and various comments were made. 8. AstroGrid liaison update PSB noted that the Cambridge AstroGrid twiki had been hacked again. Most of the links had been redirected to a site of a gynaecological interest (that's one way of putting it). 9. AOB DWE and JRL will be going to the image processing workshop in Dubrovnik. MJI noted that there will be a workshop at ESO in January and a CASU presence would be desirable. Continuing Actions ------------------ MJI finish investigating merged classification statistics WJS contact VPO to confirm that we will get no more ETC-related digitized data from them JRL implement fix to problem with UKIRT archive (missing header information problem) New Actions ----------- MJI remind WJS regarding above action EGS update INT WFC web pages with latest flat-field and fringe frames STH acquire data from the 2MASS Touchstone fields DWE update hardware web page MJI PSB have short meeting to finalise updates to DRLD JRL MJI DWE finalize content of Plone site by end of month EGS STH MR