10th October 2005 Obs meeting room 11.30 - 13:30 Present: DWE, MJI, PSB, STH, MR, JRL, RGM, WJS Apologies: NAW, EGS Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Review of SV, Calibration and VDMT meetings 4. News from ADASS 5. VDFS VISTA update 6. Data archives update - UKIRT, WFCAM, AAO, ING 7. WFCAM news 8. Optical processing update 9. AstroGrid liaison update 10. AOB - CASU web pages - APM building Minutes ------- 1. Actions from last meeting STH have installed the SADT on the Solaris system. PSB said that this Java PSB programme was dated 23 June, so a number of suggested changes have not been included in this version. WJS has also been helping with these tests. This needs to be completed (WJS to collate notes) and the comments sent off - ongoing <<<< JRL has finished writing the access scripts to the mirrored OMP database tables. He will email people their location. A username and password will be needed. JRL said that he would check to see which tables are available and which are mirrored. <<<< MR has finished writing the draft of the flat field properties and sky frames report. MJI said he had some comments to feedback <<<< MR ensured that the features web link is publically accessible EGS MJI at the AAO board meeting found out what was happening re: the AAOmega data processing MJI sorted out the problems regarding the new processed data versions that were being transfered to WFAU. PSB said that the peak transfer had been Saturday with 1.1TB transfered. With the current costing model, the transfer for the whole weekend cost £1300. It was noted that all data files were copied, not just the catalogues. It is expected that up to 20 TB of WFCAM transfers will occur each year. It was then suggested that PSB contact the University Computing Service to discuss the UKERNA pricing policy. <<<< It was noted that the transfer rate will rise to about 50TB per year for data transfered for both WFCAM and VISTA from 2007 onwards. MR reprocessed the catalogues for the V2 release (but see later comments) JRL MJI has made no progress with the URD feedback for the next VDMT meeting (was away) - ongoing <<<< There was a request to add the VDUC minutes to the internal CASU pages. RGM will place these in a suitable location. <<<< MR have started work on a survey progress web page linked to the DQC EGS database. This task will now be added to the quarterly task list. RGM pointed out that the JAC had a survey progress page: http://www.jach.hawaii.edu/UKIRT/surveys/UKIDSS_Survey_Status.html JRL has finished the AAO archive access scripts. He will distribute the URL so that we can test it and send back comments <<<< STH did a direct comparison of UKIRT FS and 2MASS photometry. The end result is that there is a 4% offset in H intrinsic to 2MASS (this is a known feature). After much discussion, including with the Chair of the calibration working group, the end result is that the frame-by-frame WFCAM system will now be based on 2MASS but with a -0.04 offset in H to bring it more accurately onto the Vega (and UKIRT FS) system. This will all be more fully documented in Simon's calibration report. MJI has read, commented and signed off the PD33s. ALL the issue of Plone and Solaris web pages was deferred until item 10. 2. Comments on WFAU minutes There were a number of WFAU minutes since the last meeting. Mike noted that there had been some problems with the transferring of reprocessed data which was mainly linked to the semaphore flags. We think that this has now all been sorted out. 3. Review of SV, Calibration and VDMT meetings SV: STH said that more science had been carried out with the data, but not many more problems had been found and in general people were satisfied. There were a few small issues raised: There were several comments about the merged classification statistic. This arose from discussions between MJI and NCH to satisfy users who wanted a single number from the results of the classification programme, which would have been run independently on several different passbands. The current version only makes use of the quantised individual classification flags and some users felt that using the entire classification information might be a better solution. MJI said he was investigating as soon as his email backlog allows. <<<< Blended objects were occassionally ending up as single objects. MJI noted that deblending was a statistical process so this was no surprise but said that he would investigate the specific cases mentioned. <<<< There was much discussion regarding the release of SV (and other 05A) data to facilitate WFCAM data exploitation. The status of the crosstalk processing was raised and it was reported that this was currently being test run on 05B data and if this proves satisfactory then 05A will be reprocessed with a crosstalk correction. Calibration: STH said he gave the talk as presented/discussed in draft form at the last CASU meeting. A number of actions are now on STH. He had been requested to redo the systematics analysis only using data from nights of good seeing and good transparency. Also a specific investigation into radial terms was requested in addition to the more general 2D systematic investigation. In general, though attendees were satisified with progress. VDMT: MJI reported on the last telecon. The inconsistencies between the pipeline release schedules of CASU and WFAU had been discussed. The main change for CASU was to align the end of the VDFS development phase to the of Q3 2007 to coincide with the end of the current VDFS development funding. Some of the nomenclature has been changed so that "Phases" of software development will replace "Versions" to avoid confusion with Versions of the pipeline processed data. MJI said that JPE had initiated some discussion over methods of transferring data for VISTA from ESO. MR noted that the current peak transfer is about 2Mbytes/s. However, due to software checking and other overheads at the ESO archive end, the effective transfer rate was 0.4Mbytes/s. This is difficult to reconcile with the claimed bandwidth of 155 Mbits/s to Garching. 4. News from ADASS JRL said that his talk was well received. Although he had no questions asked during the presentation, he got a good number afterwards (which he preferred). PSB said that both JRL and EGS gave good talks, especially since these were among the few presenting results using real data. JRL said that the idea of ADASS 2007 being held in London was well received. He has already been down to the British Museum to investigate using their Education Centre. A LOC meeting should be held soon. MJI said that he would forward the details regarding the Edinburgh NIR meeting (9/10 November) to JRL. <<<< PSB said that he had helped organize the Leap Second BOF (there will be one at the end of the year - Second that is). It was also noted that the FITS standard will imminently have a 64 bit format for data. 5. VDFS VISTA update WJS reported that an announcement has been made by JPE that the money has been found to fund both a Z and Y filter for VISTA. In fact, the Y, J, H and K_s have already been manufactured and their responses measured. Copenhagen has also placed an order for a 1.19 µm filter. Pictures of the latest developments can be seen on the ROE VISTA web page http://www.roe.ac.uk/ukatc/projects/vista/news.html (see "Civil work" and "Telescope News"). PSB suggested that it would be useful if some of CASU could go to RAL to test drive the camera (with the science arrays) and "observe" with the system to generate the sort of engineering data that we need to shakedown the VISTA pipeline development system. If feasible this would have to be before the camera departs RAL, implying ~January. WJS was asked to liaise with Gavin Dalton and Kim Ward to see if this was possible. <<<< JRL reported on his progress with the Data Reduction Library. There was a need to deliver 0.1 by the end of the year. Several recipes and modules have been written. The question of how to release the DRL was discussed. It was thought that to start off with a direct release to ESO via a CVS would be useful in order to get early feedback. 6. Data archives update - UKIRT, WFCAM, AAO, ING UKIRT: Up-to-date. WFCAM: MR reported that a new front page has been implemented for the raw data archive. New data is being received (05B) and ingested. AAO: The AAO archive interface is now mostly completed and is under test. ING: Two new disc servers (8 Tbytes each) have arrived and been installed in the APM building. One of these will be used for ING data and the other for WFCAM data. Internet transfer of data is progressing well ie. usually arrives within 24 hours of the data being taken. There is currently a gap between the last DVD data ingested and the first Internet data. This is in the process of being filled using transfer of data on 400 Gbyte IDE disks. There are now several tasks for the ING archive requiring urgent action MJI will sort this out offline with JRL, RWA and EGS. <<<< 7. WFCAM news MR reported that processing of the 05B data was about to start. The delay in processing was due to delays in generating twilight flats and finalising the pipeline implementation of the cross-talk correction. These issues are supposedly resolved, so processing will now start. It was noted that the UDS will use 3x3 interleaving but interestingly that the image quality in the recent set of test data was only ~0.7 arcsec. JRL said that he would double check the current version of the pipeline to make sure that it did correctly include cross-talk processing. <<<< Changes in the various survey MSB designs have resulted in a significant reduction in the data volume. JRL was asked to arrange the next JAC telecon since there has not been one for some time due to assorted absences. <<<< MJI pointed out that there were a number of bugs in the V2 reprocessed data (including table NaNs and repeated keywords). These should have been spotted eg. by running fverify as recommended. This (again) emphasizes the need for a test harness after any significant pipeline changes. Possible suitable data for this would be some of the much studied 8th April data. MR and MJI will investigate. <<<< MJI and JRL said that they would discuss the various bugs and related actions. <<<< STH said that the various changes to the software should be noted on the web site. MR said that the CVS has a log of changes to the software that can be used for internal use. MR will look into this issue. <<<< 8. Optical processing update MJI noted that the VST survey proposal resubmission deadline was the 15th October. He is helping provide the data management and products section. 9. AstroGrid liaison update No news. 10. AOB CASU web pages It was decided that a small task force would be formed (EGS and DWE) to see how we can improve the group's web pages. The general feeling at the meeting was that it was difficult to find things and that having two separate systems (Plone and Solaris-based HTML) was not helping. MJI said that he would forward EGS's email on this topic to DWE. <<<< DWE and EGS will report back at the next meeting with suggested ToRs and a list of issues to tackle. <<<< APM building MJI said that it was now 6 months after the completion of the refurbishment of the APM building and it was time to draw up a snag list. Various issues were raised: noise, wobbly toilet ride, esoteric heating system..... MJI said that he would forward the email from Peter Weeks so that we can see if all the snags are covered. <<<< Gaia news MJI reported that the UK consortium grant bid for Gaia processing will be submitted on the 21st October and then go before the new PPRP process. Continuing Actions ------------------ PSB finish off SADT tests, collate notes and sent off comments WJS MJI feedback comments to MR on flatfield and sky properties draft report JRL distribute location of OMP database access scripts and check which OMP tables are available and that they are mirrored correctly MJI append more details to the URD feedback for the next VDMT New Actions ----------- PSB contact the University Computing Service to discuss the UKERNA pricing policy RGM put VDUC minutes on internal web pages and email location JRL distribute the URL of the AAO archive interface ALL test drive AAO archive and feedback comments MJI investigate merged classification statistics MJI investigate blended object complaints MJI forward the details of Edinburgh IR meeting to JRL WJS sort out a possible CASU visit to RAL with Gavin Dalton and Kim Ward MJI convene meeting with EGS, RWA and JRL to discuss ING archive issues JRL check current version of the pipeline does do cross-talk processing JRL arrange the next JAC telecon MR sort out a possible test harness (8th April data) MJI JRL discuss recent bugs in V2 reprocessed data MJI MR investigate web-based presentation of various changes to software EGS sort out Plone/HTML for group web pages DWE MJI forward EGS's email regarding web pages to DWE MJI forward email from Peter Weeks regarding snag list to all ALL who will add to snag list if needed