3 June 2003 Present : MJI, STH, JRL, PSB, DWE Agenda ------ 1. Actions from previous minutes 2. Comments on recent WFAU minutes 3. Highlights/lowlights for monthly report 4. Preparation for June 12/13 WFCAM meeting 5. VISTA camera testing issues pre-delivery (Q2 2005) 6. VEGA e-science bid 7. Comments on Perry's reprocessing notes for VEGA bid 8. AOB Minutes ------- 1. Actions from previous minutes DWE obtained Ian Bond's difference imaging software and has carried out initial tests on easy data. Single 2kx4k images took 70-80s to process on a 1.9 GHz PC. The next step is to trial using data with different seeing and offsets eg. WFC data for SN searches. <<<< The current version doesn't work on MEFs so DWE will ask IAB to modify his software. <<<< MJI acquired the 2MASS catalogue and placed it on apm3 (/local/raid/mike/2mass). This is in gzipped ASCII format. There was some discussion about what format we should convert the data to, and after the meeting we decided to use sorted FITS binary tables as we did for the EDRs. MJI has reminded JRE/WJS about the need to update science/user requirements documents for VISTA JRL has reminded AA about need to acquire H and K twilight flats. STH still needs to contact AA about use of service night, possibly for <<<< standard observations though not clear how much use this will be. STH has finished off the first draft of the photometry document. MJI will discuss the new plan for a survey manager with AA at the Edinburgh meeting. STH confirmed what DWE reported at the last meeting and that the intention is for 2 x 0.5 posts one based at JAC the other in UK <<<< JRL booked the hotel for us for Edinburgh meeting DWE booked flights for us for Edinburgh meeting STH was not able to obtain more useful versions of the Astrowise documents. 2. Comments on recent WFAU minutes DWE was impressed at the turnaround speed for producing the minutes of WFAU meetings. It was generally considered that a data transfer rate of 4 Mbytes/s was readily achievable and good enough for general data transfer. Tests to Leicester from Cambridge achieved this. Similarly, pushing data from apm3 up to Edinburgh also achieved this rate. PSB has liasesd with ETH on network transfers IoA -> ROE. Get same basic speed ~4MB/s even from IoA Ultra 1/140 ftp server (when using normal ftp, ie. unencrypted). Same test to an ROE Sun gave <1MB/s, conclude some machines 'see' a real bottleneck. Sorted out firewall issues with ROE not passing some of IoA IPv4 space. 3. Highlights/lowlights for monthly report The monthly report is being collated by STH. A brief discussion of the salient points ensued and the relevant extracts have been emailed to Simon to fold into the workpackage summary and gantt charts. There some concern over the lack of progress of the simulations workpackages and its impact on other packages. JRL said that a better source of data for throughput simulations would be CIRSI data. However, <<<< there is also a need for suitable data for interpolation tests and DWE thought that the WFC option be better for this. <<<< There was also some discussion regarding the need for totally synthetic data (eg. SDSS) to test catalogue generation. STH will investigate the the SDSS software for this purpose. <<<< 4. Preparation for June 12/13 WFCAM meeting STH reported that he was working on the photometric standards presentation. The pre-meeting document will be sent to Mark tomorrow after we have all had a chance to comment on it. This will also feed into meeting on 6th here with PCH and SJW to discuss calibration and other issues. <<<< MJI and JRL will produce their documents for items 6,7 "QA pipeline processing at the summit" and "Unresolved issues ...." by Friday. <<<< PSB raised the issue of feedback of the DQC and differences between the summit and the Cambridge pipeline. DWE gave a summary of AA's viewpoint regarding this which he gave at the 14th May meeting at Imperial. PSB wondered if there were implications for VISTA design. There was then some discussion regarding current ESO policy on this topic. We should all read the documents sent out by Mark Casali before the Edinburgh meeting for discussion en route. <<<< 5. VISTA camera testing issues pre-delivery (Q2 2005) Steven Beard raised the question with PSB about whether the pipeline DQC modules will be ready in time for the VISTA in-lab commissioning (Q2 2005)? The basic answer is yes, provided the FITS headers and data container structure are MEFs - which appears to be the case. However, it is unclear exactly what quality control measurements would be required. If something non-standard is required, we need to know about it. PSB will investigate and report back. <<<< 6. VEGA e-science bid There was a general discussion regarding the AstroGrid2 and VEGA e-science bids and how it might impact our activities. General data centre activities are folded in the AstroGrid2 bid; further data processing development in the VEGA bid. MJI reported that the new CASU grant will be a revisable grant rather than a rolling grant. This reflects the project-based nature of most of the likely funding. An almost correct letter announcing the grant was received by Paul Aslin on 20th May. Minor revisions are being incorporated in Swindon and we expect (sic!) the final? version to be sent any time now ! There was some concern expressed about how much/little secretarial support we get. Some of the problem has been in identifying which tasks to delegate and a reluctance on our part to impose on the general secretarial pool. MJI pointed out that the new grant (and the old) has formula funding for secretarial effort at the level of 0.1 FTE per PDRA position. He will raise this issue with Paul Aslin and suggest we get assigned use of one or two named individuals rather than the current unsatisfactory arrangement. 7. Comments on Perry's reprocessing notes for VEGA bid The number of versions of the pipeline has been increased from 3 to 6 . In the VEGA bid Perry wrote a section about the possible need for reprocessing of ALL the data each time a new version is released and the concommittant issues of storing ALL the old versions of the data products. To summarise the discussion: Versions 1 - 6 of the pipeline and archive are major releases. In addition there will be minor enhancements, bug fixes and so on eg. 1.1 1.2 ..... Clearly, it will be impossible and unnecessary to reprocess ALL the data everytime a minor change to some package is made, but this raises the issue of when reprocessing will be necessary and how end-to-end it has to be. Clearly, major bug fixes will necessitate reprocessing; tuning of the pipeline, particularly during commissioning and the first few months of science data, will require several iterations to get the best results. For WFCAM we propose to not release any of the data to the general community during this version 1 shakedown phase (Feb-Jul 2004) other than to WFAU for trial ingestion and shakedown at their end and to the heads of the survey teams for assessment. Further downstream, as it stands the VEGA proposal is to reprocess ALL the data with each major version release of the pipeline. However, this is not sensible if only some later part of the pipeline has changed eg. the catalogue generation. The pipeline is modular not only in its operations but also in its underpinning structure. This gives us the flexiblity to only have to rerun the parts of the pipeline that have changed and to incrementally archive only those parts of the data products that have changed. 8. AOB a. Data Processing and Storage Hardware The plan for the data processing and storage hardware is to rack mount it all in the APM building. This will have to mesh in with the refurbishment of the APM building to create more office space, being undertaken by the University. MJI will contact Paul Aslin and try and get an updated schedule for this work since some of it will be disruptive. >>>> On a similar note JRL's office and several adjacent offices in the SPO are being de-asbestoss'd in the near future. We should take this opportunity to rack mount the SUN hardware with the Data Centre archives and servers stuff and move it to the APM building for the duration and possibly longer. >>>> b. UKIDSS meeting STH reported that he had attended a UKIDSS meeting (GPS, GCS and LAS). Two points of interest: 1) GPS only plan to observe offset skies every two hours which is not sufficient. They will need to revise their plan. Some science progs have a requirement to measure JHK in sequence. If we have a data reduction requirement which conflicts with this then we need to sort this out. A possible compromise is a minimum duration single filter block. 2) A significant fraction of the time, in otherwise good conditions, will be non-photometric due to thin cirrus etc. An additional problem in such non-photometric conditions is possible differential transparency variations across the wide WFCAM field, making post facto bootstrapping, using adjacent fields, of somewhat limited utility. Accurately calibrating science data taken in non-photometric conditions will be a non-trival problem and also needs addressing at the survey strategy design phase. Continuing actions ------------------ MJI carry on with Rice compression tests: randomness of the noise JRL and stacking tests STH still needs to contact AA about use of service night JRL compare dark sky flats with the twilight ones MJI contact AA regarding new plan for survey manager (2 x 0.5 posts) New actions ------------ DWE test Ian Bond's software on data with different seeing and offsets. DWE encourage IAB to modify his software to work on MEFs. JRL produce simulated data for end-to-end tests of pipeline using CIRSI data but with WFCAM-style FITS headers STH investigate SDSS software for generating totally synthetic data DWE use WFC data to simulate undersampled offset data for interpolation and difference imaging tests STH prepare pre-meeting documents for the Edinburgh WFCAM meeting MJI JRL All read pre-meeting documents for the Edinburgh meeting PSB find out what quality control measurements will be required during the in-lab commissioning for VISTA MJI find out timescale for APM building refurbishement and organise skip for junk removal in advance of PSB rack mount JRL's SUN equipement and move to APM building