16 December 2003 Present: MJI, JRL, PSB, DWE, STH, RGM Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Report from Q4M2 VDMT (2nd December) 4. Report on JAC telecon 5. Report on Astrowise visit 6. Report on ESO visit 7. Progress with ISAAC reductions 8. APM building revamp - update 9. AOB Minutes ------- 1. Actions from last meeting STH has continued checking the photometric standard fields for bright star contamination. DSS images of the relevant regions have been obtained but not fully checked yet. MJI suggested using a catalogue (eg Hipparcos) to help with the checking. <<<< STH the numbering of documents is now 85% complete. STH promised that it would be completed before he goes away. <<<< MJI the information regarding the QE variations of the 4 science-grade detectors has been distributed by email. The current information is not that detailed. All are different, so significantly different colour equations may be needed for all chips if these variations persist. A list of lab tests has been produced in conjunction with MMC, JRL and SJW to be carried out in January. MJI said that he would place the list on the internal web pages. <<<< JRL has not finished the reduction of the UFTI service data - ongoing <<<< STH is up-to-date on acquiring the UFTI service data - ongoing <<<< STH has checked with SJW about the issue of manual CIRSI and FIRES comparisons. SJW is not interested in the CIRSI data and although interested, he has no time to process the FIRES data. STH checking the spatial systematics in the UFTI standards data and new test data is an ongoing task. <<<< MJI supplied WFAU with a night of processed WFCAM-CIRSI test data with updated headers and Rice tile compressed versions to test WSA data ingestion. WFAU have said that the data was fine and that it does conform to their schema. STH has updated the risk registers (long-term and short-term illness risk) and placed them on the web MJI are providing JPE with budget requirements for the next VDFS bid (Jan). STH This will be iterated upon with JPE and RGM. An estimate of the current VDFS costs is also required with the costs partitioned into 6 month intervals starting 1 April 2002 and broken down into Travel, Consumables, Equipment and Staff. This also needs to be extrapolated to end of September 2004 - ongoing <<<< PSB finalised agenda and goals for their ESO visit. See later report on JRL visit which has been placed on the internal web pages ALL sent in brief reports on progress for Q4M2 VDMT PSB found out about date for PDR at the ESO meeting (see later item) DWE asked Paul Aslin regarding the DAC election. Apparently the information regarding the nominations was placed on the notice boards and in the pigeonholes of academic staff "on the assumption that it would reach those interested". MJI said that it looked like all Hoyle pigeonholes had got one. STH sent in the written nomination for the DAC. STH duly got elected. JRL He said "Things will change". MJI MJI Attended APM building refurbishment meeting (see later item) PSB 2. Comments on WFAU minutes The only comment was regarding the data transfer rate, which raised the point that a data transfer test to ESO has been done which achieved 1 MByte/s, for several Gbytes of recalcitrant FIRES data, a minor miracle given the current 34 Mbit/s bottleneck at Garching. 3. Report from Q4M2 VDMT (2nd December) STH reported on the last VDMT meeting, noting that CASU delivered their usual WP progress spreadsheets and summary documents, and then summarised the rest of the meeting via going through the action list: * JPE wants to work more with PSB to progress the ESO documents. This is ongoing. The issue of the VISTA Science (and User) Requirements document was raised again. WJS needs to rewrite this document before the VDFS PDR. This will be raised at the next management meeting and is also tied in with the formation of the VDUC. >>>> * JPE needs to add more management documentation to his VDFS web site. RGM said that he had visited this web site and that 50% of the links were broken. * JPE will generate an updated master list of dates needed for the CASU GANTT charts. * MJI to supply WFAU with WFCAM-like CIRSI data for ingest testing (done). * MJI/STH to supply budget information to JPE (in progress). * STH/NCH to discuss calibration plan - in progress, awaiting feedback(?) * CASU to have regular telecons with JAC. Rather than be prescriptive over timetabling these it was decided that these would be organised as and when they were necessary and at convenient times (since the VDMT there have been 2 telecons with JAC) The next VDMT meeting will be on 9 January in Cambridge. After the main VDMT meeting MJI, JPE and PMW discussed VDFS budget issues. 4. Report on JAC telecon JRL reported on the last JAC telecon which was held two weeks ago. This was a short (30 minutes) meeting which mainly dealt with the state of the pipeline and what the current plan is. The big issue was how to do pipeline acceptance. How can this be done with no real data? What has been proposed was that the CASU pipelines process UFTI data for a comparison with the current JAC pipeline. Since this test has already been carried out at JAC, JRL will repeat in Cambridge using the standard pipeline here and then write a short report on both sets of comparisons in time to present at the next VDMT. <<<< The next meeting will be 9pm tonight (the main outcome of which MJI has paraphrased in the above). 5. Report on Astrowise visit MJI reported on his visit to Groningen to find out about Astrowise. Representatives from WFAU - NCH, IAB and ETWS - were also there with a particular interest in the Astrowise database operations. Ed Valentijn and colleagues presented talks explaining the Astrowise concept and summarised progress to date. The following is a brief precis of MJI's notes of the visit: The Astrowise mission is a Virtual Survey System which seems to be attempting to factor in most of the elements of AstroGrid, the pipeline processing operations at CASU, and the science archive activities at WFAU. It is a highly abstracted layered concept, all Object-Oriented, and built around an Oracle 9i databse and Python scripting language. Everything is controlled through and by the database. One of the main uses for Astrowise will be to deal with OmegaCam VST data. The user interface is via Python I/F to SQL and the database provides both an environment (for data processing) and a dynamic archive - all the way through from survey design to end products. The pipeline processing components are virtually all imported: image processing is via the Eclipse C library written by N. Devillard (ESO) wrapped with SWIG; coaddition uses Bertin's SWARP package via system level calls; source extraction is via Sextractor; uses LDAC for astrometry and photometry; Astrowise views pipelines as data administration rather than processing since everything is DB driven. CPU is via servers that through the DB schedule CPU jobs by polling known systems to see if they have spare capacity. Gbit ethernet is used for some of the interprocess communication. Much of the processing is done using 32-node Beowulf clusters (eg. 32 CCDs in OmegaCam, philosophy is 1 node per CCD). 6. Report on ESO visit PSB reported on the visit of himself, JRL, JPE and JMS to ESO, to talk with representatives of DMD (Peter Quinn, Adam D. Benoit P., Michelle Peron) about VDFS issues. The VDFS delivery schedule was discussed. Working backwards from commissioning: PAE 3 months before 1st system integration; FDR ~1 year before PAE (ie. around October 2004); PDR ~6 months before FDR (ie. around April 2004). Peter Quinn emphasised that ESO DMD produces calibration for their raw data archive, but do not currently engage in surveys, nor produce science products of the type generated by CASU. This might change in the future but would require changes to the OB preparation tools and may require the concept of linked OBs. The issue was raised if CASU personnel would be needed for the first VDFS commissioning. If this was the case then budgeting for this will be needed. VDFS should have physical presence, during `commissioning 1', manual operations, `commissioning 2' has a working pipeline (including calibration library), with 3-month separation typically between C1 + C2. The VDFS commissioning plan should be integrated with the Camera plan. Design of allowed observing templates are frozen 4 months before phase 2 proposal (eg. a dither sequence should be one template to trigger reduction correctly). Operations/Storage/disk arrays/connectivity: it is possible that in future there will be a direct fibre connect (EC `remote') in Chile. EC have a 45 Mbit/s ring in Chile/Argentina and might get 155Mbit/s -> JANET, ESO are currently using IDE disks for data transfer, there could be ~2 week lag if problems, and so need a buffer at Paranal. There is a "small" failure rate with discs, but no one present was able to quantify this. The ESO (Garching) internet connection is quite slow. They currently have a 34 Mbit/s link to the internet, but will upgrade this to 155 Mbit/s at some point in the future. It was pointed out that there is 1 Gbit/s connectivity from CASU to the SuperJanet backbone and that the 1 MByte/s data transfer from ESO to Cambridge (mentioned earlier) amounts to 25% of the 34 Mbit/s capacity. ESO are very keen for CASU to use the ESO in-house CPL (C modules and C 'glue') and QFITS libraries, although delivering complete pipeline modules based around our own software modules and CFITSIO is also not ruled out. The use of CPL was not seen as a problem, but using QFITS would require a lot of extra work since the CASU pipeline is heavily dependent on CFITSIO. Furthermore, QFITS does not have Rice tile compression, nor WCS support - though the latter is not much of a problem since the pipeline uses WCSTOOLS for most WCS calculations. CASU will request Rice tile compression as a native (transparent) format for QFITS. The PDR for VISTA is going to be April+. The documentation needed for this probably needs to be ready at least a month before this eg. mid-March. The status of other documents has implications for the VDFS ESO document deliverables. The Data Reduction document cannot be done until the CPL document is released. The User Requirement document needs the Science Requirement document (see earlier comment). This probably needs VDUC to help push things along. WFCAM MEFs: it was agreed to add ESO-archive compliant header items on-the-fly; ESO would rather NOT have checksum in the MEF (since ESO generate their own). Benoit and Adam wanted a clearer idea of what they would be getting. MJI suggested pointing them to the online example WFCAM FITS header and possibly a WFCAM dummy CIRSI example. It was agreed CASU will only present non-proprietry (ie. UKIDSS) WFCAM data to the ESO-supplied transfer tool <<<< Doxygen and Qfits have now been installed in Cambridge for evaluation. Overall, the ESO trip was regarded as very positive. PSB said "all the problems were little ones". 7. Progress with ISAAC reductions JRL reported that he was "nearly there". All the data had been checked and the missing frames recently acquired. This now amounts to some 4000 science frames (~1000 J, 1000 H, and 2000 Ks), ~25 GB of data. Different flatfielding and sky subtraction strategies are being investigated and the results will be presented early in January for CASU group discussion. 8. APM building revamp - update On Monday 1st December, MJI, PSB and Paul Aslin met up with two architects and Estate Management representatives to go over possible choices for the refurbishment of the APM building. A number of options were presented, considered and discussed, taking on board the wish of the CASU group members not to have a large open plan office area. The end result was that the architects withdrew to draw up some plans for futher discussion in January. The timescale of the refurbishment will be summer++, stunningly well matched to the current commissioning schedule for WFCAM. It was apparent that no serious thought had been given to all the computer equipment that is currently in the APM building while the refurbishment is being carried out (let alone the huge plate and film and ING archive stores). Portacabins were suggested, but the security and network access problems with this solution caused it to subsequently be ruled out. 9. AOB STH raised the issue of the reports that have to be in by the next VDMT on 9th January. STH will be between two observing trips during that week, so that these reports must be in by the 5th January. STH also said that he would much prefer it even more if they were in before Christmas. These reports are the Q4m3 monthly report and a first pass at 04Q1 deliverables. <<<< The rest of us will have a group meeting early during that week. RGM reported that Steve Warren had submitted a memorandum to the UKIRT board which had not been circulated for comments beforehand. A number of issues were raised, mainly relating to responsibility for the the design of the MSBs and lack of oversight by the UKIRT board of the VDFS work for WFCAM. It was felt that much ado about MSBs was caused by earlier misunderstandings regarding responsibilities and that most of this was cleared up at the recent UKIDSS meeting in Oxford. In future the oversight issue will be remedied by VDUC acting as a conduit between VDFS and UKIRT and that this forum will also help mitigate against further communication problems between UKIDSS and VDFS. Continuing Actions ------------------ STH finish the southern extension of the photometric standards (eg. bright star checking) STH finish off numbering of documents and email authors with the appropriate information MJI place list of lab tests to be carried out in January on the web JRL finish reduction of UFTI service data STH acquire UFTI service data as it becomes available and oversee UFTI reduction and comparison STH look at the spatial systematics in the UFTI standards data and new test data MJI iterate with JPE regarding budget requirements for next VDFS bid (Jan) STH and the spend since April 2002 New Actions ----------- MJI raise issue of WJS writing the VISTA Science Requirements document at STH the next management meeting, stressing how important it is JRL write a report on the results of comparisons between UFTI data processed using Cambridge pipelines and using normal JAC pipeline and present at next CASU meeting JRL send example WFCAM FITS header and dummy data to Benoit and Adam MJI ALL send Q4m3 monthly report and 04Q1 deliverables to STH before Christmas (or by 5th January at latest)