26th August 2004 Hoyle meeting room 10.00 - 12:30 Present: JRL, PSB, MJI, DWE, WJS, STH, JMI, EGS & RWA (after coffee) Apologies: RGM MJI welcomed Eduardo (EGS) to the group meetings and pointed out that Eduardo will be working part-time on CASU projects from the beginning of October. Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. Recent VDMT and JAC telecons 4. Upgrades to ING archive 5. VEGA meeting report 6. Pipeline processing IPHAS data 7. Update on CPL + QFITS document 8. APM building works 9. AOB Minutes ------- 1. Actions from last meeting MJI has copied the correct version of the 040706 minutes to the online location. STH has supplied MJI with the location of the FIRES catalogues MJI has completed generating the catalogues from the FIRES data, but there has been no progress in the comparison, factoring in seeing-weighted stacking is needed first, ongoing <<<< JRL have distributed the QFITS/CPL document (see later) PSB STH has received a response from Simon Dye regarding the generated MSBs from the survey heads. Only Phil Lucas (GPS) has so far done this. STH has a copy of these and will investigate their suitability. <<<< ALL some comments have come in to WJS regarding the URD (v0.7). Is this all ? The release date is 31 August, so if there are any further comments, they should be forwarded by the end of the week. MJI has managed to find storage for most of the homeless APM office equipment during the building work. This is in the link between Greenwich House and the Observatory building. MJI added that RWA has already moved into JRL's office and that MTB is in the process of moving out. MJI's new pad, where most of the computer equipment will go, will get a portable cooling unit. PSB is now happy with his bit of the Q3 plan. JRL has acquired the WFCAM acceptance test data from Paul Hirst. There are still holdups acquiring science detector test data. This awaits unpacking the WFCAM kit in Hawaii. Meanwhile, JRL will investigate the WFCAM acceptance test data from the engineering array <<<< MJI has emailed his URD (V0.7) comments to WJS. PSB has forwarded the DIC to JPE who has passed it on to Michelle Peron. STH was unsure about the status of the RIX responses. MJI will contact JPE to see if he has collated them and passed them on to Michelle Peron. <<<< MJI raised the issue of a RIX response meeting at the VDMT meeting, but this is contigent on getting the RIX response document finalised. He will raise the issue again, but the date is unlikely to be before mid-September. <<<< JRL has started updating the pipeline to more gracefully deal with lack of fringe (or suitable flat) frames, ongoing. <<<< There was a general discussion about the need for new default fringe frames for the INT WFC. The current ones are well past their sell-by date. JMI volunteered to look through recent observers logs to see if any suitable data is available. <<<< JRL is in the process of building in an option of using local versions of the astrometric and photometric calibration catalogues into the pipeline based on JMI's local catalogue access code, ongoing. <<<< (Moral: as expected, external pipeline dependency not good.) JMI have rack mounted the apm2 and apm3 general purpose data processing MJI systems. 2. Comments on WFAU minutes From the last 3 WFAU meetings, MJI was interested in the discussion regarding access to the WFCAM science archive. There was some concern about who would police the registration, authentication and data access, particularly with regard to third-party collaborations. It was felt that the latter in particular was really an issue at the UKIDSS consortium and UKIRT Board level. 3. Recent VDMT and JAC telecons At the last VDMT meeting, in lieu of the delays with the RIX response document, it was suggested that PSB should begin the Exposure Time Calculator (ETC) specification using ISAAC as a starting point. <<<< The 2nd August JAC telecon (sans MJI) consisted mainly of a discussion about details of the pipeline between JRL and Paul Hirst. The following telecon was on 23 August where there was further discussion regarding the slow reading speed of the LTO-I tapes recently received from the JAC (with UKIRT archive data on them). At the moment our HP LTO-II drive (in the Overland loader) is reading them at 1-2 Mbyte/s ie. one night of WFCAM data would take more than one day to read in. This is undoubtedly related to a tape record blocksize=512 bytes problem, which is what gtar reports the blocksize to be (instead of the usual default of 20x512 bytes). Tests using the X-ray group's HP LTO-I drive give the same problem and tests on both Linux and Solaris, using gtar and dd also give the same problem. Rewriting the same data to the JAC LTO-I tape and to one of our own LTO-II tapes using our LTO-II drive gives read/write speeds on apm6 of between 50-100 Gbytes/hour. The rewritten LTO-I tape will be sent back to Hawaii for further (compatibility) tests, after verifying that the X-ray group's LTO-I drive can read it at flank speed. <<<< It was noted in passing that since the LTO-II tape drive system has been swapped to a Linux box (apm6) the I/O speed is about a factor of two faster. [Note added after the meeting: it transpires that the WFCAM LTO-I drives are manufactured by Seagate, leading to further avenues to explore.] The news on the corrector front is guardedly optimistic, but even if it is delivered before the end of the first commissioning period in November, the current schedule will still hold. After engineering commissioning, and autoguider commissioning, before WFCAM comes off the telescope, there will be a limited series of on-sky tests, the details of which are still under discussion. The current plan is then for WFCAM to come off for December/January and go back on UKIRT in February to complete the commissioning in February/March, followed by ~1 month of science observing. WFCAM will then be next on in ~late summer ready for the beginning of the UDS observing season. This format is similar to the original plan and will enable CASU and WFAU to thoroughly shake down the end-to-end system based on the first month of survey science data. 4. Upgrades to ING archive RWA was invited to the meeting to report on various recent upgrades that have been carried out to the ING archive. Filling user requests under the old system was becoming too labour intensive based as it was on manually reading from a mix of DAT tapes, CDs and DVDs. Since the total raw size of the ING archive is (only) 14.6 Tbytes, the availability of efficient lossless Rice tile compression and the advent of relatively inexpensive commodity mass storage disk systems have made placing the entire archive online a feasible option. This is a necessary prerequisite to the desired objective of automating the data retrieval and distribution, and indeed is one of several urgent tasks that the CASU ING archive working group had previously identified. Over the last year or two this has involved reading ALL of the collection of 6600 DAT tapes onto RAID5 disk arrays. These DATs were themselves copies of copies of ...... the original tapes and there was some concern that the more popular requested tapes were reaching the end of both their sell-by-date and their working life. These 6600 tapes (the original no. of tapes was several times that), cover the period 1982-1995, and have been checked. They now take up (only sic!) 730 Gbytes and are backed up on 3 LTO-II tapes !! Mind you a battle royale is still ongoing to fix their many assorted FITS header foibles before they are Rice tile compressed. The data since 1995 has been stored on 11,300 CDs and 1800 DVDs and a similar exercise is now well underway for the CD-rom collection, assorted problems notwithstanding. In parallel with this JMI has been doing some urgent upgrades to the archive manager software collection both to make it more efficient and to incorporate the new disk-based data scheme. He has also been investigating automating calibration frame selection (it previously was done manually) and putting the onus back on the user to select the relevant subset from the suggested calibration available. All of this activity is also a necessary prerequisite for the next stage of the planned upgrades which will include developing the schema and infrastructure to support the option of on-the-fly data processing. 5. VEGA meeting report The VEGA meeting was attended by representatives from Cambridge, Edinburgh, VISTA, Leicester, and MSSL. Much discussion of management strategy ensued with JMS giving a presentation on Earned Value Analysis (EVA) which highlighted both the advantages (simplicity) and disadvantages (simplistic). There was some concern that if external factors are not taken account of (eg. the delay of WFCAM and the consequent redeployment of some staff effort), the bald graph of weighted %completeness -v- time could be open to misinterpretation. There was also some discussion of future project oversight. VDFS will press for oversight via the VISTA Board by augmenting the board membership with a VDUC-nominated representative for VDFS matters. Oversight for the GAIA part is expected to be deferred until the larger bid is completed. JeS forms for the VDFS part of the bid have been submitted to PPARC. The plan for the GAIA part is that 1.5 FTEs will be based in Cambridge and 0.5 FTE in Leicester. Among other things their main task will be to work on preparing a bid to PPARC (mainly for the development of photometric techniques, data processing operations and a real time photometry database) in time for the ESA GAIA AO early next year. It was also noted at this meeting that CASU and WFAU have expressed an interest in potential involvement with data processing and archiving any VST public survey data that UK groups successfully bid for (a requirement on any succesful bid). To coordinate UK bids WJS and SJW (on behalf of the VISTA and UKIDSS consortia) are organising a meeting of interested parties in Cambridge for 16th September. The VST will provide a timely 3-4 year window of opportunity for doing optical surveys in the Southern hemisphere, before the likely advent of one or other Southern Dark (Energy) Cameras. 6. Pipeline processing IPHAS data JMI reported that he had been successfully running the WFCAM prototype pipeline on the IPHAS data (a Galactic plane survey in H_alpha, r' and i' +/-5 degrees either side of the GP and ~220 degress in longitude). Each ~one week run of data (3 received so far for this season begining in June) of about 250 Gbytes comes on a Firewire disc for ingestion into the processing system. Over the year the IPHAS project has been running (9 lots of ~one-week runs), over 2 Tbytes of data have been imported this way, and this transport mechanism has proven a very efficient and cost-effective solution to the bulk shipping of data (with Rice tile compression you could of course get ~750 Gbytes of raw data on each disk !!) Each ~1 week run of ~250 Gbytes takes about 2 days to process using the WFCAM hardware architecture, of which ~1 day is raw computer processing. Each MEF can be processed in about 2 minutes per CPU (there are 8x this available for parallel use). This has proven to be an excellent test and shakedown, of both the pipeline and the monitoring processes. 7. Update on CPL + QFITS document Nuria McKay (ROE) and JMS have fed back useful comments on the document and Nuria has obtained a copy of the latest CPL guide for PSB and JRL to browse. MJI and JMI borrowed PSB's copy to get up to speed on the current state-of-play with CPL. 8. APM building works On Friday 27th August, Paul Aslin will have a meeting with Estate Management and the architects in order to finalise timescales. It is anticipated that building work will commence toward the end of September and be completed by the New Year (see also item in 1.) 9. AOB MJI reminded us that CASU have 0.8 of allocated secretarial effort and that we have been encouraged to make use of Jeannette Gilbert for this now that JPO has gone. STH and MJI discussed planning extra tasks for the next quarter, in lieu of some of the previously planned commissioning work, to sort out the documents needed for the UK VDFS review scheduled for the New Year. It was suggested that all review documents be sent out as pdf files, but to minimise work that the current diversity of document types be retained, providing a minimum set of style requirements are met. <<<< Due to incorruptibility JRL will be giving a poster rather than an oral presentation at the ADASS conference (24 October). JRL will sort out the poster material soon so that Richard Sword has enough time to work it up into a more professional looking piece. <<<< STH pointed out that the next VDMT meeting will be on 3rd September and asked for the monthly reports to be sent in by Tuesday (31st August). <<<< There will be a meeting to discuss proposals with the Herschel Space Observatory (far infrared and submillimetre photometry and spectroscopy) on 21-23 September. WJS will be going and will report back. <<<< WJS noted that the next VISTA Project Board meeting will be on 2nd September, the day before the next VDMT. Continuing Actions ------------------ MJI compare FIRES project reduced data catalogues with ours MJI at next VDMT raise the issue of a RIX response meeting/telecon for mid-September JRL finish updating pipeline to more gracefully deal with lack of fringe (or suitable flat) frames JRL finish building in option of using local version of calibration catalogues in pipeline New Actions ----------- JRL investigate the WFCAM acceptance test data (that was obtained from Paul Hirst) via the slow tape STH investigate GPS MSBs (and any others obtained via Simon Dye) MJI ask JPE if he has collated the RIX responses and passed them on to Michelle Peron JMI look through the INT WFC observering logs to see if any suitable data is available to form updated master fringe frames PSB look at the Exposure Time Calculator (ETC) documentation for ISAAC data and write a draft specification for the VISTA ETC MJI send an LTO-I tape written at Cambridge to Hawaii to pursue slow tape reading problem STH allocate a WP subtask for the next quarter to sort out the documents needed for the UK VDFS review and review the style requirements JRL sort out the poster material needed for the ADASS conference ALL send in monthly reports to STH by Tuesday (31st August) WJS attend Herschel meeting and report back at subsequent CASU meeting