17 April 2003 Present : MJI, JRL, STH, PSB, DWE Apologies: Attending: JPE Agenda ------ 1) Actions from last meeting 2) Comments on WFAU minutes and CDR papers 3) Report from Edinburgh CDR (JRL,MJI) 4) Results and implications of JAC summit hardware review (JRL,MJI) 5) Hardcopy of raw data to ESO who ? how ? (MJI) 6) Grant status (MJI) 7) Management documentation and tasks/deliverables (STH) 8) Update on work done for WPs 9) Status of web pages (DWE) 10) Group name 11) ESO deliverables/responsibles - what do we need for PDR ? 12) AOB Minutes ------- 1) Actions from previous meeting * PSB has liaised with JPE. Item 11 on agenda will cover this. * DWE has restored the internal directory protections * STH has produced PS versions of the Gantt chart (with help from Microsoft newsgroups). This has been placed in ~wfcam/docs/ and been agreed. * ALL the .away files have been filled in a bit more. It would be useful if bank holidays could be filled in as well. * MJI drafts of the VPB monthly reports will be made available to CASU. * STH distributed suggestions for name change (see item 10). * DWE has updated the web pages (see item 9). * ALL project documents have been checked and agreed as OK. * MJI there has been progress on JAC-CASU data transfer (see item 4). JRL 2) Comments on WFAU minutes and CDR papers The people in, and associated with, the WFAU group were described for PSB so that he could get a better idea of who does what and who to contact if needed. As there were no further specific comments raised based on the CDR papers prior to the CDR review, issues arising from the CDR are listed under the next agenda item. 3) Report from Edinburgh CDR (JRL,MJI) An impressive body of documents were presented at this meeting and are available through our web link (and on /data/cass49_b/mike/VDF*.ps). There will be a written report by the review panel later. MJI and JRL were suitably impressed by the amount of effort and preparation made for the CDR and the stepwise plans to the real WSA. The schedule is very tight but let's wait and see. The following specific issues that impact directly on CASU activities were either raised at the CDR or arose from MJI/JRL discussions at the meeting: In the Critical Design Review, document 7, there is a brief section on photometric calibration plans. STH should have a look at this (liaising with NCH) and investigate if using the 2MASS model is sensible. <<<< MJI pointed out that WFAUs Version 2 now seems to end after Q1 of 2005. He thought that this was probably related to aligning with the grant funding. JPE queried this since V2 is supposed to end Q4 2004 and that this should not have changed (JPE will investigate). Ian Bond showed some demos of difference imaging software using the UKST H-alpha survey. DWE should contact him to see what he's done <<<< (and read 2001MNRAS.327..868B) and try and get a copy of the software. Interpolation schemes may still be an issue. This has an impact on WP 15. MJI got the impression that WFAU expect the CASU-supplied mosaicing software to be able to cope with "arbitrarily large mosaics" when it is run as part of the WSA. Overlap calibration and several other issues are implict here which MJI thought were WFAU responsibilities. MJI to clarify this. <<<< Eckhard Sutorius has been investigating the data transfer bottleneck between ROE and Cambridge (ftp-like from Solaris ftp server here gives 1 MB/s; scp from apm3 gives 2 MB/s - whereas MJIs scp test from Leicester to here gave 4-5 MB/s). This is probably related to the firewall that they have. Eckhard suggested that the default buffer TCP-IP buffer size be increased from 64KB to 256KB. It was felt that putting a fast linux machine outside their firewall and repeating the test would be useful. It was generally thought that this was an ROE problem. Help from CASU would be available if they ask for it eg. buffer size increase. Luc Simard (CADC) pointed out the use of Rice compression (built into CFITSIO) MJI and JRL did some tests and were impressed with the ~ 4 achieved. Although lossy with real nos. compression, it is not lossy with integers and is much faster and compresses more than gzip or bzip2. There are CFITSIO routines (internal and external eg. imcopy) for this such that the header remains uncompressed AND the level of loss is selectable. STH was not too sure about this since it would be unlikely that any (current) image display programmes would be able to cope with these files (true but they could be Perl-wrapped). MJI/JRL/STH need to investigate this further for generic CASU data storage issues. <<<< There was some concern about the naming and propagation of the master calibration files. More thought is needed here eg. if more that one master dark, flats, etc. are generated in a night, ditto the skys generated throughout the night. JRL to engage brain. <<<< There was some discussion regarding the roles of the JAC-based UKIDSS Survey Manager and the Survey Scientist (SJW). Where are the responsibilities split particularly with regard to survey progress/ planning tasks? Do we know who will be the survey manager? DWE was asked to contact Steve Warren to try and clarify this. <<<< 4) Results and implications of JAC summit hardware review (JRL,MJI) The hardware setup was explained. There would be 4 DAS systems and 4 data processing systems, one for each detector - plus one spare and a JAC real time processing system. Each processing unit is a PC + disk + tape unit. The 4 streams of data will be independently processed and independently output to tape. For export to CASU the summit pipeline will dump the raw SEFs to tape. JAC will dump the raw NDFs to tape for their local copy. The issue of the observer comments that go into the summit database was raised. These should go into the FITS headers. Where ? At the summit (probably not) at ingestion and conversion to MEFs in Cambridge (probably since we will mirror the DB here). JRL to ponder. <<<< The tape systems will be Ultrium LTO-2 with a native capacity of 200 GB. and a sustained data transfer rate of 100 GB/hr. We would get the tapes roughly on a weekly cycle in batches of 4-8. The tapes cost $69 each, a third of the cost of the IDE option. Individual tape drives cost about 6k pounds each and can be combined in a "library" jukebox system based on 8 tape cartridge cassettes to give 1.6TB capacity per cassette. Cambridge would probably buy a 2 drive jukebox system and also use it for backups. MJI and JRL were satisified with this solution since it automatically gives us an offline raw archive solution. The only caveat is reliability of tape. Need to instigate some tests next quarter. This is also a fallback option for sending data to ROE (if bottleneck remains or network usage costs are high) since they are using a similar setup for WSA backups. MTB will be asked to investigate these drives. <<<< 5) Hardcopy of raw data to ESO who ? how ? (MJI) Network -v- tape -v- disks JPE said that he would seeing Peter Quinn (ESO) on 30th April and will discuss the matter then. It is likely that ESO will prefer IDE drives. Although there will probably be hardware costs (who pays?) associated with whatever solution used, manpower will also be a significant cost. It was felt that delivering the raw Multi-Extension FITS (MEF) files to ESO was desirable. These are only generated at ingestion of tape data into the CASU processing setup. 6) Grant status (MJI) This has still not finished! PPARC et al. are sorting out problems at the detail level. It is expected that this will be finalised in time for next months pay deadline and backdated to 1st April. 7) Management documentation and tasks/deliverables (STH) This was combined with 8. 8) Update on work done for WPs STH said that the documents have been time-stamped and top level copies made available on the web page. Updates and alterations will be carried out in a systematic way. It was agreed that before each of these meetings we would report any progress on the work packages to STH. He would update the documents and give an overall report on progress and point out any problem areas for discussion. <<<< STH asked about naming conventions for the documents. JPE pointed out that ESO have a naming/signing convention and won't accept any documents that don't follow this. WFAU have decided not to follow this convention for philosophical reasons. JPE would like us to follow the ESO path. STH will ponder this and report back. <<<< 9) Status of web pages (DWE) DWE reported that the web pages have been reorganized. No password is now required for the pages except for an internal section intended for Cambridge only. The other main change is that the diary has been moved from the front page to its own page. STH suggested that the ordering of the diary page be reversed so that when you enter the page you get the most up-to-date information. Another suggestion was to reverse the months but keep the dates in order. DWE suggested a compromise where all links to the diary page would take you to the end of the page. This was agreed as an interim measure, but we should all think about how the diary page is organized. <<<< 10) Group name No consensus has been reached about this. MJI (although he wanted a name change) pointed out that there were a lot of consequences with a name change and that a number of documents and diagrams would have to be altered. Some of these don't originate from our group implying repercussions in various (unforeseen?) places. This item will be deferred until a general group meeting is held when the official grant announcement arrives. 11) ESO deliverables/responsibles - what do we need for PDR ? There was a lot of discussion regarding what documents were needed for the Preliminary Design Review for the VDFS. Filling these documents will be the responsibility of PSB, with help from various others, and needs to be completed by 1 July. <<<< The three doorstops are: DFS user requirements document Calibration plan document Data reduction specification document Will Sutherland can help PSB with the science case summary and system properties with input also from the WFCAM docs. The WFCAM pipeline CDR document can also be used as a basis for the calibration plan and data reduction plan. JPE has provided ESO template documents to PSB which are also available on our internal page. There is an ESO requirement that all submitted documents are in PDF format. However, there is no restriction as to how they might be generated (LaTeX or Word). All VISTA documentation is in Word format. For VISTA, RAL/ATC/Durham are building the camera, while the telescope is being built by Vertex-RS. ATC are in charge of managing both projects. PSB will need to liaise with the camera team (Camera Scientist Gavin Dalton) and with Malcom Stewart and Steven Beard from the VPO. 12) AOB None - exhaustion and Easter had set in. Actions ------- STH look at CDR document 7 (photometric calibration) and contact NCH about photometric calibration plans DWE contact Ian Bond at ROE re. difference imaging and read his MN paper MJI find out what the mosaicing requirements really are and who is doing what MJI investigate image compression some more and see how we could make use JRL of it in generic CASU data processing activities STH JRL deep thought about the naming of the master calibration files. DWE contact Steve Warren to get clarification regarding the responsibilities of the UKIDSS Survey Manager and Survey Scientist (and find out what's happening re. survey planning in general) MTB investigate Ultrium LTO-2 tape drives JRL observer comment database, plan to ingest in FITS headers, where ? how ? ALL report progress on work packages prior to meetings STH naming conventions for documents DWE alter links to diary page to take you to the bottom All think about organization of diary page PSB collect info needed for VDFS documents for PDR