A Diary of Events for April --------------------------- 1st April - JPE raises issue of the arrangement between ESO and PPARC which states, under the WFCAM Surveys heading, that "A copy of the raw survey data will be archived in the ESO archive and will be public to the ESO community at the same time as the UK Community". Who is responsible for sending this to ESO ? What medium do they want it sent on ? Neither JAC nor CASU have budgetted for this since the requirement is a new one. As a cost-neutral alternative can we pursuade ESO to simply mirror the raw archive here ? 3rd April - group meeting to finalise, among other things,management issues. 14th April - summary of work package spreadsheets, quarterly deliverables and so on, was made available. 15/16th April - MJI and JRL attend WSA CDR in Edinburgh and among other things discuss and agree to plans for summit hardware. In particular the plan is to use raw SEFs written from the front end of the CASU summit pipeline to tape (Ultrium LTO-2 one per detector) as the transfer medium between JAC and CASU. Ingestion in Cambridge will verify and convert to MEFs before further processing in Cambridge. Tapes will come in multiples of 4, roughly 1-2 lots per week when WFCAM is running on telescope and will also form basis of Cambridge offline raw archive backup. 17th April - group meeting to discuss above and other CDR-related things. 22nd April - possibility of standard 2x5s offset integrations rather than single monolithic 10s confirmed and hence potentially twice as much raw data to process and store (FITS U*2 possibility, or equivalent via integrated (almost) lossless compression, could mitigate against this). Consequences of this are being explored. Connectivity problem with main UKIRT archive at JAC solved. 24th April - JRL points out that if cross-talk between detectors is a problem then the parallel processing plans for the summit pipeline will not be able to easily deal with it. 25th April - STH and MJI prepared cost spreadsheet for ESO deliverables in consultation with JPE. 30th April - crosstalk module finished. Discussions between JRL and Paul Hirst regarding the issue of grouping wfcam observations by tile. Paul will investigate changes to the observation preparation tool so as to trigger a keyword to be written. Discussion with Paul about how best to maintain the pipeline algorithms between the summit and Cambridge. JRL will investigate ways to make the core perl routines less dependent on cirdr system information so that they can be called directly from orac-dr. JRL will also contact Frossie to get a connection to the progress database, which is held on a different server. 1st May - JRL did some tests and made some notes on the results of using various compression alogrithms. 2nd May - group meeting to discuss assorted issues and report monthly progress on WP tasks. Extracts from minutes of 030417 meeting --------------------------------------- 1) Actions from previous meeting 2) Comments on WFAU minutes and CDR papers 3) Report from Edinburgh CDR In the Critical Design Review, document 7, there is a brief section on photometric calibration plans. STH should have a look at this (liaising with NCH) and investigate if using the 2MASS model is sensible. MJI pointed out that WFAUs Version 2 now seems to end after Q1 of 2005. He thought that this was probably related to aligning with the grant funding. JPE queried this since V2 is supposed to end Q4 2004 and that this should not have changed (JPE will investigate). Ian Bond showed some demos of difference imaging software using the UKST H-alpha survey. DWE should contact him to see what he's done (and read 2001MNRAS.327..868B) and try and get a copy of the software. Interpolation schemes may still be an issue. This has an impact on WP 15. MJI got the impression that WFAU expect the CASU-supplied mosaicing software to be able to cope with "arbitrarily large mosaics" when it is run as part of the WSA. Overlap calibration and several other issues are implict here which MJI thought were WFAU responsibilities. MJI to clarify this. Eckhard Sutorius has been investigating the data transfer bottleneck between ROE and Cambridge (ftp-like from Solaris ftp server here gives 1 MB/s; scp from apm3 gives 2 MB/s - whereas MJIs scp test from Leicester to here gave 4-5 MB/s). This is probably related to the firewall that they have. Eckhard suggested that the default buffer TCP-IP buffer size be increased from 64KB to 256KB. It was felt that putting a fast linux machine outside their firewall and repeating the test would be useful. It was generally thought that this was an ROE problem. Help from CASU would be available if they ask for it eg. buffer size increase. Luc Simard (CADC) pointed out the use of Rice compression (built into CFITSIO) MJI and JRL did some tests and were impressed with the ~ 4 achieved. Although lossy with real nos. compression, it is not lossy with integers and is much faster and compresses more than gzip or bzip2. There are CFITSIO routines (internal and external eg. imcopy) for this such that the header remains uncompressed AND the level of loss is selectable. STH was not too sure about this since it would be unlikely that any (current) image display programmes would be able to cope with these files (true but they could be Perl-wrapped). MJI/JRL/STH need to investigate this further for generic CASU data storage issues. There was some concern about the naming and propagation of the master calibration files. More thought is needed here eg. if more that one master dark, flats, etc. are generated in a night, ditto the skys generated throughout the night. There was some discussion regarding the roles of the JAC-based UKIDSS Survey Manager and the Survey Scientist (SJW). Where are the responsibilities split particularly with regard to survey progress/ planning tasks? Do we know who will be the survey manager? DWE was asked to contact Steve Warren to try and clarify this. 4) Results and implications of JAC summit hardware review The hardware setup was explained. There would be 4 DAS systems and 4 data processing systems, one for each detector - plus one spare and a JAC real time processing system. Each processing unit is a PC + disk + tape unit. The 4 streams of data will be independently processed and independently output to tape. For export to CASU the summit pipeline will dump the raw SEFs to tape. JAC will dump the raw NDFs to tape for their local copy. The issue of the observer comments that go into the summit database was raised. These should go into the FITS headers. Where ? At the summit (probably not) at ingestion and conversion to MEFs in Cambridge (probably since we will mirror the DB here). JRL to ponder. <<<< The tape systems will be Ultrium LTO-2 with a native capacity of 200 GB. and a sustained data transfer rate of 100 GB/hr. We would get the tapes roughly on a weekly cycle in batches of 4-8. The tapes cost $69 each, a third of the cost of the IDE option. Individual tape drives cost about 6k pounds each and can be combined in a "library" jukebox system based on 8 tape cartridge cassettes to give 1.6TB capacity per cassette. Cambridge would probably buy a 2 drive jukebox system and also use it for backups. MJI and JRL were satisified with this solution since it automatically gives us an offline raw archive solution. The only caveat is reliability of tape. Need to instigate some tests next quarter. This is also a fallback option for sending data to ROE (if bottleneck remains or network usage costs are high) since they are using a similar setup for WSA backups. MTB will be asked to investigate these drives. <<<< 5) Hardcopy of raw data to ESO who ? how ? Network -v- tape -v- disks JPE said that he would seeing Peter Quinn (ESO) on 30th April and will discuss the matter then. It is likely that ESO will prefer IDE drives. Although there will probably be hardware costs (who pays?) associated with whatever solution used, manpower will also be a significant cost. It was felt that delivering the raw Multi-Extension FITS (MEF) files to ESO was desirable. These are only generated at ingestion of tape data into the CASU processing setup. 6) Grant status This has still not finished! PPARC et al. are sorting out problems at the detail level. It is expected that this will be finalised in time for next months pay deadline and backdated to 1st April. 7) Management documentation and tasks/deliverables This was combined with 8. 8) Update on work done for WPs STH said that the documents have been time-stamped and top level copies made available on the web page. Updates and alterations will be carried out in a systematic way. It was agreed that before each of these meetings we would report any progress on the work packages to STH. He would update the documents and give an overall report on progress and point out any problem areas for discussion. STH asked about naming conventions for the documents. JPE pointed out that ESO have a naming/signing convention and won't accept any documents that don't follow this. WFAU have decided not to follow this convention for philosophical reasons. JPE would like us to follow the ESO path. 9) Status of web pages DWE reported that the web pages have been reorganized. No password is now required for the pages except for an internal section intended for Cambridge only. The other main change is that the diary has been moved from the front page to its own page. 10) Group name No consensus has been reached about this. MJI (although he wanted a name change) pointed out that there were a lot of consequences with a name change and that a number of documents and diagrams would have to be altered. Some of these don't originate from our group implying repercussions in various (unforeseen?) places. This item will be deferred until a general group meeting is held when the official grant announcement arrives. 11) ESO deliverables/responsibles - what do we need for PDR ? There was a lot of discussion regarding what documents were needed for the Preliminary Design Review for the VDFS. Filling these documents will be the responsibility of PSB, with help from various others, and needs to be completed by 1 July. The three doorstops are: DFS user requirements document Calibration plan document Data reduction specification document Will Sutherland can help PSB with the science case summary and system properties with input also from the WFCAM docs. The WFCAM pipeline CDR document can also be used as a basis for the calibration plan and data reduction plan. JPE has provided ESO template documents to PSB which are also available on our internal page. There is an ESO requirement that all submitted documents are in PDF format. However, there is no restriction as to how they might be generated (LaTeX or Word). All VISTA documentation is in Word format. For VISTA, RAL/ATC/Durham are building the camera, while the telescope is being built by Vertex-RS. ATC are in charge of managing both projects. PSB will need to liaise with the camera team (Camera Scientist Gavin Dalton) and with Malcom Stewart and Steven Beard from the VPO. 12. AOB Extracts from minutes of 030502 meeting --------------------------------------- 1. Actions from last meeting MJI suggested that the detailed reporting regarding the work packages should only be done once a month prior to the monthly management meeting unless there are urgent issues that require more immediate attention. A reminder will be sent our prior to the relevant meeting. 2. Comments on WFAU minutes of 030425 3. Implications of possible new readout mode MJI explained that with the proposed new readout mode the basic exposure will be 5s, immediately followed by a small shift (<~10 arcsec) and then another 5s exposure. This will be used to remove most of the "bad" pixels so has no impact on the science archive data volume (providing the shift is a constant offset => no increase in no. of confidence maps required). DWE asked about how much of an overhead this will cause. It is thought about 10%. The biggest difference with this mode will be that the data rate coming out of the DAS computers will be 2 times larger than anticipated. MJI has recomputed the expected data rate. Allowing for an average of 20s dead-time in every minute (which is possibly generous) WFCAM will generate 300 Gb of raw data in a 10hr night [r*4 x4x2kx2k x8 x60 x10]. It was felt that as it stands this is too close to the margins for cpu, i/o and data transfer thro the summit pipeline. JRL and MJI propose to convert the DAS output to 16 bit FITS (which it effectively is anyway for a 5s exposure) and then consider using lossless Rice compressed FITS as the i/o transfer medium. The total reduction in i/o thro the system would be around factor of 4 - as would the amount of data requiring dumping to tape. This also gives a similar reduction in i/o at ingestion of the data into the Cambridge pipeline. 4. Possible use of various compression algorithms JRL reported on the tests he had done on various compression algorithms. The most promising is the Rice tile algorithm built into cfitsio, which is specifically designed to work with images. Rice is much faster (~x10) than gzip and bzip2 and also gives a better compression factor. JRL explained how tile compression worked and noted that the (specifiable) size of the tile was not critical regarding compression factor. PSB was concerned if any of these algorithms were lossy. In particular, there might be subtle effects present in stacking lossy data. For integer data, Rice is lossless. The notional R*4 data from the telescope will be converted to U*2 with no loss of information with the current set up (factor 2). Rice compression will give another factor of ~2 with no loss = total factor ~4. For intrinsically real data (which is a loss already for integral data) the amount of loss can be specificied eg. 1 bit, 1/2 bit 1/4 bit and so on (JRL's report for more details). STH/MJI suggested that we do some tests with lossy compression of real data to see if it has any noticeable effect on image parameter estimation. This can be done on some Wide Field Survey data. Another test to carry out is to see if the losses are random, since this may be important for stacking. DWE suggested that the data be degraded to simulate undersampling to see if this has implications on parameter estimation after compression. Compressed data could also be used to alleviate transfer to WFAU. 5. Observing strategy meeting planned in June 12/13 at ATC Is this an observing stratgey meeting or a general ukidss meeting ? It was felt that possibly too many other topics were on the proposed agenda (eg. do we really want another Science Archive session and another Pipeline session ? - the documents are all on the web). The main purpose of the meeting should be the observing strategy ie. planning the surveys, monitoring their progress, the survey definition tool, designing processable MSBs for the main surveys, planning the commissioning strategy, calibration strategy, observing schedule and so on. 6. What's been happening about the sky flat data that AA took ? Some J-band twilight sky flats have been taken. JRL has pulled these across. Tests on these will follow. 7. Report on Ultrium tape drives MJI said that the native capacity of the LTO-2 drives is 200Gb (ie. uncompressed) and that the data transfer rate is supposed to be 100Gb/hour. There is a clear upgrade path -> LTO-3 (300 Gb) -> LTO-4 (400Gb). The drives can be rack mounted and come with fibre channel or fast scsi connectivity. Tapes can be grouped by cartridge (8, 12 .....) and then form the basis of a multi-Tbyte library system. The drives appear to be about $6000 each (probably £6000 here). HP, IBM and Seagate signed up for this protocol which implies a standard and a non-proprietary format. WFAU are proposing to use this type of system for backups so the 3 sites having the same type of mass storage tape system gives us a fallback position for data transfer from Cambridge to Edinburgh. We are considering buying 2 drives (~£12k) possibly within a library system (useful for backups of other RAID arrays). MJI to contact WFAU and see if we could get a competitive quote through their supplier. [Who is supplying the 10 drives for the summit computing ?] PSB was asked to contact our usual supplier regarding these drives and get some quotes. 8. Progress on deliverables in WPs All the people present gave a brief presentation of progress with the work packages they are responsible for. STH sent in his notes by email. PSB 1.1 PSB has been working on the VDFS PDR documents but could not find 1.2 all the information he needed on the Vista web pages particularly on 1.3 the science/user requirements side. MJI dug out the relevant most recent documents he had (after the meeting), namely vista_srd_v20 and vista_ocdd10, and put them on the internal web page before the action became official. PSB is using Word and drafting out the section/subsection structure for later fill-in. It was clear that revisiting the Vista science drivers would be useful (they are a bit out of date) to check if the current science requirements are adequate/appropriate in the light of changes since the science case was assembled. Pete to contact Will about this. STH 2. STH + MJI produced spreadsheets of the costs of the ESO deliverables and keep updating and maintaining the WP spreadsheets, gantt charts 2.5 define Observing Protocols: ongoing, progress with JRL and Paul Hirst on tile keyword propagation. 11.1 photometric standards and calibration: STH is writing general 11.2 document on photometric calibration to take up where NCH's document left off. Has contacted NCH to sort out handover of this task and should present at June workshop. Has obtained and is now examining 2MASS standard stars from 32 selected regions. Assessment of their suitability in progress. Also investigating choice of several master calibration fields for WFCAM using Globular Clusters. Has discussed with Paul Hirst possibility of observing these with UFTI/UIST. MJI 2.1 ongoing and set up to do via the fortnightly meetings, minutes and monthly reports 2.2 discussed and agreed baseline transfer mechanism of data from JAC to CASU; and outline hardware setup for data acquisition and summit pipeline 12.3 has arranged to acquire a copy of the 2MASS data products. This is about 80Gb and will be supplied asap. 13.1 devised a procedure for creating master catalogues for multi-band data and incorporated it in pipeline toolkit development software 14.1 bug fixed and enhanced current version 1.0 stacking, and mosaicing software. Propagating changes into release (ie. C) version of code. Dumped current (old) version already distributed to WFAU into CVS. 15. contacted Pierre Astier from the SNe Cosmologoy Project about acquiring the SNe adaptive kernel matching code. This is in parallel with DWE's work with Ian Bond's code. JRL 2.8 solved connectivity problem with JAC's archive server. Working towards mirroring of comments and progress database. discussion with JAC and ATC staff regarding: 2.4 (a) grouping of observations into contiguous tiles 7. (b) methods to aid in synchronisation of pipeline modules between CASU and JAC 7.8 evaluating image compression schemes to speed up data flow. 7.1 CVS software modules --> CVS repository at JAC 8.1 ongoing investigation of optimal ways of removing instrumental signature with flats from various sources (sky, dome, etc.) and sky subtraction. 8.1 devised, wrote and tested general purpose pipeline module for correcting crosstalk. 8.1 ongoing work on general purpose pipeline module for persistence. DWE 2.7 UKIDSS survey planning: altered UKIDSS survey coverage plotting programme so that user-defined areas can be added. Equivalent 2-year plan plotting programme has been written. Coverage of Data Release 1 for SDSS has been incorporated into these programmes. These plots will be placed on the web and sent to SJW. 2.6 UKIDSS survey planning/observing strategy: discussed with SJW the 2.7 responsibilities division between UKIDSS Survey Scientist and Manager ongoing, find out more. 2.9 system documentation: updated web pages 15. difference imaging: read Ian Bond's MN paper and contacted him regarding possibility of using his software for tests on image subtraction, ongoing. 9. AOB - miscellaneous MJI said that JPE had a meeting at ESO with Peter Quinn et al. regarding transfer of raw WFCAM data. Unsurprisingly ESO do not want the data delivered on tape. Transfer over the Internet is a possibility that is being further investigated. JPE had also forwarded information about a lot of relevant ESO documents on their Common Pipeline Library (CPL), contact details and a new version of their Data Interface Control Document plus assorted FITS header information. MJI noted that we already have processed raw ESO WFI data and can get ISAAC data out of the ESO archive as needed to inspect the FITS headers directly. JRL ongoing evaluation of Sybase IQ RDMS for possible use with WFCAM and other VO catalogues with a view towards the demands that will be made by VISTA impressive performance, but a price of £30k!! per CPU? was mentioned. MJI noted that our part time software engineer had produced prototype standalone software for generating colour images (PNG) from 2-3 passbands, automatically using the WCS information for coaligning and rescaling using robust level estimators. This is useful both as a visualization and a publicity tool. MJI estimated that short term management overheads due to meetings, minutes, reports have gone up by 0.2 FTE since 1st April. Need to revise upward management overheads in WP spreadsheets.