ESO/VDFS meeting 4/5 Dec 2003. Peter B, Jim L, Jim E, Malcolm S and Michelle Peron present at all sessions: Thursday 4 Dec 2003 14:00-> 16:00 VISTA/operations/surveys P. Quinn discuss VISTA-related software deliverables, discuss issues like VDFS personnel travelling to ESO/Paranal for commissioning, milestones. Present: Sandra Castra - integration The VDFS delivery schedule was discussed; PAE - 3 month 1st integration FDR ~ 1 year before PAE Jim E. needs to know schedule for 4 Feb funding round. proposal to STC for survey, took it away OTC time/`public' surveys ESO doesn't have resources do do big surveys, but do produce high-quality data stream. Will need some changes to ESO tools eg OB-building tools. Operations/Storage/disk arrays It is possible that in future there will be [Jorge Felipe] a direct fibre connect (EC `remote') in Chile. EC have 45 Mbit ring in Chile/Argentina might get 155Mbit -> JANET ESO are using IDE disks for data transfer, there could be ~ 2 week lag if problems, and so need a buffer at Paranal. There is a "small" failure rate with discs, but no one was able to quantify this. NGAS. QC1 - quick-look pipeline, operator looks at it. Commissioning - VDFS should have physical presence, during `commissioning 1', manual operations, `commissioning 2' has a working pipeline (including calibration library). 3-month separation typically between C1 + C2. VDFS commissioning plan should be integrated with the Camera plan. Process of `Paranalisation' in which the local staff make things work their way and better than before... Template freeze 4 months before phase 2 proposal preparation: Jan for April, Jul for Oct. ESO will be considering ways of doing surveys. Martin Culham will work on costs of commissioning trip. Commissioning must hit Paranal Slot or loose it. 16:00 -> ~17:30 WFCAM, fits headers etc... B. Pirenne WFCAM/VISTA data transfer UK <-> ESO (and Telescope <-> Europe). with Benoit and Adam discuss WFCM FITS. It is agreed to add ESO-archive compliant header items: rather NOT have checksum in the MEF (cos ESO generate their own). DPRCAT TYPE TECH Benoit and Adam still feel that they don't know _exactly_ what they will be getting as the only document describing WFCAM headers is Alan Pickup's which is in the context of an sdf file. JRL will produce a short document before 01-Feb-2004 where the FITS file structure and content will be described Agreed CASU will only present non-proprietry WFCAM data to ESO transfer tool (Benoit to formally request to Michele to design it). I.E. the onus is on CASU to make sure that only UKIDDS data is pushed CASU have received a draft document describing the archive client tool that ESO will provide for pushing the data across. ESO have 34 Mbit to Internet; will upgrade to 155Mbit. Adam is Mr Dictionary. Malcolm suggests DIDs go in CMM as a way of synchronizing effort. Early DIDs should be at PDR (say ESO folk). Friday: 9:00-> 10:00 Pipeline infrastucture M. Peron Get an understanding of the infrastucture that will run the VISTA pipleline; who does what in deployment of VDFS. 10-> ? CPL. ESO will do some small presentations on CPL followed by a session of questions/answers. See some C code and in particular some pipeline code built on top on CPL Have detailed discussion on some examples and the technical issues involved in shareing/reusing CPL and new VISTA reduction modules (expect CPL documentation by end 2003). Michelle gave instrument-pipeline system overview. It uses C glue not scripting language. GUI: GASGANO - what an astronomer can use `at home' like P2PP Data organizer triggers data reduction. ** TPL header keywords are used as a basic trigger for on-sky data reduction to start. TPL.NEXP gives the number of exposures in the template and TPL.EXPNO gives the current exposure in the template sequence. Basically, when these match, then the template is said to have completed and reduction starts. For surveys this scheme may not be sufficient. TPL (written by BOB) ".EXPNO ".NEXP TPL.START starting time of execution of template. The techinque to catch a template abort is to notice the beginning of a new one, which is a problem if it was the last one of a night. Classification of frames: DPR_CATEGORY == "SCIENCE" Calibrations come from local calibration database. ISAAC doesn't need flat-fielding for QC1. ** So they say. I don't believe it. Templates DON'T give fixed type of frame. So recipe needs to know how to cascade. Reduction block - row1, row2, calib..., recipe name... is human-readable file. DFO - disk full of data, need to do master flats etc 1st. QC parameters, used to check commissioning OmegaCam wanted access to database during reduction. |-> Solaris | UI -> Blackboard <- |-> HP | |-> Linux Infrastructure can parallelize by sending chips off to different processors. ** have to have a routine that splits MEFs and then reassembles them. This is partly why they like having _all_ the info in the MEF PHU. One of them (can't remember which) feels that our approach whereby general stuff is in the PHU and chip specific stuff is in the extension header is a much better solution. Plugins to CPL. You need to provide a start, a recipe & a cleanup method. Can refuse parellization if you want to keep eg a MEF as a unit. Video conference kit came on and said its IP was 134.131.1.181 Klaus talked about CPL: library of components optimized for batch processing only commonly-used operational pipelines restricted user community ESO staff writing/maintaining pipeline Instrument Consortia delivering pipeline VISTA is not contractually obliged to use CPL; in future instrument consortia will be. GIRAFFE is written in C/Python - quick 'n' dirty because CPL not ready. Use ECLIPSE software. Final version of CPL 1.0 17 December 2003. Internal utility library eg string handling v.low level. (note, because they don't use perl probably...). Pipelines are pretty much completely in C. QFITS as FITS I/O, not CFITSIO: full control at ESO. QFITS is already in ECLIPSE. Integrates with DICB. Writtine in ANSI C with `OO' flavour. Maintained on CVS, distribution via ftp.eso.org. CPL data objects: - Images - Vectors - Tables - Matrices - Property Lists (eg FITS keywords) APIs Arith Image Processing Table ops Message handling Property-List manipulation DOXYGEN documentation system (suck comments out of source) CPL web pages. QFITS http://www.eso.org/eclipse/qfits CPL workshop in ~ April 2004. CASU request (will request) Tile Compression with RICE (as a native format for QFITS). ---- Recipes CPL plugin interface ---- All jitters in one template, because template triggers reduction. PM. VDFS schedule: PAE end Jan 2006: Software + test data 3 months before PAE PDR: Doc in general v0.5, data-specification document, include eg if we need more resources. Calibration plan 0.5 don't need templates. At FDR templates/recipes relationship. Template signatures a bit later. User requirements: special preparation eg can we use P2PP? Example of new feature is VISTA needs 3 `guide' stars, need to specify that P2PP needs a survey mode. PDR April 2004 FDR Oct 2004 PAE Jan 2006 -------------------------------------------------------------------------------- Some conclusions are, we need to request to ESO Tile/Rice compression, survey-definition tools... probably in the form of entries in the VDFS user-requirements document at PDR. Doxygen and Qfits have now been installed in Cambridge for evaluation (the qfits docs need doxygen).