18th April 2005 Hoyle committee room 10.00 - 12:30 Present: PSB, MR, RGM, MJI, DWE, EGS, STH, WJS, JRL, RWA Apologies: STH Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. CASU grant review outcome 4. Data archives update 5. Report from UKIDSS SciVer meeting 6. JAC telecons and WFCAM update 7. Report from VDMT meeting 8. Status of FDR VDFS documents 9. AstroGrid liaison update 10. AOB Minutes ------- 1. Actions from last meeting STH it was not known whether STH had contacted JMS regarding the misdirected links. Since it was not regarded as very important, it was decided to drop the issue. DWE has set up a new entry point to the CASU web pages, given them a small makeover and checked links are up to date (http://www.ast.cam.ac.uk/casu/) EGS has finished the calibration of the large WFS surveys. MJI has also updated the WCS information in the headers and will start updating the magnitude zero-points. STH has not been in Cambridge for most of the time since the last meeting DWE so no progress has been made on setting up the internal disc resources web page. This was regarded as being of minor importance - ongoing <<<< JRL after numerous attempts to resurrect the DVD tower, it has finally been PSB declared an ex-DVD tower. The process of transferring the ~700 DVDs of MJI data from the UKIRT and AAT archives to RAID array has been started by RWA. Problems with files names are being sorted out as copying progresses and the UKIRT archive is gradually coming back to life. It will take another 3-4 weeks for the UKIRT archive transfer to be completed. MJI neither had time to go to NAM nor write a poster for it RGM MJI has received an updated VDFS schedule. Two external dependencies are being added to highlight CASU work package progress dependency on simulated FITS files and on laboratory test data. STH have not yet arranged a meeting with Kona Andrews about AstroGrid MR deployment (see item 9) - ongoing <<<< STH has placed a copy of Simon Dye's VDUC presentation in our internal directory JRL MJI said that he has modified his development code (currently being used for catalogue generation for WFCAM transfer and ingest tests) so that the catalogue FITS keywords for unused columns are now "Blanknn" (ie. the names are now unique) plus he incorporated a few other requested table name changes (to make the 16, out of 22, character name subset more readable). JRL has still to propagate these changes to the pipeline production code - ongoing <<<< MJI has asked JRL about his desktop computing requirements and a suitable system has been identified and ordered. ALL have handed over their spare keys and they are now in a special key cupboard in the workshop. STH have updated the DRLD figures - well PSB did actually JRL PSB incorporated the new templates into the documentation and rationalised MJI the whole template set, recipes, functions. CPL plugins, DPR keyword design and so on PSB has organized the hotel for the trip to ESO to discuss the changes to JRL the CAL plan and DRLD (mid-May) including a technical discussion on CPL MJI and DICB issues. The flights have yet to be sorted out. <<<< RGM has not sorted out the installation of AstroGrid-in-a-box. Due to a lack of AstroGrid code deployment activities this is not a high priority, ongoing but subsumed into the regular Astrogrid liaison update agenda item. MR has registered for the Euro-VO workshop DWE raised the issue of the Euro-VO workshop at the appropriate IoA Gaia meeting 2. Comments on WFAU minutes There were no comments other than MJI pointed out that WFAU were again recruiting for the software position in addition to advertising a position for a data mining guru. 3. CASU grant review outcome The good news is that we have heard the outcome of the grant review and that at least some of it was positive. The panel judged that 3 of the 4 requested funding lines would be funded at more or less the requested levels bar a few caveats. The flip side was the lack of enthusiasm shown for the unique raw data archives maintained here (UKIRT, AAO and ING). This curtails any development of these systems other than for the ongoing efforts to make them all available online and clean up the many and assorted header problems, particularly in the ING archive. A puzzling decision in the light of the VO world, but not helped by the unfortunate demise of the DVD tower early this year, and the lack of a clear plan with well defined deliverables and milestones in the grant case. [MJI blames having to deal with WFCAM commissioning, ESO FDR documentation and grant proposal writing all at the same time]. 4. Data archives update RWA reported on recent activities with the archives. The JKT and WHT archives have now all been transfered to RAID arrays. The UKIRT DVDs from the tower are being manually transferred to RAID, the AAO collection will follow. The tape store in the SPO is being/will be used for storing the original tapes and DVDs for security. Discussions have taken place with Don Carlos Abrams from the ING about what medium to use in future for data transfer since DVDs are quite bulky and slow and unreliable. The ING are considering several options including LTO-II tape systems. Another possibility is to use the Internet. The link between La Palma and Tenerife is being upgraded in the near future and automated internet transfer (cf. WFCAM raw --> ESO) is a possibility. A slight snag with this method is the lack of a backup offline archive at the Cambridge end. This would have to be addressed. RWA said that there have been many problems with the ING DVDs and that some batches had a large number of failures. When this happens the ING sends replacement DVDs but this all adds to the overheads of keeping the archive viable. It was suggested that JRL trains up MR (and/or EGS) in the mysteries of the Perl Sybase archive scripts to relieve some of the archive management burden on him. <<<< RWA said that he had been approached by a student from the Perse School regarding work experience in an astronomy environment. It was thought that helping with transferring the DVD data to the RAID arrays might be a suitable task. RWA was asked to chase this up and see if the student would be interested in doing this for modest remuneration <<<< 5. Report from UKIDSS SciVer meeting RGM and STH attended the UKIDSS Science Verification meeting held at Imperial College on April 1st. The minutes of this meeting can be found on http://wiki.astrogrid.org/bin/view/UKIDSS/SciVer. This has the planned WFCAM schedule on it. Also on this page are the observing summaries from the science verification phase ie. the observing that the two Simons are carrying out now. We were all encouraged to read these. The minutes contain a summary of the schedule UKIDSS seems to be expecting for release of science verification (SV) data plus other deliverables. The SV schedule agrees with the actions from the VDUC meeting, and bar the fact that commissioning slipped 2-3 weeks, this is not unreasonable. One or two other target dates for delivery of products though seem a bit optimistic and the CASU WFCAMers are concerned about expectations being raised too high by the dates' presence on a public web page. 6. JAC telecons and WFCAM update Since the last CASU meeting there have been 3 JAC telecons: 9th March - MJI, STH at IoA and AJA, PH and JRL at JAC 17th March - MJI, STH at IoA and AJA, PH and JRL at JAC 14th April - MJI, JRL at IoA and PH, STH, SD at JAC There are still issues of processing throughput with the summit pipeline. These are actively being investigated and JRL has spent a lot of time rewriting large sections of the summit pipeline in a more monolithic fashion to try and get around some of these bottlenecks in conjunction with assorted hardware updgrades and investigations of the summit processing PCs. MJI and JRL are investigating alternative processing strategies to remove the sky creation and subtraction step since a.) this is not an ideal solution anyway, b.) the decurtaining algorithm they have developed also does a very good job of removing residual reset anomaly and c.) the latter can be run during data aquisition rather than having to wait for MSB termination to be triggered. One of the bottlenecks seems to be the conversion from SDF to FITS format. This is taking 2-6s per frame, whereas it should be taking < 1s. This may be a buffering problem with CFITSIO linked to a small blocksize or it may be a RAID controller issue. JRL said he and PH were investigating. Consideration has been given to trimming down the summit pipeline so that it only does what is required for data quality control, but this is difficult and as noted by STH impacts on observers perceptions of the processing. It was noted (see above) that the decurtaining algorithm also removes the reset anomaly, so there is no need for sky subtraction at the summit or possibly in general elsewhere. However, the decurtaining algorithm is fairly CPU intensive and needs tuning a bit to run effectively at the summit. MJI and JRL pointed out that this seems to be a robust and viable alternative that is much better than having to form local sky estimates with the inherent problems that always causes (eg. -ve halos around bright objects). Another problem relating to displaying summit pipelined data has been raised. The browser of choice there, GAIA, is not very resilient to bad data value ranges. This is undoubtedly related to not having a good enough quality bad pixel map/confidence map in place there yet and is probably caused by flat fielding bad pixels. This should be easily fixable and will be corrected when the next set of master calibration files are produced. MR reported that he had almost completed the processing for the first commissioning run. Many segmentation faults and core dumps were caused by the bad data (possibly a good stress test therefore?). In general, the pipeline is working fine. A recent test showed that there was a time-related problem with the way the pipeline is dealing with Rice compressed data. JRL said that he understood the problem and will fix it shortly. <<<< MR noted that a problem with the first commissioning run was that it contained no narrow band twilight flatfield data. Consequently no narrow band master flats could be created. Processing narrow-band data from comm-I is low priority. A summary status of WFCAM commissioning-I data reduction can be found at http://apm15.ast.cam.ac.uk:9675/casu/docs/wfcam/commissioning MJI pointed out that a fault report page would be useful. This shouldn't necessarily be a full error reporting database yet, but a simple HTML page with fault, action, who needs to do it, when was it done, comments ..... This needs setting up and maintaining. <<<< MJI and MR will prepare a list of what comm-I data is worth reprocessing (ie. potentially useful science content) after the thorough shakedown the pipeline has had during the first pass through. This will be the only comm-I data reprocessed. <<<< STH is currently setting up the secondary photometric standards system and taking lots of data for illumination correction tests. 7. Report from VDMT meeting The VDMT meeting was held by telecon on 13th April. MJI reported there were no major problems and the issue of external dependencies on the VDFS pipeline developement schedule had been raised. STH still has to complete the work progress aspect of the reporting. <<<< There was still a problem with the signing off of the board RIX response document that was now impacting on completing the changes required. MJI said that he would ask JPE about the current status. <<<< There were no quibbles regarding the quarterly progress. The plan for the next quarter is still be be drawn up. MJI said that he hoped to finish this by the end of the week and would circulate the list for comment. <<<< An interesting misunderstanding has arisen about ESO's requirements on the deliverable science products for the processed public survey VST data. ESO appear to want much higher level data products than anticipated eg. uniformly calibrated data over whole survey areas. However, this requirement on its own should not directly impact on CASU pipeline processing of VST data. The main unknowns from our point of view are the requirements on individual processed products meeting the, as yet, undefined science requirements. 8. Status of FDR VDFS documents PSB said that he is still waiting for the RIX response document to finish off many of the changes in hand. After a lot of work, we now think we understand the constraints that we have to deliver within. It was decided that the updated Cal plan and DRLD documents should be sent to Michele Peron by the end of April so that ESO have enough time to study the changes before the mid-May meeting in Garching. <<<< PSB said that the ESO workstation was up and running and that he could control the VIRCAM instruement plus VLT simulators from his office (sort of). 9. AstroGrid liaison update RGM said that although AstroGrid have regular telecons, they are unminuted. There is no recent news apart from noting that the release of AstroGrid 1.0 has slipped from 1 April to 1 May. An account (agcasu) has been set up for CASU-related Grid use. The software packages tomcat (Solaris 10) and mavan?? were mentioned as installation requirements. PSB said that he would upgrade cass123 and RGM's machine to Solaris 10 <<<< so that he (RGM) could break it. 10. AOB EGS said that Plone (http://plone.org/) was a useful tool that we should consider using for CASU. It is a user-friendly open source content management system. DWE said that the Gaia group was considering using it as the basis of their webpage and information distribution (after MR had done a demo for them). [a comment found on the website "Wicked: wiki with no wiki aftertaste!"] MJI suggested that EGS and MR package up the software documentation using Plone as a demonstration to see how suitable it is for us. <<<< DWE said that he had two requests from RGM for improving the documentation webpage 1. adding section links (easy) <<<< 2. providing a sorted-on-date version so that you can see at a glance which is the most recent document. From the way the Perl script is written, this is not entirely straightforward. This was regarded as useful, so it was suggested that DWE email STH (original author) to see if he had any quick fixes. <<<< Continuing Actions ------------------ STH set up the internal disc resources web page DWE MR arrange meeting with Kona Andrews about AstroGrid deployment STH JRL update the pipeline version of the catalogue generation software to be compatible with the recent changes made to the development version PSB book flights for the trip to ESO (mid-May) MJI JRL RGM remind Andy Batey about the installation of AstroGrid-in-a-box New Actions ----------- JRL train MR and EGS about the mysteries of the archive scripts RWA reply to Perse School student who might help with transferring the DVD data onto RAID array JRL fix compression timing problem with pipeline MR generate a fault reporting page for the pipeline and populate it MJI prepare a list of what comm-I science data is worth reprocessing MJI ask JPE about the current status of the RIX Response Document STH finish generating work package progress information and send to JMS MJI finish generating the quarterly plan for 05Q2 and circulate for comment PSB send updated ESO documents to Michele Peron by the end of April PSB upgrade cass123 and RGM's machine to Solaris 10 EGS package up the software documentation using Plone MR DWE provide section links in documentation web page DWE email STH for suggestions regarding sorted output from documentation Perl script and provide date sorted output option