5th May 2005 Hoyle committee room 10.00 - 13:20 Present: MR, MJI, DWE, EGS, STH, WJS, JRL, NAW Apologies: RGM, PSB Agenda ------ 1. Actions from last meeting 2. Comments on WFAU minutes 3. VDMT and Q2 tasks roundup 4. Data archives status 5. WFCAM processing update 6. Optical processing update 7. FDR VDFS documents 8. Plone for software documentation 9. AstroGrid liaison update 10. AOB Minutes ------- 1. Actions from last meeting DWE have set up an internal disc resources web page. It was decided that this STH would remain text only since it would be easier to maintain. The location is http://www.ast.cam.ac.uk/~wfcam/internal/hardware/hardware.html Alternatively you can find it off the documentation web page. STH have arranged a meeting with Kona Andrews about AstroGrid deployment MR (see item 9) JRL has made the required changes in the pipeline production code ("Blanknn" FITS keywords). PSB have sorted out their flights to Munich JRL MJI JRL has started to train EGS in the use of the ING archive scripts (see item 4) RWA has not taken up the offer of help from a student from the Perse School. This would not have happened until the end of May, by which time the work should be completed. JRL has sorted out the problem with the way that the pipeline is dealing with Rice compressed data MR has set up a fault report page (see item 5) MJI has prepared a list of what Comm-I data is worth reprocessing (half of it) STH has completed the work progress part of the reporting needed for the VDMT MJI has asked JPE about the current status of the RIX Response Document. The draft of the board responses document has been sent to Michele Peron MJI has finished the quarterly plan for 05Q2 PSB sent the updated ESO documents to Michele Peron on 3rd May PSB has upgraded cass123 and RGM's machine to Solaris 10 EGS have looked into using Plone for the software documentation MR DWE has provided section links in the documentation web page DWE emailed STH for suggestions regarding sorted output from the documentation Perl script. There were no apparent short cuts, but the script was successfully modified anyway. An option now exists on the documentation web page for date-sorted output. 2. Comments on WFAU minutes The last WFAU minutes were noted. 3. VDMT and Q2 tasks roundup We were reminded that the next VDMT telecon would be this afternoon. It was pointed out that the style of monthly reporting had changed yet again and that in future when reporting effort used for a work package this would be in terms of staff weeks (where 1 SW = 37 hours). MJI then did a brief run through of everyone's primary responsibilities. EGS optical data processing (will include VST) ING archive infrastructure scripts, internal SV for WFCAM MR WFCAM data processing (STH will be the backup for this) WFCAM transfers to ESO JRL pipeline infrastructure and recoding PSB VISTA documentation (50% - other 50% system management) (in future some recoding eg. ETC Perl -> C, testing CPL interface) STH management documentation, WFCAM photometry and processing DWE WFCAM documentation and PSF development (50% - other 50% Gaia) MJI management, algorithm development, internal DV and SV for WFCAM 4. Data archives status JRL said that a WFCAM raw data archive has been set up using the Comm-2/SV data. This has been set up as a basic single table with about 30 keys. This is similar to the UKIRT archive. One of the remaining tasks is to arrange how to restrict access to the data. There was then a discussion regarding the availability of WFCAM observing logfiles and how you find out what was observed on a particular night. MJI pointed out that the index files for each night, that are created directly from the FITS headers, could be used as the basis for logfiles. These are directly available from the raw wfcam data store and could also be made available from the raw data archive web page <<<< MR reported on the transfer of data to ESO. Only UKIDSS data and calibration files are being sent to ESO. It would be convenient if there was a simple way to pick these files out. However, one of the problems is that currently the entries for standard star observations in the PROJECT header keyword are not unique, sometimes being assigned the engineering code eg. U/EC/10 and at other times the main observing programme code. It was anticipated that when the "hourly" standard stars are observed, they would be done with a separate identifier. This issue needs to be raised at the next JAC telecon which needs arranging as soon as possible. <<<< MJI said that JMI had volunteered to clone a WFCAM version of the INT WFC processed product archive (for internal use only, and mainly for troubleshooting and keeping tabs on the processing). The UKIRT on-line archive has been resurrected over the last month. The main problem has been the necessity to rename files in a more sensible fashion. RWA is still in the process of reading in the DVDs but should be finished by the end of next week. It was also stressed that the AAT raw data archive needs to be up and running by the end of the month. About a 1/3 of the data has already been read from DVD (occassional problems with scratches notwithstanding). These are unique online archives and expediting their availability is crucial to the CASU archive work. EGS has been looking at the ING archive scripts. For future transfers the Internet is being considered since the link from La Palma is being upgraded. In order to test transfer speed ING have been given an account on one of our PCs. Something along the lines of the method used for automatically transferring data to ESO might be useful for this. The task of putting the 7000 CDs from the INT archive on-line has been temporarily put on hold, due to the above work having a higher priority. 5. WFCAM processing update MR said that all the (useful) Comm-1 data has been reprocessed. This was for the nights of 8-11, 14 and 26-30 November. This still needs various checks doing before it is released to WFAU over the next week. The Comm-2/SV data has a number of problems that still need sorting out. Among these are the following: A library of master darks, flats and confidence maps is needed. A master sky will be generated for each filter for each night. STH asked about the residual variation within a night and JRL said that this was below 1%. This strategy is still being tested but if successful it alleviates the problem of sky correction (really mainly residual reset anomaly) - see later notes below. JRL noted that the Comm-2/SV data seems to be linear ie. no non-linearity correction was needed. This is due to various tweaking of the electronics that was carried out prior to on-sky Comm-2. This is very good news since the full non-linearity correction is computationally intensive (currently 30s per pawprint). There was a discussion on what observations were needed for regularly generating good master flats. This needs to be raised at the next JAC telecon. <<<< MJI pointed out that there was a large apparent sensitivity variation on all detectors, whereby the response seems to vary by up to a factor of 2. This in turn causes problems for the generation of confidence maps, uniformity of surveys, and possibly also impacts on the calibration error budget. It was noted that this problem was not widely known. It was generally felt that a new report is needed on the properties of the detectors as they currently are. It was agreed that this time only one should be written and that it again would be HTML format with contributors putting in their own parts. MJI and STH will edit the report into a consistent form. <<<< STH asked for clarification on what the current processing strategy is: Dark correction -> Twilight Flatfield -> Decurtaining * -> Sky Subtraction a. Decurtaining also corrects for the majority of the reset anomaly. b. Data from point * is used to generate the Master Sky Frames via scaled combination and then the sky subtraction is carried out by rescaling first and then subtraction. c. Sky subtraction is not currently carried out in the summit pipeline JRL said that he had rewritten the SDF to FITS converter and that it was 4-5 times faster on his laptop than the previous version. However, it had not yet been tested in the summit pipeline. MJI reported that he and JMI had done extensive I/O tests on the RAID file systems, the result being that all the CASU RAID EXT3 filestores (bar apm6_a) have been converted from EXT3 to XFS (designed for very big file systems). This has resulted in a factor 3 improvement in I/O performance giving results close to the expected data transfer rates. STH asked about the calibration files that are used on the summit. The answer was that we create them here and update the summit ones periodically. In the calibration directory only one version of each calibration file is present in order to reduce confusion. The old ones are placed in a different directory. It was pointed out that MR needs to train STH in the use of the pipeline. <<<< In future a rota system will be in place to carry out the processing. MJI will sort out getting a whiteboard for this and for general use. <<<< A colourful M51 picture is needed for publicity, MJI will organise. <<<< 6. Optical processing update EGS had been asked to process Phil James's INT WFC data covering some of the UKIDSS survey area. This has been done. The processed data is being copied and sent back to Phil. EGS noted that this had been very straightforward. This is good, since that's the goal of pipeline processing. STH said that he currently has a visitor (a student of Sofia Feltzing) who is working on processing 5 nights of INT imaging data. This was taken with Stromgren filters, so it is likely that they will have to sort out the calibration of the photometry themselves. In the near future (from June onwards) we will be getting more IPHAS data. This will consist of runs of one week of data per month until December. MJI said that the VST proposals are currently being assessed. There remains an open issue regarding advanced data products from these surveys ie. what exactly does ESO want? 7. FDR VDFS documents These were sent off to ESO on Tuesday (3rd). PSB, JRL and MJI will be going to ESO next week (Monday to Wednesday). 8. Plone for software documentation EGS gave a demonstration of a Plone version of our web site he has set up. This is currently on http://apm14.ast.cam.ac.uk/. He said that it was possible to migrate the whole VDFS documentation site onto this setup. MJI noted that we may not want to migrate all of the VDFS documentation. Meanwhile, we were all invited to explore the site and provide comments. <<<< Usernames for the site will be sent out shortly. <<<< DWE reported that the system managers were reluctant to install Plone on the Solaris system due to the additional management overhead. Apparently, the University Computing Service has looked into Plone and was not in favour of it since it had too high a management cost. EGS also showed the document search facility that came with Plone as standard. This seemed quite impressive and definitely adds something that is not currently available with out current system. The initial reaction was that Plone contains some impressive features and should definitely be given a longer trial for the software documentation, and pipeline progress web pages, fault reporting and so on. A few of the luddites among us felt that Plone was possibly a bit off-putting and maybe an overkill for the simpler tasks such as the reports documentation. However, the consensus was that it was generally an impressive demonstration. MJI also felt that the look-and-feel of the Ploned web pages was a bit cluttered and asked if the side bar information could be optionally turned off. EGS said he would investigate doing this. <<<< MR demonstrated the fault reporting web site that had been set up: http://apm14.ast.cam.ac.uk/devel/collector/. This was positively received. MJI and NAW raised the issue of backups of these web sites. Since they are not on the Solaris home disk file system network they have to be backed up specfically. NAW suggested rsnapshot as a good candidate for software. MJI pointed out that a mirror harware setup was needed as a backup since we are otherwise vulnerable to single point hardware failures. 9. AstroGrid liaison update STH reported on a meeting he and MR had with Kona Andrews, Guy Rixon and RGM last week. AstroGrid want to set up 2 tools (sextractor and hyperz) that can be used on the fly on our WFS data. The timescale for this is by the end of July. Guy, Kona and STH will have another meeting on Tuesday afternoon in order for STH to be shown how to configure cass123 so that it can do this. <<<< STH suggested that cass123 should go down in the dungeon and pointed out that the system managers would continue to manage the Solaris side of this machine. He has been volunteered to broach the relocation topic with the system managers. <<<< 10. AOB STH announced that his bet with PSB has now been paid off (bar some pud) and that a very good time was had at Grantchester last Friday lunchtime. WJS raised some VISTA items of news including the availability of a new version of the UK URD. There is a push for randomisation of the order of the pawprints for generating a filled tile to minimise the effects of persistence in deep stacks. This is an SDT issue. WJS also showed some interesting pictures of cosmic ray events on some of the VISTA detectors. EGS asked for clarification regarding the high-order products that WFAU will be providing (stacking etc.). MJI said that CASU will be providing WFAU with software to perform various tasks including stacking and mosaicing, difference imaging, list-driven photometry and so on. He also pointed out that global large area calibration of surveys is a work package that will be provided by WFAU. Continuing Actions ------------------ None! New Actions ----------- JRL include access to "observing" log index files in raw wfcam archive JRL arrange next JAC telecon as soon as possible MJI raise issue of flagging observations of standard stars and other calibration frames at JAC telecon MJI raise issue of what is needed for the generation of master flats at JAC telecon MJI produce a new report on the detector properties as they currently are JRL and illustrate with processing examples EGS MR train STH in the use of the pipeline MJI sort out getting a whiteboard MJI organise producing M51 colour picture needed for publicity EGS generate and distribute usernames and passwords for Plone website ALL explore Plone version of our website and provide comments EGS generate option for hiding side bars in Plone-generated web pages STH meet with Guy Rixon and Kona Andrews on Tuesday afternoon to configure cass123 STH approach system managers with view of placing cass123 in dungeon and it being managed by the system managers