From nch@roe.ac.uk Tue Jun 26 00:27:54 2007 Date: Mon, 25 Jun 2007 09:17:40 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , John Taylor , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 22/06/07 Minutes of WFAU VDFS Science Archive meeting: 22nd June 2007 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RSC, JB, NJC, PMW, ETWS, MAR Apologies: JDT, BCW, JPE, AL, RGM, JMS, MSH, LGR DONM: 10am, Friday 29th June 2007 in the Plate Library Actions discharged this week: ----------------------------- None this week. Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; will get sorted (honestly!) as part of the general ingest DB schema revamp to facilitate rapid generalised photometric/astrometric recalibration. Actions carried forward from 08/06/07 meeting: ---------------------------------------------- The following from last time continue: ACTION: NCH to discuss and review AstroWISE interfacing with MAR and JDT. ACTION: NCH to start some hard-nosed negotiations with the UKIDSS PI and CSS concerning the contents and timing of DR3. - CONTINUES; remaining problems with ZY NDR data from early 06B should be sorted quite soon now, so the schedule should become easier to tie down. ACTION: AL to check on the exact wording of the survey release policy with reference to world release of proprietary survey data. - CONTINUES Specific points and new actions: -------------------------------- Project management: PMW reported back from the VDMT, at which the subject of network transfer speeds was raised (see Networking below); other than that all is well. WFCAM & VISTA updates: JPE has emailed a note around to VISTA ESP PIs, viz: "I thought it would be an appropriate moment to update PIs on the status of VISTA's M1, which is 'driving' the VISTA schedule. As I have mentioned before the VISTA schedule shows about six months of work after M1 is accepted in Russia before VISTA is ready for the science verification period, which will precede the start of Public Surveys. Frustratingly the completion of M1 polishing continues to hold up progress, and the delivery in May was not met; they are now expecting to complete polishing by the end of July." Comments and issues arising from CASU fortnightly minutes: The team noted the minutes of the meeting of 12th June, and noted and echoed CASU's concern over network transfer speeds (see below for a few thoughts on this). Networking: JB noted: "CU1 transfers have been happening as and when there is data to transfer from CASU - we have finished with 06B and are pressing ahead with 07A. We will get to test raw speed over the next few days due to a new batch of days to transfer and no processing happening on the WFAU end of the connection, this should provide more data on what and where the problem is with transfer speeds. The new NAS box should also help with this (see Hardware below)." As regards transfer speeds, possible NFS log-jams on all the cross-mounted RAID arrays, particularly under heavy processing conditions, at the WFAU end were discussed (presumably set-up is similar at the CASU end?); JB also noted that it might be an idea to use ftp as opposed to scp to relieve encryption CPU overheads. WFAU will be very interested to hear PSB's thoughts on these issues. NCH noted a conversation with Mike Watson earlier in the week concerning the next (3rd) public release of the XMM source catalogue; the plan is to include this in the next release as an external survey joined in with the usual cross-neighbour table(s). WSA Operations: JB reported: "CU2 is now complete for 06B (excepting the missing data) and CU3 is complete save for three files that were broken (these will be ingested within the week). Ingests of data are continuing ... CU2 is starting 07A ..." JB noted that after a recent pause in transfer during which WFAU was up-to-date, some more 07A has just been checked and flagged as ready to copy so transfer/ingest has resumed. Hardware: JB reported: "Thutmose's backplane has now been fixed successfully and it appears to be working at normal(ish) speed - excepting the disk rebuild. A meeting held by Information Services (UoE) by edikt was rather interesting and some interesting contacts and things have been learnt from it, ETWS has created a twiki page with links to the talks' slides. A NAS box has turned up and (assuming it is ours!) will be installed over the next week, this will be used to replace the storage on Djoser and Sneferu and thereby give us a performance increase once we have rejigged their local disks." Software: MAR reported: "Still re-factoring and tweaking java SQL access code to make it easier to maintain and implement for VISTA. Looked into doing a web gallery of pretty images, will probably use existing Python scripts/modules which has led ETWS to doing an SCOS installation of Python on thoth." ETWS noted attending software overview meetings with NJC and RSC; bug fixes as NJC tests ingest code in a test database environment; and starting installation of third party software on thoth. NJC and ETWS noted a small "unintended consequence" (to quote Bill Pence) in the latest version of CFITSIO: "After installing the latest version (3.04) of CFITSIO it occurred that the FITS keyword EXTNAME isn't set anymore when compressing files not containing it, or just copied over without changing when compressing files containing it, resp. The older version (3.006) we used previously created/overwrote this keyword with the card entry: EXTNAME = 'COMPRESSED_IMAGE' / name of this binary table extension" and Bill Pence's reply: "This change to imcopy is an unintended consequence of some major internal changes to the image compression code. We may be able to restore the original behavior to imcopy in the next release, but I recommend that instead of using imcopy, you switch to using the new fpack/funpack utility programs (see http://heasarc.gsfc.nasa.gov/fitsio/fpack/) which are much more robust and offer more features." NCH noted that the list-driven photometry toolkit code has been delivered by CASU (many thanks to JRL and MJI). This will be integrated into the database-driven curation applications on NJC's return from honeymoon in the States. RSC has been co-ordinating several software review meetings and has also investigated the use of SVN to replace the venerable CVS source repository. In tandem, a release branch has been created in the current repository to give operations a stable code base with which to work. Software testing notes are on the internal TWiki topic at http://apache.roe.ac.uk/twiki/bin/view/WFAU/SoftwareTesting while repositry branching notes are at http://apache.roe.ac.uk/twiki/bin/view/WFAU/BranchingCVS RSC anticipates migrating the current CVS repository to SVN sometime around the middle of the coming week, after which work on implementing the recalibration-optimised schema and associated application codes will commence. Survey Data Release: A couple of users have enquired about the row x column limit in the WSA User Interface; the team discussed what the implications may be if the current limit was increased from the current 15,000,000. Non-survey Data Release: NCH asked about how we go about informing registered non-survey users of the availability of flat-file access, and suggested we formalised things as follows. When a non-survey user registers, we should set up the programme metadata and then inform them that their registration has been attended to, but make the point that from this point on it is up to them to keep an eye on the WSA monitor pages to find out when some (or all) of their data becomes available for flat file access, rather than them to expect us to let them know every time some new data comes in. For those who request a survey-like database release, we will continue (of course) to notify the PI of the availability of newly released static database products. JB noted that he intends to run CU21 next with the next SW release (just before the SVN upgrade) of the operations software (only one NS will become operational at this point but it will be a good test of the current CU21 software). Astrogrid deployment: MAR and NCH attened some of the Astrogrid consortium meeting this week. Both were encouraged by the extensive use of WFAU DAS resources being made. One interesting item emerged from the future ("AG3") plans, i.e. resource limits have meant that effort from within AG to help data centre staff in deploying their software is reduced. Some concern was expressed about this. Miscellaneous: NCH suggested early doors after work to see NJC off on his nuptial travels.