From nch@roe.ac.uk Fri Feb 27 17:10:32 2009 Date: Fri, 27 Feb 2009 16:18:37 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 27/02/09 Minutes of WFAU VDFS Science Archive meeting: 27th February 2009 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, NJC, RSC, PMW, MSH, MAR, RPB Apologies: JPE, JMS, AL, LGR, RGM DoNM: 10am, Friday March 6th 2009 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: ALL to look at JENAM programme and consider possible attendance Discharged ACTION: RPB to scrub all reprocessed catalogue data from the *Detection tables in the WSA ingest DB. Discharged ACTION: All interested parties to get together Tues 24th at 3pm in Crawford Meeting Room #2 for a clustered DB brainstorming. Discharged; plan is to do some initial trawl and crossmatch tests using the SSA & 2MASS Actions partly discharged but continuing: ----------------------------------------- ACTION: ETWS to parse that latest keyword info into the VSA metadata schema files. Continues ACTION: NCH to assign U/08B/14 H2/K frames to the GPS in the WSA. Continues Actions carried forward from 20/02/09 meeting: ---------------------------------------------- None this week Specific points and new actions: -------------------------------- Project management: Nothing new this week. WFCAM & VISTA updates: Nothing new to report this week. Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: ETWS noted that SDSS DR7 is *very* nearly copied over from UChicago ... just 2 backup files out of 64 missing at their end (no response so far to calls for help...). Also one or two glitches of unknown origin on the WFAU-CASU link, but nothing major. WSA Operations: RPB noted: "Deletion of detections with deprecated >= 128 completed. Sent useless 250Gb disks back to supplier. Waiting for a butch 400Gb disk instead. Problems with thutmose becoming unresponsive. Was able to access the console this time and reboot cleanly thanks to keeping the console logged in and locked rather than logged out. Seems to be something to do with the regular system backups. It was agreed at the meeting that system backups on'y need to be run every 6 months or so, possibly by hand. Investigated WSA user permissions weirdness. Looks like permissions have been changed as a result of the WSA restore after the move to C2. ACTION: RPB to carry out a full audit of existing user accounts on DB servers to check permissions accurately match the requirements. ETWS noted: -- Transfer and ingest of re-processed 08A data for March and April has finished. -- Installed different versions of third-party software for RSC's tests of our code on 64bit khafre. Hardware: MSH noted that new processing node menkaure is up and working, although the memory upgrade hit a slight snag. Also a coup d'etat regarding WFAU membership of the IT Support Group and access to networks is now on the cards... Software: RSC continues to be very busy: "Bill Pence wrote back to me about the CFITSIO problems to explain that he intentionally changed the compression algorithm in a way that is not backwards compatible. So, although we'll hit this problem eventually, it would be best to track the Starlink version of CFITSIO from now on, presently CFITSIO 3.06 (and fpack 0.9.3), which is working fine for us on the 64-bit machine. Also, having tried various software version combinations ETWS and I have come to the firm conclusion that the epydoc parsing problems were indeed caused by it not supporting Python 2.6. However, this is a very minor issue and a bug report has been filed with the developer. To help diagnose the segmentation fault in the Pairing C module, ETWS kindly upgraded the software on 32-bit sneferu to the same versions as 64-bit khafre, which revealed the exact same problem. This indicated that it was not specific to 64-bit systems and instead was a software related problem, which we eventually narrowed down to it being a problem with GCC version 4.3.3. GCC was upgraded from 4.1.2 (the standard Debian version, which we also use on the 32-bit servers) because of problems with the 64-bit installation that the new version overcame. So we now need to try GCC 4.2.4, to see if that version both works on the 64-bit system and does not cause segmentation faults in the Pairing C module. If we can't find a suitable version of GCC then we'll need to refactor the module to work around the problem: either compile it as an external application, change the Python interface to use ctypes, or write a C++ version. I've performed further tests on the Python version of the pairing module, which is producing identical results, but was initially 200x slower than the C module. This is not surprising since the algorithms were copied directly from the C code to avoid introducing new bugs. Making the code more Pythonic only got as far as making it 30x slower than the C module. Following profiling I've identified various ways to make it faster, but it won't ever be a suitable replacement for the C module unless it can be re-written to be fully vectorised." NJC has been working on CU13/14, improving the speed of the matching between bright stars in the deep stacks and intermediate stacks for the purposes of calculating the ZP and recalibration. While testing this, also found that fitsio_stack uses huge amounts of memory when the number of input stacks becomes large, so has been modifying CU13 to build up deep stacks in parts. MAR has been concentrating on the web server migration ot 64bit horus. Survey Data Nothing to report this week. Non-survey Data Release: Nothing of note this week. Astrogrid deployment & Data Analysis services: Nothing new this week. Miscellaneous: Chris Davis from JAC has asked for WSA update input into the imminent UKIRT Newsletter - MAR kindly volunteered to have a think about some possible material. RSC noted that Ken Smith from Queens University Belfast is interested in browsing our codes for his PanSTARRS work ACTION: RSC to organise username/permission for Ken Smith to access the VDFS SVN projects. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================