From nch@roe.ac.uk Fri Mar 6 14:49:02 2009 Date: Fri, 6 Mar 2009 13:53:12 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 6/3/09 Minutes of WFAU VDFS Science Archive meeting: 6th March 2009 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, NJC, RSC, PMW, MSH, MAR, RPB Apologies: JPE, JMS, AL, LGR, RGM DoNM: 10am, Friday March 13th 2009 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: RPB to carry out a full audit of existing user accounts on DB servers to check permissions accurately match the requirements. Discharged (mostly - see below) ACTION: RSC to organise username/permission for Ken Smith to access the VDFS SVN projects. Discharged Actions partly discharged but continuing: ----------------------------------------- ACTION: ETWS to parse that latest keyword info into the VSA metadata schema files. Continues ACTION: NCH to assign U/08B/14 H2/K frames to the GPS in the WSA. Continues Actions carried forward from 27/02/09 meeting: ---------------------------------------------- None this week Specific points and new actions: -------------------------------- Project management: PMW asked the team to fill in the progress chart for February. A general discussion took place on the prioritisation of development work for VSA SV readiness, and the plan is noted in the next section. PMW briefly noted that things have stalled somewhat on the official WFAU Rolling Grant announcement from STFC ("official" in the sense of something sufficiently reassuring to the University...). Quelle surprise. WFCAM & VISTA updates: At PMW's suggestion, the team discussed general VISTA Science Archive readiness for when SV data arrive. The plan is to get the most up-to-date processed products from CASU to prototype the metadata schema as accurately as is currently possible. MAR noted that he has started some initial interface work using existing products, and no show stoppers so far. There are a few worries over data volume of quick-look preview jpegs which will need some attention, and also the more tile-based science products likely to be prevalent in VISTA data. In any case, NCH suggested that if we can prototype flat-file access of VSA data products through a proto-interface by the end of March, that would be great. Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: SDSSDR7 files finished downloading by ETWS. RPB is close to finishing uncompressing at which point will be in a position to restore the database to one of the SQL servers. Propose loading it initially onto hatshepsut to investigate splitting the database over several volumes. NCH noted that iff it's easy to restore in it's existing form (i.e. if a sensible distribution of database files allows), it should be possible to include SDSS-DR7 crossmatches in UKIDSS DR5, which should please a few users. WSA Operations: RPB noted: "SQL Server Users: identified four superfluous users on ahmose which can safely be deleted. Rather than deleting immediately, I've simply changed the password so they'll be easy to reinstate if something breaks. Will delete once everyone's happy things are still working. Configured SQL Server Agent to allow proxy access to master..xp_cmdshell. This allows ldservro to run bcp without requiring sysadmin access. Have confirmed that sysadmin access overrides the denydatawrite setting for ldservro, so clearly a bad thing. Unfortunately, the same settings for pubdbrw don't seem to work, so WFCAMPROPRIETY update broke last night. Will investigate further. Minor change to djedefre's RAID monitor script. Instead of emailing people every day, now it will only email if there's a problem. This should prevent people from getting too used to just deleting the email without really reading it." ETWS noted: -- 08A reprocessed data transfer and ingest is nearly complete and will finish over the weekend. I will start catalogue processing and ingest as soon as everything is available. -- Moved all 08B data from the temporary storage on disk15 to the new NAS box, pami. -- The last files of BestDR7 have finally been available for copying. All 64 files are successfully copied over with a transfer rate of ~210Mbit/s per file (each was 36GB) and md5 checksums have been done. -- Installed the 3rd party software on khafre again this time with gcc4.2.4, and according to RSC this seems to work well with our code. Hardware: MSH noted some final fettling of the cluster hardware, and also that a new switch is needed as we're running out of ports on the archive private LAN. With reference to tha latter, MSH also hopes to increase local transfer to disk bandwidth to improve the CASU-WFAU transfer speeds in advance of the onslaught of VISTA data - NCH noted that any possible improvements will be very welcome. ETWS noted some frustrating issues with processing node khafre, which seems to be producing excessive logging blether to / from an unused SCSI card. IT Support are struggling to get to grips with this for some reason. Software: RSC reported: "The problem with the Pairing C module causing segmentation faults has been fixed as a result of Eckhard's successful installation of GCC 4.2.4 on the 64-bit system. Our problems with the Pairing C module were clearly due to a bug in GCC 4.3.3. This means we now seem to have a working systems software set-up for both 64-bit and 32-bit installations. I also think all of the 64-bit issues in our software are now fixed, but I will need to double check when khafre comes back on-line. So, I'm now preparing a new release_10 branch of our software from the SVN trunk, containing the necessary fixes brought about by the systems software upgrade as well as 64-bit compatibility. This will allow operations to continue on all curation servers; both 32-bit & 64-bit and with either the old or the new systems software. I'm also continuing the ground work to redesign the software to make it easily switchable between WSA and VSA operations, by continuing to upgrade older scripts to use the new DbSession database API and CLI interfaces." Thanks also to RSC for liaising with JAC over issues related to GCC funnies and CFITSIO etc. NJC noted that he has enhanced CU13 to optionally be able to break a stacking run into parts commensurate with the amount of physical memory on which the process is running. Unfortunately it seems as though the s/n in the final stack degrades as the number of chunks it is divided into - NJC plans to investigate a little further and communicate with CASU [note added after meeting: dialogue already in progress; thanks to MJI for rapid response] Survey Data As noted above, tidied up 08A data area nearly ready for forward curation (source merging & crossmatching). Hopefully those activities will start towards the end of next week. Since there has been no news concerning the status of the UDS and GPS reprocessed data from the respective survey heads, it is looking like UKIDSS DR5 will have a phase-1 release consisting of LAS, GCS and DXS; hopefully by the end of March (but no promises!). Non-survey Data Release: Nothing of note this week. Astrogrid deployment & Data Analysis services: Nothing new this week. Miscellaneous: Nothing else this week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================