From nch@roe.ac.uk Fri Sep 2 21:33:42 2011 Date: Fri, 2 Sep 2011 14:27:31 +0100 From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Jim Emerson , "Noddle, Keith -- Keith Noddle" , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Bob Mann , Stephen Warren , Tom Kerr Subject: WFAU VDFS Science Archive project meeting mins, 2/9/11 Minutes of WFAU VDFS Science Archive meeting: September 2nd 2011 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, RPB, MAR, AL, RGM, MSH Apologies: JPE, RSC, NJC, KTN, C37, 89X DoNM: 10am Friday September ** 30th ** 2011 in the VH Actions discharged this time: ----------------------------- ACTION: ETWS to enquire about availability of SDSS DR8 SQL DB from the States. Discharged; SDSS DR8 ftp'ing over as we speak. Will take the next week or three to complete, then it will be installed on latest addition to the database server cluster, ramses11. ACTION: MAR to make release status and usage stats more visible on the VSA web pages. Discharged. ACTION: RPB to review crossneighbour definitions for WISE (and at the same time) the SSA for all WSA/VSA surveys. Discharged. Actions partly discharged but continuing: ----------------------------------------- ACTION: RSC to restore temporary ingest file writing to high bandwidth shares between curation clients and DB servers. Continues; system level tests carried out by MSH now point the way forward so MSH and RSC to implement the optimal solution ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Work in progress (some problems with OS level permissions on r/o files - MS DOS strikes again) ACTION: RPB (or whoever communicates with the PIs) to keep up the pressure on world public release DB preparation by emphasising that the recently prepared DBs are not simply for internal consumption. VHS and VIKING nudged, with so far no response. The plan is to send one-month reminders, escalating to fortnightly, as any potential release dates approach... Actions carried forward from 19/08/11 meeting --------------------------------------------- ACTION: RPB to set up mirror/sync for flat file DBs using MS SQL replication. Continues; NCH noted that the MS SQL implementation may not be as streamlined as might be hoped... ACTION: RPB to try a proof-of-concept on detection table monthly partitions using the VHS to start. Continues Specific points and new actions: -------------------------------- Project management: Major discussion this week on the status of public release DBs for the Vista surveys. Too many expletives and insensitive comments to minute here, but NCH volunteered to take a look at the ensemble quality stats of the current VHS proprietary DB, and AL volunteered to get up to speed on the state of things. General feeling is that VHS and VIKING are the surveys to focus on from our perspective. WFCAM & VISTA updates: Nothing of note. Comments and issues arising from CASU minutes: No new minutes this time. Networking: SDSS DR8 copying over from JHU at the moment; UKIDSS DR6 is being copied on to a USB disk for shipping back to Baltimore (in exchange for GalexDR6 from STScI). Some discussions concerning the optimum choice of neighbour radius for the Galex crossmatch proposed 15" as a likely choice given the 5" FWHM image size, but RGM offered to dig around amongst some learned papers to check on this. WSA/VSA Operations: ETWS reported: -- Finished transferring the reprocessed WFCAM stacks and catalogues from CASU. All the GPS metadata is ingested, JPGs are still processing. -- Started BestDR8 transfer of the backup copy (64 files, ~83GB each) via ftp/wget from JHU. The average transfer speed is 2.3MB/s. -- Transferred and ingested the metadata of the VISTA July data. -- Added more functionality to the browser parser to include the new survey based views. -- Started recompiling the 3rd party software on kakai after the latest upgrade. -- Minor upgrades to different parts of the code to smoothly process the reprocessed WFCAM GPS data while working on general releases for the other surveys. RPB reported: "P86 release databases started. Initial problems with VHS (may need input from NJC), but VMC is progressing well. Problems with replication between ramses2 and ramses9 this week. After stopping and starting, it's currently turned off until I can take a good look on return from my holiday. VVV CU6 continues interminably. WISE neighbour tables defined for all surveys in VSA and WSA." Hardware and Systems: MSH noted that ramses11 is set up and ready (to host the 12TB SDSS DR8 in the short term). Interestingly, MS-DOS can't cope with RAID volumes larger than 16TB, so ramses11 is currently configured with 15.4+2.5 TB volumes. MSH also noted that a new 0.15 PB (native) NAS box has arrived, and that number cruncher kakai has had it's memory expanded to 64 GB. Finally, the latest Debian upgrades are being rolled out across the linux servers, with the usual fettling required in all third-party software installations. Software: Nothing of note this time. Survey Data Release: DR9 preparations held back slightly while GPS reprocessed ingests are finalised. RPB noted that the crossmatch runs for DR9 should not wait for the availability of SDSS DR8 - these can be done as soon as possible after release and added in. AL reminded all earlier in the week that UKIDSS DR7 should go world public this week; also DR6 should be tweaked for world public access for the GPS tables ACTION: RPB to fettle the permissions on UKIDSS DR7 (sans GPS) and UKIDSS DR6 GPS to make them world accessible. Non-survey Data Release: Nothing new this time. Astrogrid deployment & Data Analysis services: RGM noted that the current batch of MSc students finished their projects with presentations this week, and that there has been some particularly good work on web interface and column-oriented database testing. MAR noted that he's looked at the specification of the beta release of MS SQL Server 2011, which (if the propaganda can be believed) has support for column-oriented storage. In any case, given the impressive results obtained so far, there seem to be sufficient evidence to consider rolling out a terabyte, billion row scale survey database with column oriented storage... maybe deploy the SSA this way? Miscellaneous: Nothing else this time. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ============================================================= -- Scanned by iCritical.