From nch@roe.ac.uk Sun Nov 28 10:18:04 2010 Date: Fri, 26 Nov 2010 17:44:56 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren , Tom Kerr Subject: WFAU VDFS Science Archives project meeting mins, 26/11/2010 Minutes of WFAU VDFS Science Archive meeting: November 26th 2010 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RSC, ETWS, MAR, NJC, RPB Apologies: JPE, AL, RGM, KTN, MSH ** NB: DoNM: Friday January 10th 2011 in the Vista Hut ** Actions discharged this week: ----------------------------- ACTION: RSC to test updated Starlink on khafre Discharged; only issue was a backward step in the default Java configuration (to 1.5 from 1.6). Actions partly discharged but continuing: ----------------------------------------- ACTION: NCH to ask RPB to look into the perennial functions permissions problem. Discussed during this meeting; RPB will check on the necessary T-SQL scripts/users and then communicate with RSC to insert into CU19 Actions carried forward from 12/11/10 meeting --------------------------------------------- ACTION: TBD to draft and MOU on behalf of PSPIs for ESO-SAF deliveries Continues; on hold until at least next PSPI teleconf, and possibly until "Phase 3" workshop at ESO. See Project management below for an update. ACTION: NJC to suggest workshop at next PSPI meeting. NJC noted that he needs to liaise with JPE about the next teleconf. ACTION: ETWS/RPB to update WSA/VSA schemas for new keyword NICOMB Continues; also watch out for TOTEXP ... suggestion was to auto check a couple of the latest examples of both WFCAM and VIRCam MEFs. Specific points and new actions: -------------------------------- Project management: The team discussed WFAU's position for the PSPI Phase 3 meeting in Garching next week, noting that the agenda for the meeting currently concentrates on the minutae of the delivery process, while what WFAU really wants to know is: o exactly which products are expected to be delivered (images: tiles, paws, cals, ...?; cats: single detection, merged source, list-driven, ...?; time period: sv, dry run, P85, ...?) o who is doing what for the deliveries out of CASU, the PSPIs, and WFAU (all of which have at some time or another made statements about delivering) o what are the prospects for automated, bulk deliveries i.e. how is it to be organised. ETWS/RPB will try to raise these issues at the appropriate time. WFCAM & VISTA updates: No updates this week. Comments and issues arising from CASU minutes: No new minutes this week Networking: ACTION: NCH to contact CDS concerning copies of a few bulk catalogue datasets requested to be joined in the VSA WSA/VSA Operations: ETWS noted -- Continued to go through the list of external catalogues and creating ingestable data (reformatting, default values, htmIDs) and SQL schemas. -- Updated the monitor page creation tool to use both the VSA and the VSAVVV databases to show ingest progress. -- Enhanced the WFCAM non-survey browser pages for easier navigation. -- Went to the VST ATLAS kick-off meeting in Durham. The VST Commissioning phase will run from 06/01/11 till 30/06/11, after that observations will start in earnest. CASU will produce photometrical/ astrometrical calibrated data as for VISTA/WFCAM. Points mentioned as an add-on to existing products where a PSF image along the given postage stamps and elliptical appertures (either in the catalogue or in a list-driven tool). RPB noted: "Separate VVV ingest database created and working. Currently ploughing through existing data from the start of SV period. SV and DRYRUN now fully ingested. Starting the slightly more meaty task of ingesting P85 data. Concluded experiments with statistics. It seems the current CU19 approach of creating stats for multiple columns at a time doesn't work. Creating stats for all columns takes varying amounts of time depending on how much data SQL Server is instructed to sample. If SQL Server is not given a direct instruction it will (according to Microsoft) decide for itself how much data needs to be sampled. Running this way ran even faster than only sampling 1% of the data, yet it still seems to speed up queries. Suggest we implement this in CU19 in future. VHS release including all P85 data currently progressing. Source merging has now finished. Just waiting for reseaming to start to hopefully get some idea of how much longer it'll take." ACTION: RPB/RSC to implement in CU19 auto-stats on all non-indexed individual colums in release DBs with default (quick) sampling. RSC enquired about VSA-WSA crossmatching, and after a brief discussion: ACTION: RPB to check that VHS, VIKING and VIDEO have required neighbours specified for UKIDSS-LAS and VVV for UKIDSS-GPS Hardware and Systems: MSH noted: "Ramses9 is installed and running, all it needs are some software updates and it is good to go. It is already on the infiniband network and has SQL server installed. It will run the VVV. After some shuffling of servers on the rack Ramses10 will soon follow. Specs for a new 96TB NAS box have been sent to Keith to initiate procurement. Userkaf, the new curation machine, is having infiniband problems. The new IB drivers and software release isn't out on Debian yet, and building from scratch has failed due to linux kernel module conflicts. I've contacted the debian developers involved about the potential date for a new release, but have received no reply. I'll be testing a possible solution next week, but if that fails we will need to explore running a new flavor of linux on the server - most likely Scientific Linux (which has IB support)." Software: RSC noted: "Following NJC's work on a curation process for tiling deep stacks it has become clear that running this process in a serial manner will result in unfeasible curation times of over an hour per product (when potentially thousands of products require curation). Thankfully it turns out to be an easily parallelisable process, with memory constraints easily matched by our current hardware, whereby we can curate 4 products simultaneously on our latest 24-CPU curation server, userkaf, with each product utilising 6 CPUs in parallel to process the 6 deep stacks that make a tile product. This will cut curation times down to just a few minutes per product on userkaf and be only 2-3x slower on our next fastest server, shepseskaf, though to get these gains we need to use a curation server with at least 6 CPUs, which rules out all other servers. Also, it will help if we could dedicate userkaf to this task, as if any other processes are running at the time it will significantly decrease overall performance. In other news, I've got stuck into implementing a proper motion solution for the shallow UKIDSS surveys by converting NCH's prototype FORTRAN code for calculating the source position error maps of frame sets into C++ and shoe-horning it into existing source merging code, mkmerge. A new release branch, release_19, of the curation software has been made, following various bug fixes all related to alternating between paw-print & tile curation, and updated for the new configuration of the catalogue server cluster (following the addition of the ramses9 & 10 nodes)." Survey Data Release: Much interaction over the last few days with the VIDEO team, who have made available to us their latest stack/mosaic products for their survey and NJC is co-ordinating the incorporation of these into the VSA and the forward-curate a la UDS for preparations of a release database. A new release of the VHS is busily preparing. Non-survey Data Release: RPB will be reviewing the 10A WFCAM non-survey projects for any lingering problems. Astrogrid deployment & Data Analysis services: NCH noted that UKIDSS world-released images are now available in CDS Aladin v7 beta - e.g. via java webstart (if UKIDSS doesn't appear in the selection next to SDSS, press "reload" under user preferences): http://aladin.u-strasbg.fr/java/AladinBeta.jnlp thanks to MSH/MAR's SIA web services. Currently world releases up to DR4 are available. Miscellaneous: ETWS reported back from the VST-ATLAS meeting in Durham. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ============================================================= -- Scanned by iCritical.