From nch@roe.ac.uk Mon Jun 7 09:48:27 2010 Date: Fri, 4 Jun 2010 17:29:52 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren , Tom Kerr Subject: WFAU VDSF Science Archive weekly project meeting minutes, 4/06/10 Minutes of WFAU VDFS Science Archive meeting: June 4th 2010 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RSC, ETWS, RPB, RGM, NJC, AL Apologies: JPE, KTN, MAR, MSH *NB DoNM: 10am, Friday June 18th 2010 in the VISTA Hut; NJC in the chair* Actions discharged this week: ----------------------------- ACTION: NCH to brief NJC and to get included an agenda item on ESO-SAF data delivery Discharged ACTION: RPB to reinvoke the VSA metadata mirror cronjob Discharged (by ETWS) Actions partly discharged but continuing: ----------------------------------------- ACTION: ALL to review new hardware requirements/procurement over the next couple of weeks Proposal is for one new curation server and one new server dedicated to web application running (though not necessarily a web server). This will be discussed with KTN next week. ACTION: RSC & NJC to document new operational procedures on the TWiki Rough notes done but needs review by NJC and polishing for ops use. CASU have released some prototype tiles to WFAU for ingest tests. ACTION: ETWS to try some test ingests of VIRCam tiles. ETWS has started to look into this as a top priority Actions carried forward from 28/05/10 meeting: ---------------------------------------------- ACTION: NCH to fill out a Q2 2010 plan and continue with the progress monitoring for the time being. Continues ACTION: RPB/KTN to order several crates of LTO-4 tapes. Continues ACTION: ETWS to enhance the early CUs to cope better with CASU's process versioning Continues Specific points and new actions: -------------------------------- Project management: RGM and KTN attended a meeting at ESO to discuss the VDFS-related portion of the VISTA compensation package: Mike Irwin and Colin Vincent were the other members of the UK team. ESO Council will consider STFC's proposed package next week, and if its overall shape is approved, further definition of the VDFS-related part can take place, following up the discussion at ESO this week. The one definite outcome from the meeting was that ESO said that they have no objection to WFAU uploading data products to the SAF on behalf of the PSPIs, contrary to earlier indications. Regarding this last point, NCH pointed out that the timing here is particularly unfortunate, since when WFAU approached ESO-SAF over a year ago when there was time to engineer a decent automated dataflow solution, they weren't intereseted, and now the PSPIs priorities presently lie elsewhere (see next para) so ESO's priorities are now coming into conflict with those of the PSPIs. NJC distributed the minutes of the telecon with the PSPIs earlier in the week, at which the clear priorities were more frequent and flexible catalogue releases (apparently mainly for QC), and flexibility in supplying user-defined products for inclusion in the archive. None of the PIs seem particularly concerned about the mechanics of delivering products to the ESO-SAF (although many are concerned about timescales/schedules). The team agreed to try to service the more frequent release requirement initially by allowing more flexible free-form SQL access to the VSA ingest DB in the first instance (provided MAR/RPB are happy to make the appropriate mods to the user permissions and User Interface) to see how things go. Regarding ESO-SAF interfacing, it seems to be agreed that some kind of MOU is required as a first step. ACTION: TBD to draft and MOU on behalf of PSPIs for ESO-SAF deliveries WFCAM & VISTA updates: Nothing of note this week. Comments and issues arising from CASU minutes: The team noted the minutes of the meeting of Wed 1st June, in particular the liklihood of new keywords coming through in the VIRCam files, and the relationship between tile catalogue files and their unfiltered (as opposed to filtered) progenitor frames. Networking: ETWS noted that the latest available processed data products from CASU are currently being transfered. WSA/VSA Operations: RPB reported: "VMC release continues, though progress has been made, both with optimising the operations and fixing problems caused by certain framesets. Cross neighbour tables for the VHS with UKIDSS LAS (DR4 and DR7) have been created. Will be copied over to the latest VHS release database early next week." ETWS reported: -- Updated the cronjob to create VISTAPROPRIETY on a nightly basis. -- Created a set of new WSA browser pages to reflect latest non-survey releases. -- Installed latest versions of 3rd party software on shepseskaf. RSC asked RPB to set running the deblended parameters fix to the non-survey detections on the new curation server this weekend since there's a suitable quiet period in WFCAM processing. Hardware and Systems: MSH noted: RSC noted: "ETWS has finished installing the latest systems software on our new curation server shepseskaf. I've verified that our curation software is running correctly, though need to do further tests to make sure that the new v2.3.1 PyFITS doesn't cause us the same problems that other v2.x releases have thus far. The instructions to get a user up and running on the new server are on the TWiki under CurationSoftwareInstallation that explains that the user .login file needs to be updated to the latest one in SVN to include shepseskaf in the list of 64-bit servers to make use of the correct libraries including Starlink. Also, as shepseskaf is not on the SRIF network, access to the SVN repository has been setup by MSH via a proxy, so users' .subversion/server files need to be updated with the proxy settings - just replace the file with the one in SVN as described in the same TWiki page." NCH noted that the power to the site is going down on June 11th for 30mins somtime in the period 17:00 to 20:00, and we should probably put a note on the webpages and email vsa- and wsa-announce. RPB wondered if the UPSs might keep the home fires burning, but there is uncertainty about the network infrastructure (shurely shome mishtake?). ACTION: RPB to ask ITSG about the impact of site power-downs and if necessary inform the community via our webpages and announce lists of any interuptions to normal service. Software: The indefatigable RSC noted: "I've refactored the BestMatch table creation stage of CU6 to aid performance profiling so that we can identify exactly the slowest stages and not get distracted by aspects that are already efficient. We've now optimised the database queries; NCH's SQL profiling resulted in a factor of 2 speed up for the VVV non-correlated synoptic survey case, and RPB's suggestion of a synopticID index made the VMC correlated synoptic survey case a factor of almost ten times faster. This code is now entirely limited by the co-ordinate transformations at C-level that, although individually fast, are called more than 1e9 times. It may be possible to speed up the individual calls by converting the Python-C interface to use ctypes, but in the long run maybe the best we can do is to reduce the number of calls with a new algorithm proposed by NJC. However, this stage is now performing sufficiently fast following the database query optimisations that have now made the code CPU-limited and thus the new curation server may help here too. The current VISTA releases have revealed a few bugs in the synoptic curation software, mostly to do with the lack of date ranges applied to various parts of the code. These have been mostly fixed by NJC save for CU16, which now also needs to make use of date ranges for cross-matches with the detection table. Also NJC noted that CU6 was not making use of the higher order WCS co-efficients for VISTA data that will affect earlier releases up to the VVV, but has now been fixed for the VMC and future releases. I've experimented with speed ups to the CU4 ingests; the TABLOCKX hint suggested by NCH is not valid for BULK INSERTs under SQL Server 2008 as apparently this is now the default behaviour for TABLOCK. Supplying the ORDER hint by primary key failed, possibly due to the mislocation of the detector-level default rows. Currently this implemented as a (multiframeID, extNum) order. I've included an option in NonSurveyRelease to include non-registered non-surveys in the automatic selection of non-released programmes. However, more work is required here to make this useful, by breaking the releases up into effective and manageable date ranges. Also the NonSurveyRelease script now updates non-survey schema at the end of each release so that the documentation is updated for the schema browser, as the new AutoCurate process modifies the neighbour table join criteria. This is also true for the regular VSA survey releases, where we must remember to manually update the schema documentation." NCH thanked the team for their hard work in optimising the later curation stages of the VSA synoptic surveys - this should help improve the release frequency. Survey Data Release: UKIDSS LAS SH informs us that DR8 eyeballing may be finished by the end of June, so maybe a new UKIDSS release sometime in the first half of July...? Non-survey Data Release: No news this week. Astrogrid deployment & Data Analysis services: Nothing to report this week. Miscellaneous: Nothing else this week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================