From nch@roe.ac.uk Fri May 13 16:08:39 2011 Date: Fri, 13 May 2011 15:28:49 +0100 From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Jim Emerson , "Noddle, Keith -- Keith Noddle" , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Bob Mann , Stephen Warren , Tom Kerr Subject: WFAU VDFS Science Archive occasional team meeting mins, 13/5/11 Minutes of WFAU VDFS Science Archive meeting: May 13th 2011 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, MAR, NJC, MSH, RGM, RSC Apologies: JPE, KTN, AL, RPB DoNM: Friday May 27th 2011 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: NCH to ask RPB to look into the perennial functions permissions problem. Discharged ACTION: NCH to ensure a clear schedule for public release of VISTA survey DBs is agreed between all interested parties Discharged/re-actioned; see Project Management below Actions partly discharged but continuing: ----------------------------------------- None this week Actions carried forward from 01/04/11 meeting --------------------------------------------- ACTION: RPB to look into automated RAID monitoring across all servers. Continues; ramses9 (most recent SW installation) has such a facility; it was noted that on the linux side monitoring is working as normal. ACTION: MSH to set up ramses10 in the cluster rack and configure as a further ingest DB server. Specific points and new actions: -------------------------------- Project management: Most of this meeting was spent discussing a proto-plan for world releases of prepared VISTA survey DBs. NCH noted that it is now very important that WFAU presses home our advantage over other "repository" as opposed to "survey science" archives with our provision of SQL DBs with richly functioned user interfaces. Obviously there are tensions with the requirements of the PIs for ESO deliverables, and possible worries over having enough time to check (and we dare say, exploit) such facilities before they go world-public for the first time. However, WFAU's service remit to the world community in delivering science-ready survey DBs must now be prioritised to enable, for example, the successful VISTA science exploitation as has been demonstrated for UKIDSS, and ultimately to deliver on the survey-science promise of the VISTA Public Surveys. (Needless to say, this will form a key part of the justification for on-going operations resources for WFAU in the next grant renewal... RGM noted that STFC have still not specified if this will be required this summer, or if it can be delayed to spring 2012). Following some discussion, it was agreed that WFAU should get the ball rolling by suggesting to the PIs in individually tailored emails that: o a public survey science DB release incorporating those data already delivered to ESO for the "April 1st" deadline be prepared and released asap after that date +6months (i.e. October 1st) o the processing version should be v1.1 ACTION: NCH to draft messages to the VISTA PIs concerning the next steps in public survey DB provision and all to comment. WFCAM & VISTA updates: No WFCAM updates this time; no news as to resumption of VISTA survey operations following the recent downtime. Comments and issues arising from CASU minutes: There were two sets of meeting minutes since our last meeting: those from the meeting of March 23rd, and from April 28th. The team noted the WFCAM GPS/UDS reprocessing issues, and are ready, willing and able to receive the new stuff when required so to do (see also Survey Release below). Regarding "v1.2" VISTA processing, more definitive/final details as soon as available would be appreciated. Networking: NCH noted that IPAC have kindly supplied bulk bz2 flat-files for the WISE preliminary catalogue, and these have been pulled across the Atlantic at a respectable 3.5 MByte/s, and checksum verified. NCH is putting the finishing touches to the schema for inclusion in the archive, and will then pass on to ETWS/RPB for ingest/deployment. The team agreed that all WSA/VSA surveys may benefit from crossmatch to this dataset in the same way as to the 2MASS PSC. MSH noted that we can publish it to the VO via the DSA if required; NCH suggested we clear this with IPAC first. RGM noted also that GALEX might be a good one to do next, and will look in to the practicalities. WSA/VSA Operations: ETWS noted: -- The 3rd party software is now updated to the latest versions on all servers. -- CU4 has now the MJD updates included and the necessary DB and FITS file updates are nearly finished, so that CU4 ingests can start at the beginning of next week. -- Several non surveys have been released by RPB and the browser pages updated. -- Data has been transferred and metadata ingested up to March 2011 for VISTA and WFCAM (apart from one day that needs to be retransferred and fixing missing keywords in the second half of the March data.) Hardware and Systems: MSH noted that new processing nodes (x2), web server and DB server have been ordered. Software: RSC reported: "Lots of work has gone into eradicating bugs from the curation software during the recent VISTA release runs and the EsoRelease script was revised and finalised to handle the VIKING release successfully. With the version 1.0 data releases completed a new vsa_1_0 software branch was created to enable us to easily re-do a release against a v1.0 schema release database. The v1.1 schema changes were then applied, the software was bug tested and revised resulting in a new release_21 svn branch. Some revisions to the detection quality bit flags for the v1.1 VISTA data (and also upcoming UKIDSS releases) were implemented - flags for sources from VISTA tiles in regions of missing detectors and also detector 16, a flag for poor average confidence level in the default aperture of the detection, also an improved saturation flag for the WFCAM (or just GPS) data - but more discussion is required before their specifications are finalised and can be applied to the data." NJC has been revamping the variability analysis to allow it to deal with very dense regions with large numbers of epochs, where currently there is a problem with the amount of memory. This has also split the code into a number of smaller functions which makes it easier to maintain and will make it easier to develop further in the future. Survey Data Release: The next UKIDSS releases were briefly discussed, following on from communications with Phil (GPS) and Omar (UDS) and also comments made in the current UKIRT Board reports. Looks like the UKIDSS Survey Head prefers a combined 10A+10B DR9, which is fine by us (indeed, less QC work for MAR by doing 2 semesters at a time). Eyeball QC has been received for 10A from DXS and GCS but not LAS; all three are awaiting eyeball QC for 10B. Regarding the out-of-sync GPS, there is the complicating issue of the photometric calibration updates prior to the next (DR8) release. This will be done as quickly as possible following release of the necessary from CASU; the team noted that Phil is assuming 2 months for the retrospective ingest and subsequent processing, but ETWS cautioned that we don't really know how long it will take (but at least it's only the photometry being changed, and this can go in parallel with other loading and curation activities). Non-survey Data Release: MAR noted that RPB has released a tranch of 10B non-surveys for WFCAM last week. Astrogrid deployment & Data Analysis services: No news this week. Miscellaneous: Nothing else to report this time. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ============================================================= -- Scanned by iCritical.