From nch@roe.ac.uk Sat Jan 15 15:04:03 2011 Date: Fri, 14 Jan 2011 17:36:01 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren , Tom Kerr Subject: WFAU VDFS Science Archives project meetings mins, 14/1/11 Minutes of WFAU VDFS Science Archive meeting: January 14th 2011 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RSC, ETWS, MAR, NJC, RPB, MSH Apologies: JPE, AL, RGM, KTN DoNM: Friday January 28th 2011 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: TBD to draft and MOU on behalf of PSPIs for ESO-SAF deliveries Deprecated owing to being overtaken by current developments... ACTION: NJC to suggest workshop at next PSPI meeting. Deprecated ACTION: RPB/RSC to implement in CU19 auto-stats on all non-indexed individual colums in release DBs with default (quick) sampling. Discharged Actions partly discharged but continuing: ----------------------------------------- ACTION: NCH to ask RPB to look into the perennial functions permissions problem. Discussed during this meeting; DRs 6 & 7 done while DRs4 and before pending; RPB will check on the necessary T-SQL scripts/users and then communicate with RSC to insert into CU19 (ACTION RSC) ACTION: NCH to contact CDS concerning copies of a few bulk catalogue datasets requested to be joined in the VSA Discharged but no response; NCH to nudge again, hence continues ACTION: RPB to check that VHS, VIKING and VIDEO have required neighbours specified for UKIDSS-LAS and VVV for UKIDSS-GPS Partial credit; it was noted that UKIDSS-UDS for VIDEO is required as well. Actions carried forward from 26/11/10 meeting --------------------------------------------- ACTION: ETWS/RPB to update WSA/VSA schemas for new keyword NICOMB Continues; also watch out for TOTEXP ... suggestion was to auto check a couple of the latest examples of both WFCAM and VIRCam MEFs. Decision made to wait for next wholesale reprocessing (version 1.1? see below). Specific points and new actions: -------------------------------- Project management: The team discussed WFAU's position following the PSPI Phase 3 meeting in Garching before Xmas (unfortunately poor weather disrupted ETWS and RPB's travel plans and precluded WFAU participation). Links to documentation and a concise summary are provided in the latest CASU minutes; the first formal request to come in has been VVV. While some uncertainties remain over the exact details of columns and headers, it seems straightforward enough for WFAU to outgest the required catalogue data into flat FITS binary tables on a frameset basis and provide a first pass to the survey PIs for spot inspection prior to bulk upload to ESO. The team decided that a generalised outgester should be written to do the job, driven by an SQL prescription (which can be fettled for each case) and using the available metadata to populate the FITS headers. ACTION: RSC (with input from MAR and NJC) to prototype a generalised catalogue outgester for ESO-SAF deliverables. MAR raised some concerns about scalability of doing this through ODBC, but it was agreed to implement via SQL rowsets in the first instance (as opposed to ad-hoc binary outgests). WFCAM & VISTA updates: No updates this week. Comments and issues arising from CASU minutes: The team noted the minutes of the meeting of 1/12/10. The only (minor) comment was a little confusion over the version labelling of current tile products, which is 1.0 according to semaphor files, while we are warned that they "are not yet v1.0" in the minutes. Also EGS informed NJC at the UKIDSS meeting that a wholesale reprocessing of the VISTA data is imminent, and that this will be v1.1 ... maybe everything (including the latest grouted tile products) will be a uniform v1.1? Networking: File transfers as up to date as they can be (see next section). WSA/VSA Operations: ETWS noted -- Created SQL schemas and parsed data for the following external catalogues: 2XMMi-DR3, 2dFGRS, COMBO17 CDFS, FIRST08Jul16, MACHO, MCPS, Spitzer SAGE & SMC, and 2MASS6x2. -- Relaxed the ingest of data with missing extensions in the metadata and catalogue ingest phases. RPB noted: "WSA data fully up to date. Most VSA data ingested up to September. October and November data recently released by CASU and copied up to WFAU. Beginning curation now. One caveat to that is the VVV. This has now finished ingesting up to the end of September. Some problems with data files to be communicated to CASU." Hardware and Systems: MSH reported that new beefy curation server userkaf is now up and running with infiniband connectivity to all except ramses9 (VVV ingest DB server); a new 100 TB (native) NAS box is on order for the anticipated deluge of reprocessed data; venerable web server thoth died yesterday so has been replace with a reconfigured hatshepsut; and finally a major private area network reorganisation will take place over the next few weeks (with djoser moving off the PAN and various other units moving to the SRIF network). MSH noted that a network switch problem that occured over Xmas is now sorted (the offending unit has sprung back into life, and we now have a swappable spare on shelf in case of further problems). NCH asked about the status of backups, and RPB noted that while there is a new LTO4 drive in the DB server backup library, the stand-alone unit for flat-file backups never materialised ACTION: NCH to bug RGM/KTN about the status of the hardware (incl. LTO4) bids put into the last IfA round. Software: RSC noted: "CU19 has been updated for the new, corrected, creation of release database table statistics - all columns for now - if we find it's too slow we can start marking up columns to exclude/include in the schema. We are now having to deal with the occasional table that exceeds 2 billion rows and with the recent VHS release we have proven that the latest SQL Server 2008 version of the BCP utility can handle such tables, unlike the previous versions, so I've also updated the database API to always produce correct row counts in these situations, using various ODBC workarounds. I've implemented the parallelisation of the creation of individual deep tile products - filtering the 6 components of the tile in parallel and then mosaicking the filtered/unfiltered tiles in parallel takes about 12 minutes per product on shepseskaf using CASU's version 1.0.11 software. CU13 will require some further refactoring to enable us to fully utilise the CPUs available through parallel creation of multiple several deep tile products. This could reduce the creation time down to under 3 minutes per product on our newest server, userkaf." NJC has been working hard on incorporating the latest CASU codes into the DB-driven pipelines. Tiling incorporating the latest grouting process is underway, and much work has been done on the list-driven remeasurement software. RSC noted that CU13 (stacking/tiling) needs a bit more work at the python scripting level for thread safety in a parallelised environment. ETWS noted that broken/absent extensions are now handled in the ingest codes by relaxing the ingest validation checks; all agreed this was OK as downstream QC should catch any funnies. MAR noted that SJW has sent a modified detector-edge flagging recipe that he would like put into the curation process for UKIDSS. Survey Data Release: NCH noted that UDS PI (Omar) has asked if DR3-constituent reprocessed frames can somehow be made available to the world, since the existing DR3 release has the old versions while the new ones are currently only accessible to the proprietary community. It was agreed that the way to do this is to bung the links into WFCAMOPENTIME and put a note on the web site pointing any interested users to this. ACTION: RSC to copy DR3 constituent reprocessed UDS frames into WFCAMOPENTIME; MAR to fettle UI and document as necessary. Non-survey Data Release: RPB noted tidying up 10A non surveys that failed to run at the end of last year: "have managed to shift a few Korean surveys out the door." NCH noted the importance of keeping our chums from the far east happy as far as possible ... Astrogrid deployment & Data Analysis services: No news this week. Miscellaneous: ETWS noted that he helped Abraham Chatzidimitriou to finalise the legacy F287 DB schema browser webpages, and MAR set up a query servlet for the new client-side GUI. SuperCOSMOS is long dead, but it's legacy lives on... ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ============================================================= -- Scanned by iCritical.