From nch@roe.ac.uk Fri Mar 11 16:02:36 2011 Date: Fri, 11 Mar 2011 14:18:58 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Jim Emerson , "Noddle, Keith -- Keith Noddle" , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Bob Mann , Stephen Warren , Tom Kerr Subject: WFAU VDFS Science Archive project team meeting mins, 11/03/11 Minutes of WFAU VDFS Science Archive meeting: March 11th 2011 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, MAR, NJC, RPB, MSH, RGM, RSC, AL, KTN Apologies: JPE DoNM: Friday March 25th 2011 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: ETWS/RPB to update WSA/VSA schemas for new keyword NICOMB Discharged ACTION: RPB to make WFCAM 09B non-survey data world readable. Discharged Actions partly discharged but continuing: ----------------------------------------- ACTION: NCH to ask RPB to look into the perennial functions permissions problem. Discussed during this meeting; DRs 6 & 7 done while DRs4 and before pending; RPB will check on the necessary T-SQL scripts/users and then communicate with RSC to insert into CU19 (ACTION RSC) ACTION: RPB to investigate LTO4 tape library backups from the linux client side. Continues; linux client identified and will be installed for testing. Actions carried forward from 25/02/11 meeting --------------------------------------------- ACTION: RPB to look into automated RAID monitoring across all servers. Continues; ramses9 (most recent SW installation) has such a facility; it was noted that on the linux side monitoring is working as normal. Specific points and new actions: -------------------------------- Project management: NCH and AL noted that the UKIDSS+ extension proposal was nearing completion. This makes the case for a year-by-year extension to a maximum of ~4yr and has implications for continued WSA operations at CASU and WFAU. Component surveys comprise UHS (uber LAS/GCS/GPS), molecular H survey, UDS deep(er) K and DXS-like in PanSTARRS MDS. (noted here rather than under Survey Release as it's v. important; input from JPE would be particularly welcome): AL enquired as to the public release schedule for the VISTA survey DBs that are currently proprietary to the survey teams. [ Ed.: suddenly occurs to me that there may be an ESO policy on this? ] AL and RGM noted that this is an important issue for WFAU, and we should be proactive in suggesting schedules to the PSPIs. The team noted that there may be a reluctance on the part of the PSPIs to let anything other than the most polished survey DBs out into the world public domain, especially given the inevitable teething issues with DBs created so far. So lots of potential for confusion and procrastination then. ACTION: NCH to ensure a clear schedule for public release of VISTA survey DBs is agreed between all interested parties WFCAM & VISTA updates: No updates this week. Comments and issues arising from CASU minutes: The team noted the minutes of the meeting of 23rd Feb and echo the get-well-soon message to JRL. Also noted was the extraordinary (but highly variable) network speeds achieved to ESO: "12 - 16 GB/s (or 0.7-1 GB/min if you prefer)" ?! Shurely shome mishtake ... Networking: Network speed CASU-WFAU continue at the embarrassing dribble (compared to the ESO proud stream) of ~25 MByte/s; good thing too as v1.1 VISTA and Dec/Jan/Feb WFCAM data are flying up north (when the connections hold up - some stability issues in the path, or possibly at the CASU server end?). WSA/VSA Operations: ETWS noted: RPB noted: - VMC database released yesterday. - VHS database ready for release later today. - Replication of tables set up between ramses2 and ramses9 for VVV operations. The team again reviewed the status of the various VISTA surveys, particularly regarding the ESO deliverables: o VMC: ESO products and survey DB (albeit v1.0) release to the PI. Some issues regarding double-vision morphology in CMDs possibly attributable to v1.0 pre-grout effect? Investigations on-going. o VHS: new survey DB for PI QC ready to go today. o VVV: pressure's off on the ESO deliveries from WFAU - new data of June 1st for DB-driven products incorporating v1.1 data. o VIKING: still a few tiling-related issues here o VIDEO: pressure's off as CASU have stepped in to send OB-level products (as for other surveys) this time around. o UltraVISTA: still SEP. Hardware and Systems: MSH noted that IP switch rearrangement will begin from Monday next week, including retirement of the oldest servers. Osorkon has a new motherboard and is all better; data transfers from the (degraded) RAID arrays of djedefre are complete, and all old non-NAS local disk stores will either continue to be available for local processing and testing, or will be cannibalised for spares. RSC asked about the status of ramses10 as a further ingest DB server, and it was agreed that this machine should be installed in the cluster rack and set up ACTION: MSH to set up ramses10 in the cluster rack and configure as a further ingest DB server. RGM noted that a beefy DB server is still required for an MSc project over the summer, and the team agreed that it would be a good idea to get this in asap anyway. Not wishing to be excluded from the splurge party, MAR asked for a new web server to keep on top of the demands currently being made on horus. ACTION: MSH and KTN to spec/tender for another cluster node and another web server Software: RSC noted: "Work on implementing the FITS header metadata in the ESO release script is now complete and the first files have been released to a PI, of the VMC, for inspection. Provided a workaround for the memory leak problem that I noticed, using a variant of MAR's suggestion at the last meeting, though it's not clear whether mxODBC, NumPy or PyFITS are to blame (given the latter's recent reputation I'd point my finger there). Also looked into creating PyFITS tables in a row-oriented way, which is more convenient for database queries, but this is implemented in a ridiculously inefficient way in PyFITS, so my experiments have been committed to a separate dev_esorelease branch. Trawled through the complete curation code to identify potential issues brought about by splitting the ingest database across multiple servers to give the VVV its own load database. All obvious problems have now been fixed, so the code in trunk is now ready for live testing of the curation procedure against a synchronised VSAVVV database of the v1 data prior to the real release run once the v1.1 data are ingested." NJC has worked on further shaking down of the deep tiling DB-driven pipelines, integration of the v1.1 tools, and has set-up a variability co-ordination TWiki page for pooling thoughts and suggestions on VSA developments in this area. Survey Data Release: It was noted that UKIDSS DR5 with GPS/UDS should now go world public. ACTION: RPB to make the full-blown UKIDSS DR5 including UDS/GPS world public. Non-survey Data Release: WFCAM 09B non-surveys are now live to the world. Astrogrid deployment & Data Analysis services: No news this week. Miscellaneous: Nowt else this week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ============================================================= -- Scanned by iCritical.