From nch@roe.ac.uk Mon Oct 3 11:53:05 2011 Date: Mon, 3 Oct 2011 10:30:31 +0100 From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Jim Emerson , "Noddle, Keith -- Keith Noddle" , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Bob Mann , Stephen Warren , Tom Kerr Subject: WFAU VDFS Science Archive team meeting mins, 30/09/11 Minutes of WFAU VDFS Science Archive meeting: September 30th 2011 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, RPB, RGM, MSH, KTN, RSC, NJC Apologies: JPE, MAR, AL, C37, 89X DoNM: 10am Friday October 14th 2011 in the VH Actions discharged this time: ----------------------------- ACTION: RPB to fettle the permissions on UKIDSS DR7 (sans GPS) and UKIDSS DR6 GPS to make them world accessible. Discharged Actions partly discharged but continuing: ----------------------------------------- ACTION: RSC to restore temporary ingest file writing to high bandwidth shares between curation clients and DB servers. Superseded by: ACTION: RPB to set up a linux -> windows share and RSC to test ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Work in progress (some problems with OS level permissions on r/o files - MS DOS strikes again) ACTION: RPB (or whoever communicates with the PIs) to keep up the pressure on world public release DB preparation by emphasising that the recently prepared DBs are not simply for internal consumption. VHS and VIKING nudged, with so far no response. The plan is to send one-month reminders, escalating to fortnightly, as any potential release dates approach... Actions carried forward from 02/09/11 meeting --------------------------------------------- ACTION: RPB to set up mirror/sync for flat file DBs using MS SQL replication. Continues; NCH noted that the MS SQL implementation may not be as streamlined as might be hoped... ACTION: RPB to try a proof-of-concept on detection table monthly partitions using the VHS to start. Superseded by: ACTION: RPB to try monthly detection table partitions for VVV Specific points and new actions: -------------------------------- Project management: RGM noted that we're on the cusp of making some new temporary appointments to use up the dregs of the current grant - more about these next time. WFCAM & VISTA updates: Nothing of note. Comments and issues arising from CASU minutes: No new minutes this time. Networking: The local copy of SDSS DR8 is up and running on it's own database cluster node. Since UKIDSS DR9 is not quite live yet, it was decided to try to incorporate the crossneighbour tables with SDSS DR8 this week. ACTION: RPB to run cu16 on LAS/GCS/DXS against SDSS DR8. For WFCAM/VIRCam transfer updates, see next section. WSA/VSA Operations: ETWS reported: -- Finished running the release software on the VMC. -- Ran software up to CU19 on the LAS, GCS, and VIKING. -- DXS is still producing stacks, but will finish early next week. -- Transferred the VIDEO reprocessed data as well as the July data for VISTA and ingested the metadata. -- Transferred the June data and the reprocessed GPS data and ingested the metadata. -- Installed the 3rd party software including Python 2.7.2 and pyFITS 3.0.2 on sahure. RPB reported: "VMCv20110909 released for P86 data. VHS and Viking P86 releases imminent (as soon as we've made room for them on the public servers) VVV release currently being produced. It's started outgesting the detection table. Could take a while!" Hardware and Systems: MSH noted that a new database ingest server is being ordered for the VVV, and he is scoping out storage requirements for the next 2 years. It was noted that the next phase of acquisitions will result in total online storage at WFAU of 0.5 petabytes. NCH asked about offline backups for mission critical datasets, and the status of LTO4 (and 5?) was discussed; it was agreed that if such a thing as LTO5 exists and it is sufficiently mature, we should acquire a drive. It was also suggested that the old tape library should be upgraded, and ops requested that were possible existing server's physical RAM should be maxed out as well. Finally, advanced warning: MSH and ETWS noted that at some point within the next year, there will have to be a "big bang" in the linux systems when they are all migrated to Linux3.0/Python3. Software: RSC noted: "The new detection quality flagging of sources in dodgy regions of the WFCAM detectors were implemented well in time for the DR9 release and Steve Warren has confirmed that the implementation agrees with his specification. The documentation, including the website, has also been updated. Work progresses now on the flagging of detections in close proximity to bright stars in VISTA data, which will be low importance information bits - 5 around 7 < J < 12.5 magnitude stars and 2 for brighter stars that have additional filter ghosts, offset along the vector that points away from the telescope's optical axis. This should be ready for the first P87 VISTA releases. The code changes used to experiment with alternative drives to store the curation script intermediary files have been committed to a dev_diskperf branch of trunk. This work will continue when we've got some Linux hosted share drives available for testing on the catalogue servers." NJC reported: "I have made improvements to CU6 to speed up the astrometric calculations by simplifying the SQL - possible now that the mjd is in the DetectionRaw table and putting selection on the ppErrBits directly into the SQL rather than later in the python code. I have made some improvements to the grouping in ProgrammeBuilder and have fixed several other minor bugs. I have added some new options into the EsoRelease code and tested this on VIKING data for their release. I have made SVN branches for the July releases and updated the webpages for known issues and releases and fixed some errors in the SQL cookbook." Survey Data Release: UKIDSS DR9 is *nearly* there; the plan is to try to release next week (following MAR's return from hols). Non-survey Data Release: It was noted that open time on Vista for proviate PI programmes has recently become available, and that there is some activity in the community in preparing proposals. Astrogrid deployment & Data Analysis services: MSH noted that he's off to the next IVOA meeting in Pune (India) in a week or so. Miscellaneous: NJC noted that a request from Gaia-DPAC has been received to provide all WFCAM survey/non-survey detection photometry for the purposes of building a multi-epoch variability dataset. Should be interesting... ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ============================================================= -- Scanned by iCritical.