From mar@roe.ac.uk Mon Jan 18 15:40:45 2016
Date: Mon, 18 Jan 2016 15:40:39 +0000
From: Mike Read <mar@roe.ac.uk>
To: VDFS Science Archive Team -- Clive Davenhall <acd@roe.ac.uk>, dmr@roe.ac.uk, Eckhard Sutorius <etws@roe.ac.uk>, Mark Holliman <msh@roe.ac.uk>, Nicholas Cross <njc@roe.ac.uk>, Nigel Hambly <nch@roe.ac.uk>, Rob Blake <rpb@roe.ac.uk>, Ross Collins <rsc@roe.ac.uk>, Stelios Voutsinas <stv@roe.ac.uk>
Subject: WFAU VDFS Science Archives progress meeting minutes, 8 January 2016
WFAU VDFS Science Archives progress meeting minutes, 8 January 2016
Present: ETWS, RPB, MAR, NJC, RGM
Apologies: MSH, RSC, KTN, AL, STV, DMR, ACD
DoNM: 10am Friday 22 December 2015 in the Villas meeting room,
ACD in the chair (room is booked).
Actions discharged this time:
----------------------------
ACTION: ACD would organise a meeting to discuss the dead-locking that
results from attempting to run more than one different curation
activity simultaneously on the same server. Attendees should include
ETWS, RPB, MAR, NJC and RSC.
Done. A meeting was held and a plan put in place to make copies of the
ingest server and run releases from there. Copying back over relevant
updates and transactions.
ACTION: ACD to speak to KTN about funding a new KVM switch.
Done. It was thought that funds had been secured.
Actions carried forward from 4/12/2015 meeting:
-----------------------------------------------
ACTION: RGM to investigate the size and content (how does it differ
from DR9?) of SDSS DR10 to see if it is worth us hosting and
whether it is necessary to keep all of the earlier releases.
Continues. The first part had been done: the difference between DR9
and DR10 is largely the presence of additional spectra. The part
of this action that continues is that RGM had not yet established
which early releases could safely be scrapped. RGM also noted that
that DR11 and DR12 were now available and now need to be looked at.
For further details see the WFAU SDSS strategy wiki page:
http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan
ACTION: ETWS, with RPB, to develop an automated helper script to
keep release DB files links updated.
Continues: this action still has low priority.
ACTION: NJC noted that VISTA mosaicing crashes with normal filtering,
which he should fix.
Continues. NJC had implemented a fix. He had run some tests, which
completed successfully. However, the software had still to be run
'in anger' on a complete ingest dataset.
ACTION: NJC noted that he had received a report of a problem with the
ESO VVV DR1 release which he should investigate. In one pointing
the merged band catalogue contained fewer sources than expected.
Continues. This action was very low on NJC's priority list.
ACTION: MAR, NJC. Either MAR or NJC should go to the VMC meeting to be held
at Keele University during June 2016.
Actions resulting from the VVV DR3 brainstorming meeting held on 24/2/15:
------------------------------------------------------------------------
ACTION: RPB to experiment with ingest via partitioned tables, along the lines
of the SSIS paper.
Deferred (changed back from 'continues'). This action has now been
overtaken by events: the VVV DR3 processing would now be done differently.
However, the technique was still potentially useful for future work and
should be investigated, but not as a matter of urgency.
ACTION: RGM (was RSC, RGM): (i) RSC to write a succinct summary of the ingest
and outgest problem that is the principal bottleneck with the VVV
processing (briefly the ingest and outgest speed is limited to approx.
100 Mbit/sec for no apparent reason) and (ii) RGM to compose an e-mail
message around this summary and send it to Ani Thakar.
Continues. RSC had composed a succinct summary and sent it to RGM. RGM
had emailed Ani Thakar but no reply to date. RGM had sent a reminder but
had still received no response. RGM had also contacted Alex Szalay and
not received a reply from him either. He would consider how to proceed.
Specific points and new actions:
--------------------------------
Project management:
* The grant review panel would be held in London of Friday 22nd January.
RGM and MAR would attend.
* RGM noted that he would be writing up the PDRs over the next few weeks.
* It was hoped that we would host a meeting on space junk (July 4-5th)
following the Edinburgh SPIE meeting.
WFCAM, VISTA and VST updates:
* Nothing to report.
ACTION: MAR to arrange telecon with CASU.
Networking:
* The network was operating nominally.
WSA/VSA/OSA/GES Operations:
* ETWS reported that P95 ingests (CU4) were proceeding for VSA and OSA.
* RPB noted that he was copying over VVV P92/93 detections to the main load
server.
Hardware and Systems:
* RPB noted that there were no real HW problems over the holidays. The new
A/C seems to be working well. There's still some ongoing work in the server
room and at some point our servers will need moving but this could probably
be done in small stages.
- The KVM switch had come back to life but a new one was still needed to
handle the increased number of servers.
- Tarhaqa had been issuing failure predictions but continued to work,
Eclipse would have a look into it.
- Various new servers (NAS box and curation machine) had arrived.
Software:
* NJC reported:
- Work on optimizing the ESO release code for VVV.
- Creation of SWARP mosaics and discussions with Alastair Edge.
* ETWS had been rolling out 3rd party software updates.
- RGM raised the issue of unit tests and integration systems to help spot and
resolve issues with software and updates. He would table this for discussion
at the next quarterly.
* MAR noted he'd updated OSA web services to handle VPHAS queries.
Lockheed Martin:
* Waiting to hear back about hardware and software installation at Palo Alto.
Review of Deadlines:
* Progress towards forthcoming deadlines was reviewed. See:
http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines
Survey Data Release:
* P94 releases had been made for VMC and VIKING. CU19 was currently being
run on the VHS P94 release.
Non-survey Data Release:
* None.
Astrogrid deployment, VO & Data Analysis services:
* Nothing for the minutes.
Miscellaneous:
* Nothing for the minutes.