This change set:

- Adds the steam ID to the aborted error message for better
  readability and troubleshooting.
- Reverses the order between:
  - The log message that says that the stream was terminated.
  - The actual termination of the stream.

Change-Id: If3537b4770eb00d5c67b032bdb49303c9007e794
Signed-off-by: yacovm <yacovm@il.ibm.com>

This change set adds cluster communication metrics:

	EgressQueueSize          metrics.Gauge
	EgressWorkerCount        metrics.Gauge
	IngressStreamsCount      metrics.Gauge
	EgressStreamsCount       metrics.Gauge
	EgressTLSConnectionCount metrics.Gauge
	MessageSendTime          metrics.Histogram
	MessagesDroppedCount     metrics.Counter

And adds corresponding unit tests for them.

Change-Id: I769d83b0721cc8dbf35b07b51b1d024539064009
Signed-off-by: yacovm <yacovm@il.ibm.com>
(cherry picked from commit 37fc516d6465be36c4e3daabd612155b9c2c18d3)

This change set makes consensus messages be sent asynchronously, over
a buffered channel with a size of 10.

Consensus messages are dropped when the buffer overflows,
and Submit messages are blocking on the buffer.

Without this change set, the sending of large messages takes milliseconds,
while with the change set it takes micro-seconds.

Change-Id: Id60b05b96eed6d9d04f89b8967945b18ddfbef94
Signed-off-by: yacovm <yacovm@il.ibm.com>

This change set removes Step RPC from the cluster protobuf,
and renames Submit stream to a Step stream, and makes both
transaction forwarding and consensus messages use the
new Step stream.

It also makes both egress Send() and Recv(), have a maximum
timeout (the RPC timeout in the config).
A Send or Recv that is used to send a consensus message,
or send (receive) a transaction (status) will now abort prematurely
in order to protect against any liveness issue on the remote node,
and also to return an answer to clients within a timely manner.

Change-Id: Id942b248212f5c324e12af34fce48f96fdbb6aea
Signed-off-by: yacovm <yacovm@il.ibm.com>

Per the gRPC documentation:

If an RPC is issued but the channel is in TRANSIENT_FAILURE or
SHUTDOWN states, the RPC is unable to be transmited promptly.
By default, gRPC implementations SHOULD fail such RPCs immediately.
This is known as "fail fast," but usage of the term is historical.

However...

RPCs SHOULD NOT fail as a result of the channel being in other states
(CONNECTING, READY, or IDLE).

Therefore, if it takes too much time for gRPC to move from
a state of CONNECTING to TRANSIENT_FAILURE (i.e - packet drop,
or DNS lookup failure) - it will slow down the entire Raft FSM.

This change set makes Step RPCs inspect the underlying
gRPC connection state prior to being invoked.
If the connection is in state connecting, then
it fails fast.

Change-Id: I50df1f758a00fc99bed54ff1a2056f83f53efdf7
Signed-off-by: yacovm <yacovm@il.ibm.com>

This change set:

1) Wires the consenter into the registrar
2) Wires the communication to the consenter and
   to the registrar, in preperation for multi-node Raft.
   This makes a new chain configure the communication layer
   of the cluster according to the TLS certificates
   of the channel.
3) Adds a unit test that spawns a Raft OSN

Change-Id: I3923f6f5e0fae6428c2be2e056b82355332f3324
Signed-off-by: yacovm <yacovm@il.ibm.com>

The cluster code was using the go-logging logger
instead of the flogging.FabricLogger.

Changed it accordingly.

Change-Id: Idcb4f6a593baac78afb50bff5993545d98155356
Signed-off-by: yacovm <yacovm@il.ibm.com>

FAB-12083 #done

This change set adds a call to the context cancel function
when the stream cannot be established.

Change-Id: Ie38d5e59e6f6a4635dcf427ea4c2a4cf4d9ff44c
Signed-off-by: yacovm <yacovm@il.ibm.com>

Also update the linter script to disable the lostcancel check until we
can fix them.

Change-Id: Iadeec48b4347f19108c474a241a04a04825594ab
Signed-off-by: Matthew Sykes <sykesmat@us.ibm.com>

This change set implements the higher level of the raft
communication layer which is to be used by the consumers
of the communication layer.

It uses the base communication layer from FAB-11585 and
exposes a higher level API over it for sending messages,
via an RPC object, and also implements a gRPC service
that implements the Cluster protobuf API and passes
the messages to the base communication layer.

Change-Id: Id5f5f60179b4cac5ef7ab88bc4c9f5b7425391e6
Signed-off-by: yacovm <yacovm@il.ibm.com>

This change set implements a base communication layer for
the raft orderer, but it can actually be used for any
consensus protocol that has:
- A specified number of members at any given point in time
- uses TLS pinning for identification
- Represents members as integers

The communication has 3 basic capabilities:

1) Obtain a remote context given a destination node ID and a channel
2) Configure the communication on the context of a given channel
3) Get notified of Step() and Submit() messages from other remote nodes

Change-Id: I87f193b7723f95f359cc4071e28b362be703ab74
Signed-off-by: yacovm <yacovm@il.ibm.com>