When CheckQuorum is enabled, leader steps down if it cannot reach
the quorum of network, so that clients have a chance to disconnect
and try other nodes.

Change-Id: I901c0e3009f9d354a2b504fe16174432345055b3
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

When gRPC buffer of `Submit` stream is full, `SendSubmit` would
block, which freezes the `serveRequest` go routine. This CR moves
this out of go routine, and clients should be blocked on waiting
for room in buffer.

Change-Id: I62cd261b9419bd8df3fa1bfaeff14551168d2e65
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

If there are MaxInflightMsgs blocks proposed but not
committed, chain blocks further incoming requests.

Change-Id: I58c84e23c882ccc152e5c9a248434e466a8b5266
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

This CR changes Errored to return a channel that is
closed when node becomes candidate.

Change-Id: Ibd0ece763b9d93c4da93825d1b302ecc55a9b32e
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

Store raft SoftState in raft chain so it returns error
while election is ongoing. This prevents a disconnected
follower from returning success on Broadcast API.

Change-Id: Ib6619b230938f0d6c10240b8cd8e34e346056145
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

This CR changes type of etcdraft observe channel from uint64
to raft.SoftState, so that chain_test can assert not only leader
id, but also the state of node.

Change-Id: Ia0c5f8c9060c234ceb84133e0c5598ed064dd1ee
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

Newly elected raft leader should wait for in flight blocks
to be committed, before accepting new envelopes and creating
new blocks. Otherwise all those blocks created would be uncle
blocks and we don't permit this situation in Fabric.

Change-Id: Ia5adac185263735eace1fc805ebea0f5c98b2fb1
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

Instead of returning status several levels up, several methods
in etcdraft chain can just set member var to store current state.

Change-Id: I67612917bf3bb3225f1507c8b7376d730b18e9f4
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

Change-Id: Ib77c866a30ed5108ad53908b0ca25a60a89e9a7c
Signed-off-by: Artem Barger <bartem@il.ibm.com>

This CR rewrites BlockCreator so that it doesn't return nil block.

BEFORE:
blockcreator holds a channel of created blocks, which is buffered
with size of createdBlocksBuffersize (default 20). It also stores
the hash and number of latest block.

When requested to create new block, blockcreator does so
by assembling a block based on that hash and number, enque the
block to buffered channel. If channel is full, a nil is returned.

When commit a block, it drains the channel. If there's nothing in
the channel, it implies the blockcreator is manipulated by raft
follower, therefore blockreator simply updates hash and number.

NOW:
what we need is actually as simple as: a blockcreator holds the
hash and number of latest block. When it is requested to create
a block, it just uses that hash and number to assemble one.
And ONLY raft leader holds a blockcreator. Followers blindly
commit whatever comes from consensus. When a follower is elected
as new leader, it simply looks up the ledger, find hash and number
of latest block, and creates a new blockcreator.

Change-Id: I226ee34d666fbb1e8d034dc22ea6800df993f7a4
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

This CR removes the global leader var in etcdraft chain because
it is racy in following case: several requests are to be enqued
into submitC while leader loses its leadership.

This also removes the lock on rpc.SendSubmit because it's guarded
by the channel.

Change-Id: If5e785e05dcf9bfc60e403f2d5813baf769ee103
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

This CR makes raft-facing logic more contained by encapsulating
them in a seperate object, in a new file. This CR also simplify
the implementation by removing some unecessary locking and channels

Change-Id: Ibefe74a35f0645955cbb3885efc8a3f8aaa0a8ca
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

When joining a fresh node to existing etcdraft cluster, it
should be using empty peer list to call `StartNode`.

Change-Id: Ib6acf6fd9b2956680c99d5d7370ce439228d3bfa
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

When new leader performs failover check and finds out leftover
ConfChange, it should re-propose that AND do not accept new env
until ConfChange is applied.

This CR also moves type b config failover check from `serveRaft`
to `serveRequest` go routine. `serveRaft` now only takes care
of raft related logics, therefore making the code easier to
reason about.

Change-Id: I4e32cb155abff52e7d1c0fbe4c6f2aa5e5ef1605
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

This CR moves the logics that applies raft data to state machine
from `serveRaft` to `serveRequest` go routine to avoid some
coordinations between them.

Change-Id: I7331976d0f0c256c853f09e15ca8682dd9a0322d
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

Change-Id: Ie755722f7db1df5efca5dea43d9912fdf36b6d25
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

This commit address code review comments left from review of CR related
to FAB-12552.

Change-Id: I04635d36d7076cf89d43f42aa5351692b8d1782e
Signed-off-by: Artem Barger <bartem@il.ibm.com>

For etcdraft chain, if there is applied index (>0) in
block metadata, then the node is expected not to be fresh
and should have wal data.

This CR removes this constraints because that's not the
case anymore with onboarding, since one node may pull blocks
from other nodes before bootstrap chain.

Change-Id: I678db4d1d71a24e08685d9c73eee7f4bc281771f
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

This commits handles the case where config transaction of
type B is submitted and Raft cluster quorum fails/shutdowns.
During startup, leader compares last known configuration with Raft
config state and if there is a difference, finalize reconfiguration by
proposing delta.

Change-Id: I3cdf03533602489cb56c503f1d6651f27a5fc6a1
Signed-off-by: Artem Barger <bartem@il.ibm.com>

This CR adds a mutex to guard parallel access to `SendSubmit`
rpc call in etcdraft chain, due to thread-unsafe of gRPC.

This is going to be tested by subsequent benchmark tests.
Without this serialization, loading etcdraft orderer with
many parallel requests would most likely cause a dead-lock
in communication, which causes benchmark test to stuck.

Change-Id: Id2f6e8f86f7465880725e1aecc7374c3e9e4106e
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

This CR enables optimistic creation of a chain of blocks in etcd/raft to
pipeline block creation and consensus on the created blocks. We cannot
do the pipelining for config blocks since all messages need to be
revalidated upon a config change.

Change-Id: Iabf1d4c75584afe8f641a18153d5e1b4b94f6bcc
Signed-off-by: Adarsh Saraf <adarshsaraf123@gmail.com>

Raft cluster reconfigiration consists of two parts, first leader has to
consent on configuration block, next leader has to extract new cluster
configuration and propose raft configuration changes. However leader
might fail between first and second parts, therefore newly selected
leader should be able to detect there is unfinished reconfiguration and
to finish reconfiguration.

This commit adds logic to manage leadership failover, where new leader
checks whenever last committed block is configuration block, whenever
there are pending configuration changes and complete reconfiguration by
proposing raft clust configuration changes.

Change-Id: I05dc1f60c9ab692521887b50f726d96ea47878dc
Signed-off-by: Artem Barger <bartem@il.ibm.com>

This option prevents a reconnected node from disturbing network.

Change-Id: Ie7073cc2e84c3945d7b708fae9e52b999ba793d2
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

This commit adds support for Raft cluster reconfiguration, takes care to
extract cluster config changes from the config block and apply those
changes to the replic set.

Change-Id: I564fafa418757e565ec2c0d872174d5d4f1ad719
Signed-off-by: Artem Barger <bartem@il.ibm.com>
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

This CR adds enables etcd/raft based chain to take snapshot for
every `SnapInterval` blocks. It stores snapshot to wal and disk.
When node restarts, it loads latest snapshot and uses Term and
Index stored in it to replay WAL to memory storage.

Change-Id: Ib8c22173cd5b05b7a1b27b16e2003bdabbecf82a
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

Prevents the etcd/raft-based orderer from writing to the production path
during tests. Instead it uses os.TempDir for WALs and snapshots.

This makes FAB-12824 #done.

Change-Id: I6bcdf5b0ac59575158e6bdb9e8d0e033f926b369
Signed-off-by: Kostas Christidis <kostas@christidis.io>

This CR refactors storage etcd/raft chain implementation
to bundle memory storage and wal together.

Change-Id: Ib0dc95497dc9a0747c365ef78d0a702382445f55
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

This CR adds WAL to etcdraft to persist log entries, so
that an etcdraft based chain can continue from where it's
left. The index of log entry is persisted with block as
its metadata so written blocks are not replayed.

Change-Id: I7e4131541d90b256d767a9421ab6ae2a47a88bcd
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

This commit adds new message to hold information about mapping from
consenters to theirs ids and to keep track on the next id to be assigned
for new coming replica. Moreover update write block path to actually
persist information about raft cluster membership mapping.

Change-Id: I638b30fc5729a0d02f21a40989deb93e42b09836
Signed-off-by: Artem Barger <bartem@il.ibm.com>

This commit allows Raft transactions updates which changes algorithm
parameters, while no update or change of consenters set is supported
yet.

Change-Id: Iefefa130302c78a8e00de005ff3421d2a686eb38
Signed-off-by: Artem Barger <bartem@il.ibm.com>

This CR makes etcdraft.Chain consume communication layer to pass
messages among Raft nodes, so that they are able to form a cluster.

Change-Id: I60327f1c1c6942db037e55f17da4d6d5b632ee2a
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>
Signed-off-by: yacovm <yacovm@il.ibm.com>
Signed-off-by: Kostas Christidis <kostas@christidis.io>

This commit enables creation of Raft based channel and aligns unit-tests
accordingly.

Change-Id: Iead5cddcab5137b0bce9740f47db8a3f583f2f4f
Signed-off-by: Artem Barger <bartem@il.ibm.com>

This CR Allows Type A config updates while rejecting the type B ones for
the etcd/raft consenter.

For definition of Type A and Type B config updates please refer to the
JIRA.

Change-Id: If7351eb28fe66ff7ca31e45206ed044ee2937051
Signed-off-by: Adarsh Saraf <adarshsaraf123@gmail.com>

This change set:

1) Wires the consenter into the registrar
2) Wires the communication to the consenter and
   to the registrar, in preperation for multi-node Raft.
   This makes a new chain configure the communication layer
   of the cluster according to the TLS certificates
   of the channel.
3) Adds a unit test that spawns a Raft OSN

Change-Id: I3923f6f5e0fae6428c2be2e056b82355332f3324
Signed-off-by: yacovm <yacovm@il.ibm.com>

This change set adds Step() to etcdraft.Chain in order
for the etcdraft.Chain to implement the etcdraft.MessageReceiver
interface.

Change-Id: I49f53a5164942723af705ea4bedd6763f5b325f3
Signed-off-by: yacovm <yacovm@il.ibm.com>

This CR also adds tests for etcdraft chain.

Change-Id: I8f86a117a9b1642df2745ef7e48e8bddf8f898e3
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

This CR implements a bare minimum raft-based chain, which only
supports single node. It does not support following features:
- process config type message
- raft snapshot
- raft WAL
- reconfigure raft

This CR all pulls in a fake clock library for testing purpose.

Change-Id: I5e93ff9eeb524ea1afd0348e1180fb9f4535fb5c
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>