- 20 Mar, 2019 2 commits
-
-
Jay Guo authored
When a node is restarted, it loads WAL data since last snapshot into memory, however this chunk of data is not currently taken into account as part of accumulated data, which is compared to snapshot interval. This CR fixes this. Also it changes some log level to improve serviceability. FAB-14656 #done Change-Id: If152071e64fd8268d20362c593d24af4ab2be355 Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
Jay Guo authored
When the certificate of leader is rotated, it will certainly be disconnected after reconfiguring communication. Instead of waiting for ElectionTimeout and elect new leader, the old leader should be more cooporative and transfer its leadership to others. Note that proposals sent during this transition will be automatically dropped by etcd/raft, however transition should be fairly short. Change-Id: Iabd005d00864afe09b4738f1ed36b939b1d83eed Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
- 16 Mar, 2019 1 commit
-
-
Jay Guo authored
- MaxInflightMsgs is internal to etcd/raft and should be exposed to users with a more appropriate name: MaxInflightBlocks - MaxSizePerMsg is also internal to etcd/raft, and it's defaulted to PreferredMaxBytes in BatchSize, so that if a big block is created, it is sent in a its own etcd/raft message, instead of being batched with other blocks. This parameter takes effect when a batch of entries is sent to lagged node. During normal replication, each block is sent in its own message. It's not necessary to expose this config option to users. - SnapInterval is renamed to SnapshotIntervalSize FAB-14593 #done Change-Id: Icaf2848a41c5f0f0a02f4b0b4a80ba852fddd584 Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
- 15 Mar, 2019 1 commit
-
-
Jason Yellick authored
Presently, the block metadata encodes the TLS certificates of all of the Raft consenters in the system for each block. Because these TLS certs are non-trivial in size, and there may be a large set of consenters, this actually creates a significant amount of waste on the filesystem. As a small optimization, this CR modifies the block metadata to only store the nodeIDs instead of the full set of consenter info. It then correlates the consenter slice found in the channel config data with this slice of nodeIDs to build a mapping between the two (which was previously persisted). Change-Id: Iaa66dacbcc48a041318c8a718099a873b9626240 Signed-off-by:
Jason Yellick <jyellick@us.ibm.com>
-
- 14 Mar, 2019 2 commits
-
-
Jason Yellick authored
The detectConfChange presently returns several values, not all of which are always non-nil, and not all of which being nil caused fatal results. This is a pre-amble to converting the consenter id -> TLS cert map to a slice of ids. Change-Id: I02d01137e5cb0ce1250c166539170c44575e4fd4 Signed-off-by:
Jason Yellick <jyellick@us.ibm.com>
-
Jason Yellick authored
There are presently two etcdraft protos around metadata. One is the metadata stored in the config and it is named 'Metadata', the other is the metadata stored in each block, this is named 'RaftMetadata'. This causes confusion when reading the code. This CR transforms those names to be: Metadata -> ConfigMetadata RaftMetadata -> BlockMetadata Change-Id: Ia0394ebe78f5541996c010c3c67d760f336f75d8 Signed-off-by:
Jason Yellick <jyellick@us.ibm.com>
-
- 13 Mar, 2019 1 commit
-
-
yacovm authored
This change set adds Append() to the consenter support, and changes the eviction logic to use Append() instead of WriteBlock, so that the blocks that are pulled from OSNs are not changed and are written as is into the ledger. Change-Id: I76abe64990f1855b53dadb0655c5830169ef7ed1 Signed-off-by:
yacovm <yacovm@il.ibm.com>
-
- 11 Mar, 2019 1 commit
-
-
yacovm authored
This change set adds a basic validity check that prevents admins from adding the same consenter twice. Change-Id: I0e5efbd78a77a060b060c20b447609e604749815 Signed-off-by:
yacovm <yacovm@il.ibm.com> (cherry picked from commit afc680484e503a7a4366177993da195906fddc73)
-
- 09 Mar, 2019 1 commit
-
-
Jay Guo authored
If a single config tx adds and removes 1 cert in consenter set, it is considered to be cert rotation, therefore should be allowed and supported. Change-Id: Id2e78ae294cfb21501d1344e61ee1430088a0a68 Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
- 08 Mar, 2019 1 commit
-
-
Adarsh Saraf authored
This CR adds the following metrics for etcdraft: - data_persist_duration - # normal_proposals_received - # config_proposals_received Change-Id: I34b9bdcb3ba7eb0bb074eb7abc93c2f5b5b5c5ad Signed-off-by:
Adarsh Saraf <adarshsaraf123@gmail.com> (cherry picked from commit cca81e5581073a1fad036146b64d9321950c4f3d)
-
- 07 Mar, 2019 3 commits
-
-
Jay Guo authored
If a lagged node is not aware of newly added node(s), it simply cannot communicate with them. When it suspects its own eviction, it inspects certificates in them, would find itself still among consenter set. And if the leader happens to be among those new nodes, we have a deadlock. To break this, when a node suspects its own eviction, it also triggers a catchup to actually pull blocks and commit them, so that it can eventually recognizes new nodes. Change-Id: Id2e9a423221e4a3bebeeeaf7da5f120396f342dd Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
Jay Guo authored
Two nodes can exchange messages only if `configurator.Configure` is called on both sides. Currently, etcdraft UT network mock assumes nodes can accept messages from any node as long as it's connected, however this does not reflect the actual behavior of network and UT failed to capture bugs such as FAB-14413 because of this. This CR makes network mock more realistic by adding `links` to it. A link is open if nodes on both sides have called `Configure`. Also, this CR fixes another minor problem in UT, where `support` should get prepared *prior to* the start of new chain. Change-Id: I08b0cd84e9e0774dd130a70293d7c6b328ffb94c Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
Jay Guo authored
If two config tx that modify consenter sets are sent back-to-back, the second one is revalidated because sequence number has advanced. And we should perform consenter set check to make sure it only udpates set by 1. Change-Id: I9f7e54a3c0854d1f130d4a061466edd398e35bde Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
- 05 Mar, 2019 7 commits
-
-
Jay Guo authored
If revalidation fails, it should have no impact on block cutting timer (not start/stop it). Change-Id: I9f8cb6bdffa6f95319fe14cdb66e4cfcac9851aa Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
Jay Guo authored
Reconfiguring SnapshotInterval should take effect immediately without rebooting orderer. Change-Id: If3018c05218d843d189ea802298e999c7276a851 Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
Jay Guo authored
If channel creation config contains only a subset of orderers in system channel, it should just create a new channel with specified set of orderers, instead of reconfiguring system channel. This CR fixes config block commitment in etcdraft to properly handle channel creation config. Change-Id: I408e707ae580ce041e8fc98e76311226437720ae Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
Jay Guo authored
When chain is started from latest snapshot, it should also load the ConfState from snapshot, to be persisted correctly persisted in subsequent snapshots. Change-Id: I0d4ae3419e7ab7f4d4dfae6251b88ec4b051885b Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
yacovm authored
This change set stop the eviction once-per-10-seconds check when the chain is halted. Change-Id: Ifa51419a731fa2ea19046e2cb1e49e60c5af4aba Signed-off-by:
yacovm <yacovm@il.ibm.com>
-
yacovm authored
This change set increases the interval of checking if there is no leader to 10 seconds, in order to reduce CPU cycles. Change-Id: Ic92a6cbc250e27a3efe519f9312264b65056e6c1 Signed-off-by:
yacovm <yacovm@il.ibm.com>
-
yacovm authored
This change set adds logic in the etcdraft chain that detects that the node is evicted from the channel, even if it was disconnected from the cluster while it was evicted. If a node fails sending to other nodes a consensus message, it starts suspecting that something is amiss. It then tries to pull the latest config block, and see if it is in the channel or not. If it is not, it: 1) Halts the chain 2) Pulls all blocks until the block that evicts the node. Change-Id: Ic3526d834fbef515119bb899fe62d30fdcf53267 Signed-off-by:
yacovm <yacovm@il.ibm.com>
-
- 04 Mar, 2019 6 commits
-
-
yacovm authored
When a single node etcdraft cluster is expanded and a new node is added, the new node needs to pull blocks from the existing node. However, the existing node loses leadership and then rejects deliver requests. This change set, makes a raft leader not reject deliver responses if the cluster has less than 3 members in it. Change-Id: I75bd028d5a46fcb6ae81dc29012e3e839149c319 Signed-off-by:
yacovm <yacovm@il.ibm.com>
-
Jay Guo authored
This is using a specif revision instead of release because we need commit 23731b to fix FAB-13920: long leader failover when `CheckQuorum` and `PreVote` are enabled, due to lease check in raft. At the time of upgrading, that commit is not included in any of etcd release yet. Change-Id: I0467352d35180f45a9931d7afd66f82d6c10990d Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
Jay Guo authored
Commitment of block is doen aync win blockwritter, therefore `support.Height` may not correctly reflect the actual number of latest block. Decision made based on `support.Height` while chain is running may lead to replaying of same block. This CR fixes this by keeping a reference to the last block, which is initialized by reading the tail of ledger at start. Change-Id: I6c5e5fed4c1464c459603f4484a44b8b91b017b6 Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
Jay Guo authored
`Propose` and `ProposeConfChagen` block when node is leaderless. However they are currently invoked by the same go routine that consumes data from applyC, which processes SoftState to reflect leader change. This may lead to deadlock: - block is created on leader and about to be `Propose`ed, - leader steps down due to loss of quorum - `Propose` is called and blocks - SoftState is passed on applyC - `serveRequest` is not able to consume applyC - deadlock Change-Id: If4dc048f82983862b5f253231dafd513b442bf53 Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
Yoav Tock authored
This is the fourth of four (4/4) sub-tasks that focus on the "green" path of consensus-type migration from Kafka to Raft. By "green" we mean that there are no failures or aborts along the way. The 4 sub-tasks are staged in a way that minimizes dependencies between them. In this sub-task we introduce changes to the etcd/raft-base OSNs such that they can restart from a ledger that was started as Kafka, migrated, and restarted. This change concludes all the changes needed to implement the green path on the "Raft" side. See respective JIRA item for further details. Change-Id: I5b408e1cfcb8cf42c39bed4df6c5496792175ef0 Signed-off-by:
Yoav Tock <tock@il.ibm.com>
-
Yoav Tock authored
This is the third of four (3/4) sub-tasks that focus on the "green" path of consensus-type migration from Kafka to Raft. By "green" we mean that there are no failures or aborts along the way. The 4 sub-tasks are staged in a way that minimizes dependencies between them. In this sub-task we introduce changes to the Kafka-base OSNs such that they implement the entire green path, until they commit migration and are ready to be restarted. This change concludes all the changes needed to implement the green path on the "Kafka" side. See respective JIRA item for further details. Note: some of the mocks had to be re-generated by couterfeiter to make the unit tests build. Change-Id: I2747f7d8017c344e9ff3bdd9dd98cbaa1480083f Signed-off-by:
Yoav Tock <tock@il.ibm.com> Signed-off-by:
Artem Barger <bartem@il.ibm.com>
-
- 03 Mar, 2019 14 commits
-
-
Adarsh Saraf authored
This CR adds the following raft-specific metrics: - cluster_size - is_leader - committed_block_number - snapshot_block_number - # leader_changes - # proposal_failures Change-Id: Ie35f2984bc8eaecdd5826b1e1aa21e6c759fdd83 Signed-off-by:
Adarsh Saraf <adarshsaraf123@gmail.com>
-
Jay Guo authored
This CR adds more debug logs to etcdraft chain to facilitate debugging. Change-Id: I2d70869bc8823babb3ab50782bd4472637ed5820 Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
Jay Guo authored
Instead of taking snapshot every N blocks, this CR changes it to taking snapshot every N bytes. This also sets default SnapshotInterval to 100MB, if it's unset. Otherwise data in memory is never compacted till OOM. Meanwhile, DefaultSnapshotCatchUpEntries is shrunk so it does not take too much space to preserve extra entries every time a snapshot is taken. Slow nodes are catching up using blockpuller, which is also efficient. Change-Id: I79cfeb8652fcbafdeb5793bf4f06267b95a858d6 Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
yacovm authored
This change set removes Step RPC from the cluster protobuf, and renames Submit stream to a Step stream, and makes both transaction forwarding and consensus messages use the new Step stream. It also makes both egress Send() and Recv(), have a maximum timeout (the RPC timeout in the config). A Send or Recv that is used to send a consensus message, or send (receive) a transaction (status) will now abort prematurely in order to protect against any liveness issue on the remote node, and also to return an answer to clients within a timely manner. Change-Id: Id942b248212f5c324e12af34fce48f96fdbb6aea Signed-off-by:
yacovm <yacovm@il.ibm.com>
-
Jay Guo authored
The creation of BlockPuller takes latest certificates, therefore should be done on-demand to guarantee its validity. Change-Id: I327275da495a85126feb58c84b460bed98f7b860 Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
Jay Guo authored
Every failed attempt to send Step request is logged at ERROR level, which pollutes etcdraft orderer leader logs when a follower is down. This CR changes it to log at DEBUG level, except for the first failed attempt and the first successful delivery after failure(s). Test done: manually run cft integration test and inspect logs. Change-Id: I1dd3468de1f6745f658c15e83a5e644e0b0492d6 Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
Jay Guo authored
This CR puts Raft snapshotting into a go routine to avoid excessive snapshotting due to extreme small SnapshotInterval. This is also preperation for WAL files pruning. Change-Id: Ib43a2197c533bdc224a4bc52ff6cb418b62a0c33 Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
Jay Guo authored
When CheckQuorum is enabled, leader steps down if it cannot reach the quorum of network, so that clients have a chance to disconnect and try other nodes. Change-Id: I901c0e3009f9d354a2b504fe16174432345055b3 Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
Jay Guo authored
When gRPC buffer of `Submit` stream is full, `SendSubmit` would block, which freezes the `serveRequest` go routine. This CR moves this out of go routine, and clients should be blocked on waiting for room in buffer. Change-Id: I62cd261b9419bd8df3fa1bfaeff14551168d2e65 Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
Jay Guo authored
If there are MaxInflightMsgs blocks proposed but not committed, chain blocks further incoming requests. Change-Id: I58c84e23c882ccc152e5c9a248434e466a8b5266 Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
Jay Guo authored
This CR changes Errored to return a channel that is closed when node becomes candidate. Change-Id: Ibd0ece763b9d93c4da93825d1b302ecc55a9b32e Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
Jay Guo authored
Store raft SoftState in raft chain so it returns error while election is ongoing. This prevents a disconnected follower from returning success on Broadcast API. Change-Id: Ib6619b230938f0d6c10240b8cd8e34e346056145 Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
Jay Guo authored
This CR changes type of etcdraft observe channel from uint64 to raft.SoftState, so that chain_test can assert not only leader id, but also the state of node. Change-Id: Ia0c5f8c9060c234ceb84133e0c5598ed064dd1ee Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-
Jay Guo authored
Newly elected raft leader should wait for in flight blocks to be committed, before accepting new envelopes and creating new blocks. Otherwise all those blocks created would be uncle blocks and we don't permit this situation in Fabric. Change-Id: Ia5adac185263735eace1fc805ebea0f5c98b2fb1 Signed-off-by:
Jay Guo <guojiannan1101@gmail.com>
-