There is a regression caused by https://gerrit.hyperledger.org/r/27558


where the error caused by connection failure to CouchDB does not get handled.

This leads to a nil pointer panic when the nil vdbProvider tries to get
a handle to a database.

This change will handle the CouchDB connection error gracefully and panic
the peer with a good error message (e.g. connection problem,
username/password problem) as was done in prior releases.

Change-Id: I5f459ab92c189de49442aef24fcd80abcd3a0391
Signed-off-by: David Enyeart <enyeart@us.ibm.com>

It is tied strongly to MaxInflightBlocks which is exposed,
so we should expose it as well.

Change-Id: I9cf49001eb1ebce89ddb7d969496d48c15a7e182
Signed-off-by: yacovm <yacovm@il.ibm.com>

We use the ones from the channel config anyway...

Change-Id: I5023fa124a60d5720e1a8a8a939afc0793f1154c
Signed-off-by: yacovm <yacovm@il.ibm.com>

Fix best-practives.rst:
     1. Line 89 code not highlight.
     2. The End of doc miss as follows.
Fix private-data-arch.rst:
     1. Line 214 word order error.

Change-Id: I9ef91c72550cc1a6e9aafaecc617772901bf7b50
Signed-off-by: TopJohn <xzj19922010@gmail.com>
(cherry picked from commit db68f57605b4c6929193a1b73e4e42b51173b0f4)

Change-Id: I726ea00654c7fda2d72a180cc104038109ce8a34
Signed-off-by: Artem Barger <bartem@il.ibm.com>

There is a deficiency in the way viper handles environment variable
overrides when unmarshaling to a structure.  If the variable is not
present in the config file (but present in the structure) and is
overridden in the environment, then this override is ignored.  This is
bizzare behavior which we have hacked around by putting all options into
the config file, but this policy is becoming untennable.

This CR adds (yet another) hacky layer in the viperutil package to
include fields from the structure which are not in the config into the
map which is ultimately used to populate the structure values.

The correct long-term approach is to stop using viper and handle this
ourselves.  The actual logic we have stacked on top of viper to handle
these problems is very likely more code than it would take to implement
our own sane config unmarshaling.

Change-Id: I47102c64162f1efcd9e1bd21a563e6aedb88abab
Signed-off-by: Jason Yellick <jyellick@us.ibm.com>

The config update detection code in Raft is a re-implementation of the
fabric config checking mechanisms.  This should probably be changed in
the long term, but one of its currently unhandled cases, is when a
config update references the consenters in the write set, but does not
modify them.  This can occur especially when adding a new organization
to the orderer group.  This CR adds the additional version checks
necessary to detect and ignore these sorts of updates.

Change-Id: Ib35a97e1cdbd557705f11da183c8547f9f85539a
Signed-off-by: Jason Yellick <jyellick@us.ibm.com>

Support for CAR packages will be removed in
v2.0.0.  Created initial release notes for
1.4.1 and included the deprecation announcement.

FAB-14720 #done

Change-Id: I6f032280bbdd16869134b6745d173e3c8db27b99
Signed-off-by: Gari Singh <gari.r.singh@gmail.com>

The CR adds consenter cert validation to incoming config update
to avoid applying bad certificate.

FAB-14648 #done

Change-Id: I6ff457846fb0c4efa216f109dd9ddc13862f6957
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

When a node is restarted, it loads WAL data since last snapshot
into memory, however this chunk of data is not currently taken
into account as part of accumulated data, which is compared to
snapshot interval.

This CR fixes this. Also it changes some log level to improve
serviceability.

FAB-14656 #done

Change-Id: If152071e64fd8268d20362c593d24af4ab2be355
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

When the certificate of leader is rotated, it will certainly be
disconnected after reconfiguring communication. Instead of waiting
for ElectionTimeout and elect new leader, the old leader should be
more cooporative and transfer its leadership to others.

Note that proposals sent during this transition will be automatically
dropped by etcd/raft, however transition should be fairly short.

Change-Id: Iabd005d00864afe09b4738f1ed36b939b1d83eed
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

Change-Id: I07833487adc9ee26a0058192247ffd17536fb7a1
Signed-off-by: Matthew Sykes <sykesmat@us.ibm.com>

Change-Id: Idf90487ecc145bc0213d355a7d4b6e6c3f7770b8
Signed-off-by: David Enyeart <enyeart@us.ibm.com>

This change set makes the orderer include the:
- Last config block
- Consensus metadata

In the same value that is used as the block signature value,
and this way it paves a way for orderers to be able to sign
both the block and the last config and the consensus metadata
in one signature invocation.

Change-Id: If5f331a7f7e859251af221937f2a3259e615369f
Signed-off-by: yacovm <yacovm@il.ibm.com>

When adding a block to the message store, if it is added and
not rejected - it is then added to the block puller.

When the block is removed from the message store, a callback
is triggered to remove the block from the block puller.

However, these 2 operations are not atomic.

Since we can add a block to the message store from both AddToMsgStore
(which is invoked by Gossip() ) and from HandleMessage,
we can have the following schedule:

  1) A block with sequence of 100 is gossiped by the upper
    layer of the peer, and AddToMsgStore is called, which
    adds the block to the message store, and the CPU is preempted.
  2) A block message 210 is received via HandleMessage and it causes
     the block 100 to be evicted from the message store, and the
    callback to remove the message from the block puller is called,
    but it is not removed because it is not there yet.
  3) The block 210 is added to the block puller, since it was added
     to the message store.
  4) The CPU is back to perform AddToMsgStore, and adds block 100
     to the block puller.

Now the block puller has block 100, and the message store doesn't
have block 100 anymore - which means it will never be evicted from
the block puller.

To prevent this we need to make these 2 operations be atomic.

Change-Id: I3b7d0d013ce8da5d9a0e40f8b0cdbc3edaed22c9
Signed-off-by: yacovm <yacovm@il.ibm.com>

This change set properly sets the last config metadata for inactive chains.
Inactive chains are chains that the orderer isn't part of, but still has the
genesis block in its ledger, to avoid state forks.

The replication code artificially creates a genesis block for the channel from
the channel creation transaction.

Change-Id: I57a4fbb609ae447d25041e64aaf71e151e83f68f
Signed-off-by: yacovm <yacovm@il.ibm.com>

CouchDB 2.3 changed format of fields update_seq and purge_seq.
This causes a problem when Fabric maps the CouchDB
JSON resposnes to structs.

Since Fabric doesn't use these fields anyways, they can be removed
from the Fabric CouchDB structs. This will make Fabric
compatible with CouchDB 2.3.

Change-Id: I89f789ef79ea485aa74b99ef047af417a8be044e
Signed-off-by: David Enyeart <enyeart@us.ibm.com>

This change set adds some more debug logs in the gossip pull
for blocks.

Change-Id: I9f2d3609c2ba5aba3e06d3741c061ce6e7bc90f0
Signed-off-by: yacovm <yacovm@il.ibm.com>
(cherry picked from commit ceef61fb282e645dc2b6b5d043ef5b45b7e41ffe)

This CR replaces the old rather opaque and unclear:

 Failed to reach implicit threshold of 1 sub-policies: required 1 \
  remaining

with

 implicit policy evaluation failed - 0 sub-policies were satisfied, \
  but this policy requires 1 of the 'Readers' sub-policies to be \
  satisfied

Although the message is a bit verbose, and likely could be made to be
even clearer, this attempts to strike a reasonable balance between
verbosity, and information leakage about policy structures to users
which are unauthorized and might simply be attempting to probe the
system for information on membership, etc.

Change-Id: Icd53e4438117936ca7254d08040e9fad27e24163
Signed-off-by: Jason Yellick <jyellick@us.ibm.com>

The gossip code that handles pull block data updates
has an optimization that first performs a simulation
of adding an inner block message to the message store,
and only if the simulation is successful  -
it then verifies the message.

The problem is that the simulation is done against the outer
message, and thus the optimization does not occur.

This change set fixes this.

Change-Id: I71c94e9b1cad2bfa6e9cfce700d8bd3209e9b5e7
Signed-off-by: yacovm <yacovm@il.ibm.com>

The gossip channel's method AddToMsgStore adds a block message
to both the block store and to the block puller.

However, it can be that the message isn't added to the block store
due to it being too old.

In such a case we must not add it to the block puller, because it is evicted
by the invalidation trigger of the block store which is programmed to
remove it from the block puller.

Change-Id: I64e7f4e64a337df5593a58a5d52a821b376b2bfb
Signed-off-by: yacovm <yacovm@il.ibm.com>

- MaxInflightMsgs is internal to etcd/raft and should be exposed
to users with a more appropriate name: MaxInflightBlocks

- MaxSizePerMsg is also internal to etcd/raft, and it's defaulted
to PreferredMaxBytes in BatchSize, so that if a big block is created,
it is sent in a its own etcd/raft message, instead of being batched
with other blocks. This parameter takes effect when a batch of entries
is sent to lagged node. During normal replication, each block is
sent in its own message.
  It's not necessary to expose this config option to users.

- SnapInterval is renamed to SnapshotIntervalSize

FAB-14593 #done

Change-Id: Icaf2848a41c5f0f0a02f4b0b4a80ba852fddd584
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

This change set:

- Adds the steam ID to the aborted error message for better
  readability and troubleshooting.
- Reverses the order between:
  - The log message that says that the stream was terminated.
  - The actual termination of the stream.

Change-Id: If3537b4770eb00d5c67b032bdb49303c9007e794
Signed-off-by: yacovm <yacovm@il.ibm.com>