Change-Id: I07833487adc9ee26a0058192247ffd17536fb7a1
Signed-off-by: Matthew Sykes <sykesmat@us.ibm.com>

This change set hides etcdraft configuration from the orderer.yaml
in order for the configuration to be more consumable and user friendly.

Change-Id: Ic8e5c0aeeaf841313f6b5e6460d02cbd07243ea7
Signed-off-by: yacovm <yacovm@il.ibm.com>
(cherry picked from commit 56dc7991de0e14bc06e37713b2dba44348e6cf4f)

This change set adds logic in the etcdraft chain that detects
that the node is evicted from the channel, even if it was
disconnected from the cluster while it was evicted.

If a node fails sending to other nodes a consensus message,
it starts suspecting that something is amiss.

It then tries to pull the latest config block, and see
if it is in the channel or not.

If it is not, it:

1) Halts the chain
2) Pulls all blocks until the block that evicts
   the node.

Change-Id: Ic3526d834fbef515119bb899fe62d30fdcf53267
Signed-off-by: yacovm <yacovm@il.ibm.com>

This change set makes cluster type OSNs autonomously detect channels
that exist and that they should be part of (the channel configuration
has their public credentials as a consenter for the channel),
but that they do not run chains for, or have the blocks in their ledger.

This can happen from several reasons:
- The OSN is added to an existing chain, and since it didn't participate
  in the chain so far, it didn't get the blocks that tell it is now
  part of the channel.
- The OSN tried to detect whether it is part of a channel, but it
  wasn't able, because all OSNs of the system channel returned
  service-unavailable. This can happen if:
  - a leader election takes place
  - the network is acting up so the leadership was lost
  - a channel has been deserted (all OSNs left it).

To take care of such use cases, all OSNs now:
- Track inactive chains that they know of, but they do not participate in
- Periodically(*) probe the system channel OSNs to see if they are now
  part of these chains or not.
- If so, then they replicate the chains, and create instances of them,
  and replace the instances of the inactive chains in the registrar
  with the new instances of type etcdraft.

(*) - 10 seconds after boot, then after 20 seconds,
      then after 40 seconds, etc. etc. eventually- every 5 minutes.

Change-Id: I3c2a84e6f4f402e011e7a895345b3d3982247083
Signed-off-by: yacovm <yacovm@il.ibm.com>
Signed-off-by: Artem Barger <bartem@il.ibm.com>

This change set, adds an option for a separate TLS listener
for intra-cluster communication.

Change-Id: I059e4d45ddeaf066017c758b83a3e7422783a403
Signed-off-by: yacovm <yacovm@il.ibm.com>
Signed-off-by: Artem Barger <bartem@il.ibm.com>

The doc around TLS and client certificates
was not clear, specifically that client
certificates are actually required even when
the clientAuthRequired setting is false.

Change-Id: I37f1718bb74d0139a3a36fa917b82ddeb0abff1a
Signed-off-by: Gari Singh <gari.r.singh@gmail.com>
Signed-off-by: David Enyeart <enyeart@us.ibm.com>

FAB-13084 #done

Change-Id: I808e7566227a59fa55a408c8b1e5e618f740218d
Signed-off-by: Matthew Sykes <sykesmat@us.ibm.com>

This CR updates the orderer to use the new logging
environment variables FABRIC_LOGGING_SPEC and
FABRIC_LOGGING_FORMAT.

FAB-12663 #done

Change-Id: I2e69107f1227d60ba86829f5625af91bcc310392
Signed-off-by: Will Lahti <wtlahti@us.ibm.com>

This change set addresses minor cosmetic issues pointed out
in code review to the orderer cluster commit chain.

Change-Id: I1e27f69e251d9946d21f4eb5a4d890de214b7378
Signed-off-by: yacovm <yacovm@il.ibm.com>

This CR adds an opaque section 'Consensus' in orderer.yaml to
contain arbitrary config options needed by consensus plugin.

Consensus plugin should use viperutil.Decode to decode the
content of this section to a predefined struct.

In this way, we don't need to expand TopLevel config for each
consensus plugin, therefore maximizing flexibility.

Change-Id: Ie657f1ca69cc79a8aaaf629d2df1bd0164cf5164
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

This change set prepares the block puller to be attached
to the etcdraft consenter and chain.

It adds the needed configuration in the orderer.yaml,
and also adds code that creates a block puller.

Change-Id: Ieb7c430613b16ba8625b323cdd8e170ed073ee1a
Signed-off-by: yacovm <yacovm@il.ibm.com>

This change set:

1) Wires the consenter into the registrar
2) Wires the communication to the consenter and
   to the registrar, in preperation for multi-node Raft.
   This makes a new chain configure the communication layer
   of the cluster according to the TLS certificates
   of the channel.
3) Adds a unit test that spawns a Raft OSN

Change-Id: I3923f6f5e0fae6428c2be2e056b82355332f3324
Signed-off-by: yacovm <yacovm@il.ibm.com>

Change-Id: Ibed9be1aa0bdf7b5a6606691a841a87f505ab48c
Signed-off-by: yacovm <yacovm@il.ibm.com>

The Kafka AdminClient API is now used in order
to create the topic required for a channel.
Prior to this, the orderer relied on the Kafka
cluster to have auto create topics enabled.

This API is only supported for Kafka v0.10.1.0
and higher.

If the API fails, the code still falls through
to the auto create path.

Change-Id: Id23d5b5f5826b24cb0cc3202e3ac6088e5ae3a49
Signed-off-by: Gari Singh <gari.r.singh@gmail.com>

Add support for using the SASL/PLAIN
mechanism for authentication with
Kafka brokers

Change-Id: I5064a033c9857ec60d0efff2f1f2c5c705b4d037
Signed-off-by: Gari Singh <gari.r.singh@gmail.com>

Our users are frequently confused because the sample configurations
set to the sample organization's MSP ID of 'DEFAULT'.  This leads to
very misleading error messages, and it is not at all clear that
'DEFAULT' should be changed to match their organization.

Unfortunately, many of our 'unit tests' are actually integration tests
which hardcode the string 'DEFAULT', so this CR (which should only be a
few lines long) actually cascades through much of the code.  These tests
should be updated to remove their external dependency on the sample
configuration, but, that is outside the scope of this CR.

Change-Id: Icf56924629b69400e0d70991a6ecfb3e831b0641
Signed-off-by: Jason Yellick <jyellick@us.ibm.com>

The peer turns on mutual TLS with clientAuthRequired, while the orderer
turns on mutual TLS with ClientAuthEnabled.  This is inconsistent, and
as it's a v1.1 feature, may be safely fixed without worry for backwards
compatability.

Change-Id: Ic3c2817177df0dae6e76bc3d6c8cd965d0502d40
Signed-off-by: Jason Yellick <jyellick@us.ibm.com>

- even more forgiving kafka version parsing
- removed default Kafka.Version from orderer.yaml
- kept default value of Kafka.Version in orderer binary as 0.10.2.0
  (future changeset to move this to 1.0.0 once kafka image has
   been updated)

Change-Id: I9220fccd6ebb03529458367c21dd0256483abb8c
Signed-off-by: Luis Sanchez <sanchezl@us.ibm.com>

- Tweak the documention of Kafka.Version provided
  as comments in the orderer.yaml.
- Update the Fabric documentation to provide better
  guidance on setting Kafka.Verion and on its impact
  after an upgrade.
- NOTE that Fabric would, in effect no longer have a
  'supported' version of Kafka. Also note that does
  not stop downstream projects, such as ibmblockchain,
  from declaring which versions of Kafka they support.

Change-Id: I4da50d1186dedd18618b637edeb757c8cbdd5f7c
Signed-off-by: Luis Sanchez <sanchezl@us.ibm.com>

This CR adds validation checks for the timestamp (all envelopes) and
the TLSCertHash (only if mutual TLS is enabled) of incoming Deliver
envelopes.

Change-Id: I9359ee0ed5bf9057e91948951c59a2196c484afc
Signed-off-by: Will Lahti <wtlahti@us.ibm.com>

Currently the gRPC server keepalive parameters
are hard-coded to the defaults set in the
core/comm package.  This change makes the
following settings configurable:

- minimum permitted interval for client
 pings on the peer endpoint

- keepalive interval and timeout for
 server to client pings

Change-Id: I5f4c4b330f7d31483b63e8fd293039fd1bb352b9
Signed-off-by: Gari Singh <gari.r.singh@gmail.com>

Deliver clients no longer fail at the first sign of
trouble with the orderer/kafka cluster link.

- Custom sarama logger enhanced to allow listening to logging
  'events'.

Change-Id: I4a30e86ff77fd3555e1b67cf79fd44a04ab352f9
Signed-off-by: Luis Sanchez <sanchezl@us.ibm.com>

1. In accordance to what we do for all other default values, the default
Kafka version should be filled out in the orderer.yaml config files.
2. The default value for Kafka was actually set to 0.10.1 even though
our images use 0.10.2. This changeset fixes this to 0.10.2.

Change-Id: I7c8f0da1421d1cca6ec67b494f60debd85fbf5f4
Signed-off-by: Kostas Christidis <kostas@christidis.io>

[FAB-5981] Fix syntax error in order.yaml,the `BroadcasTraceDir` should
be `BroadcastTraceDir`.

Change-Id: I45de13241e5bec38e064e046381d1079c3a7717a
Signed-off-by: Zhangjiong Xuan <xuanzhangjiong@hyperchain.cn>

- Updated vendored sarama to v1.12.0
- Updated default kafka to latest version supported by sarama

Change-Id: I56e49acc511005292fcecaaf1538d29eebc8232f
Signed-off-by: Luis Sanchez <sanchezl@us.ibm.com>

The orderer's logs always print using the default format string which
includes colors in the output.  Although helpful when looking at a
console, this makes logs difficult to handle programatically.

This CR adds a LogFormat option to orderer.yaml and uses it to
initialize the logging format for the orderer.

Change-Id: I7c595aa42e34255114953fab3dcbeafffc5cd377
Signed-off-by: Jason Yellick <jyellick@us.ibm.com>

For hard to debug problems, actually having access to the bytes of a
message can be the only definitive way to diagnose.

This CR adds a debug configuration section, and allows for setting a
directory to log all Broadcast messages, as well as a directory to log
all Deliver messages.

Although there is no support for dynamically changing debugging
parameters at this time, the code deliberately retrieves the debug
parameters from the debug struct at every instance to allow for dynamic
control of the debugging.

Change-Id: Ib046f263dc95b374b5883af66397a9d29049ffef
Signed-off-by: Jason Yellick <jyellick@us.ibm.com>

Kafka version can now be set in orderer.yaml, eg:

Kakfa:
  Version: 0.10.0.0

Default value is '0.9.0.1'.

Change-Id: Ieb69b003b400bff9c7918743e99ce76c4ad4363c
Signed-off-by: Luis Sanchez <sanchezl@us.ibm.com>

Change-Id: I78ce310a041666a8c7a5c047c3f80095af36e307
Signed-off-by: Kostas Christidis <kostas@christidis.io>

Fix duplicated words in orderer.yaml.

Change-Id: I80e6677090972bd8155b0195a7ed920aee943795
Signed-off-by: Baohua Yang <yangbaohua@gmail.com>

Change-Id: I9da1365a65ec8b5b63e784d67056c4c130449a9a
Signed-off-by: Kostas Christidis <kostas@christidis.io>

This changeset modifies the `Retry` options of the Kakfa-based orderer
configuration. It adds short/long retry interval switches as suggested
in FAB-4121 and exposes some retry/timeout-related settings from the
underlying sarama client library. Defaults are introduced for all of the
new options.

This is part of fixing FAB-4136.

Change-Id: I75b9a726a9948e0c68e062f6500b6376c3a1bba7
Signed-off-by: Kostas Christidis <kostas@christidis.io>

Change-Id: Ib1a1b78ce34193a164dde8500669ea9b58eb20fb
Signed-off-by: Luis Sanchez <sanchezl@us.ibm.com>
Signed-off-by: Kostas Christidis <kostas@christidis.io>

Removed all traces of sbft package.

Change-Id: Ia3b91b67d69abd8f0c320537ce0f7dfcb4086c1b
Signed-off-by: Luis Sanchez <sanchezl@us.ibm.com>

use SPDX identifiers

Change-Id: Id5c02190bd07ec40eb4b1d7677f38b0736098527
Signed-off-by: Christopher Ferris <chrisfer@us.ibm.com>

This limits an end users options to override the config
how they see fit.  Instead, we put all defaults into
orderer.yaml.

Fixes FAB-3641

Change-Id: I3c3caa6c37bb3cbf99f73e6245c1f4b6a1eb96a7
Signed-off-by: Gregory Haskins <gregory.haskins@gmail.com>

The current defaults for the orderer have it:

1. Use the "ram" ledger
2. Set the "file" ledger's location to /tmp which means it won't persist
across restarts.

Both of these choices are unacceptable for a production environment.
This changesets sets the "file" ledger as the default and its location
to a path outside the /tmp dir, and inline with the peer's ledger.

Change-Id: I59413ceb45d0676d00fd2204e0bda9a659600dec
Signed-off-by: Kostas Christidis <kostas@christidis.io>

This patch structures the cryptogen output in a way that
makes it more directly consumable.  The output for each
node looks like:

└── peer4.org1
    ├── msp
    │   ├── admincerts
    │   │   └── Admin@org1-cert.pem
    │   ├── cacerts
    │   │   └── org1-cert.pem
    │   ├── keystore
    │   │   └── 0aa7a89070ce8322a4dc3fac2206fa8313b88fb625c70963934714d4129d2897_sk
    │   └── signcerts
    │       └── peer4.org1-cert.pem
    └── tls
        ├── ca.crt
        ├── server.crt
        └── server.key

The notable differences are that we push the msp content down under ./msp, and we
add the ./tls directory.  The crypto material under tls is simply duplicated
content from the MSP, as appropriate, to present a consistent layout for
consumption by the TLS layer.  We also update the default paths in the config
to be consistent with this layout.

Fixes FAB-3453

Change-Id: I8e149035b92a4758d6c87c03306c08b82687683f
Signed-off-by: Gregory Haskins <gregory.haskins@gmail.com>

Introduction
======================
The primary goal of this patch is to create the notion of a
"config-relative" path reference.  For example, a configuration file
"/etc/foo/bar.yaml" that contains a key "bat" with a value
"baz/blaz/blamo" can be used to specify that "baz/blaz/blamo" should
be considered relative to the configuration file itself.  In this case,
it would be expected to be found at /etc/foo/baz/blaz/blamo.  FAB-2037
does a much more thorough job of explaining the rationale on why
config-relative is considered important/good-form.

This is in stark contrast to what we have today, which is a jumbled
mess of assumed GOPATH relative, CWD relative, ENVVAR absolute and
sometimes even ENVVAR relative.  Therefore, an additional positive
side-effect of this endeavor is that this patch also substantially
cleans up some technical debt that had been accumulating in the tree
for some time related to ad-hoc pathing, DRY violations, and just
general inconsistencies in how configuration files were managed.

Design Details
==========================
This patch refactors the basic configuration system into the notion of
a tree rooted at a configuration-path.  By default, this path is
$GOROOT/..../fabric/sampleconfig during dev/test and
/etc/hyperledger/fabric during runtime.  The root may be overridden
at any time by specifying the environment variable FABRIC_CFG_PATH.
(Note that this variable unifies and replaces the former PEER_CFG_PATH
and ORDERER_CFG_PATH).

The dev/test environment will operate out of the ./fabric/sampleconfig
configuration root.  The build-system will package that root into
/etc/hyperledger/fabric in the runtime context with the intention of
the end-user/admin/deployer replacing parts or all of the sampleconfig
to suit their application.

Since configuration-relative paths are now possible, the configuration
files may reference other relative files and they will behave
appropriately regardless of the context in which they are executed.
For example, consider the files ./sampleconfig/tls/server.[crt|key].
A configuration file may contain a key "tls/server.key" and the system
will properly resolve this relative file even at runtime. This is (IMO)
far more natural than assuming a path is relative to the CWD of where
the command is executed, which is how most of the system behaves today
(or requires awkward and very specific ENVVAR overrides).

This will be conducive to something like a package-installer
(e.g. RPM/DEB) or a docker environment to augment/replace elements
of the configuration root and to freely move the configuration around
as the package/deployer sees fit.

As an example, a deployment on Kubernetes might opt to volume mount
/etc/hyperledger/fabric to replace the entire config, or it might just use
a secrets mount on /etc/hyperledger/fabric/peer/tls.  An RPM packager
might opt to install the configuration files in the default
/etc/hyperledger/fabric, whereas an unprivledged user might install them
in ~/hyperledger.  The point is, it shouldn't matter where they are and the
user shouldn't need a PhD in CORE_* variables to get it to work.

This is part of an overall effort to improve the user-experience as we
march towards a v1.0 release.

Fixes FAB-3169 as part of FAB-2037

Change-Id: I5f47f554c2f956ec2e1afebd9bd82b0bbb62892a
Signed-off-by: Greg Haskins <gregory.haskins@gmail.com>

https://jira.hyperledger.org/browse/FAB-2591



This changeset allows the user to choose the file ledger, introduced in
a previous changeset.

Change-Id: Id518961af10fddb4954291592cf53b12f1e3d1fa
Signed-off-by: Kostas Christidis <kostas@christidis.io>