- MaxInflightMsgs is internal to etcd/raft and should be exposed
to users with a more appropriate name: MaxInflightBlocks

- MaxSizePerMsg is also internal to etcd/raft, and it's defaulted
to PreferredMaxBytes in BatchSize, so that if a big block is created,
it is sent in a its own etcd/raft message, instead of being batched
with other blocks. This parameter takes effect when a batch of entries
is sent to lagged node. During normal replication, each block is
sent in its own message.
  It's not necessary to expose this config option to users.

- SnapInterval is renamed to SnapshotIntervalSize

FAB-14593 #done

Change-Id: Icaf2848a41c5f0f0a02f4b0b4a80ba852fddd584
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

There are presently two etcdraft protos around metadata.  One is the
metadata stored in the config and it is named 'Metadata', the other is
the metadata stored in each block, this is named 'RaftMetadata'.  This
causes confusion when reading the code.  This CR transforms those names
to be:

 Metadata -> ConfigMetadata
 RaftMetadata -> BlockMetadata

Change-Id: Ia0394ebe78f5541996c010c3c67d760f336f75d8
Signed-off-by: Jason Yellick <jyellick@us.ibm.com>

A max of 10 transactions per block is too low because
it creates tiny blocks. We should increase it to a
saner value which is more optimized for throughput
instead of letting the users always do it themselves.

Correspondingly, 256 blocks circulating the consensus
is too high and should be tuned down appropriately.

Change-Id: Ib2f2c0f9c1875102e14d73f054f67103036b3d8d
Signed-off-by: yacovm <yacovm@il.ibm.com>

Instead of uint64 with unit Millisecond, use string in time
duration for TickInterval for etcdraft chain options, e.g. 500ms

Change-Id: I1549d4278654b04baa6e3ab6a93de1d9c453cdcc
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

Instead of taking snapshot every N blocks, this CR
changes it to taking snapshot every N bytes.

This also sets default SnapshotInterval to 100MB, if
it's unset. Otherwise data in memory is never compacted
till OOM.

Meanwhile, DefaultSnapshotCatchUpEntries is shrunk so
it does not take too much space to preserve extra entries
every time a snapshot is taken. Slow nodes are catching up
using blockpuller, which is also efficient.

Change-Id: I79cfeb8652fcbafdeb5793bf4f06267b95a858d6
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

Increase default etcdraft tick interval to 500ms, for several reasons:
- in a WAN/Cloud environment, this is more realistic
- WAL sync in CI often exceeds 1s, which causes heartbeats not being
  sent timely. Increasing election timeout can decrease the chance of
  unexpected leader failover.

This CR also increases default timeout of peer cmd, because now
it takes 5~10s to elect a leader for a newly created channel, and
`peer channel create` can only retrieve genesis block of that channel
when leader exists (Deliver API returns error if leaderless).

Note that this value is still configurable by users.

Change-Id: I94fbbc750fa096cce6ef9e2d65eb981c6202b675
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

A package initializer buried in the configtxgen/localconfig package
modifies the logging spec in the orderer at runtime. Remove it.

Change-Id: I11bc289769b7069b0cad384b50ec8588e7f9a714
Signed-off-by: Matthew Sykes <sykesmat@us.ibm.com>

FAB-12610 #done

Change-Id: If71efa93555adb3445945c0324acfda4cc262974
Signed-off-by: Will Lahti <wtlahti@us.ibm.com>

Loggers shouldn't need to "register" in order to determine
their level. Updated the Level() logic to evaluate a logger's
module name against the current log spec.

FAB-12528 #done

Change-Id: I54bf378cd021ad1e4310b16cf1e313c9e4a39bb0
Signed-off-by: Saad Karim <skarim@us.ibm.com>
Signed-off-by: Will Lahti <wtlahti@us.ibm.com>
Signed-off-by: Matthew Sykes <sykesmat@us.ibm.com>

Let the tool do what it wants with the imports.

Change-Id: I115230369be6349d89b5869951ce088ce38d8fb6
Signed-off-by: Matthew Sykes <sykesmat@us.ibm.com>

This CR adds configuration defaults to the etcdraft plugin Metadata
definition under the Options namespace.

Change-Id: I37214d6ff3da9e7f5a26820feaea882d49e79736
Signed-off-by: Adarsh Saraf <adarshsaraf123@gmail.com>

Replace go-logging with a zap based implementation. The implementation
is mostly compatible with the legacy format but differs in a two key
ways:

1. CRITICAL and NOTICE log levels no longer exist. Calls to Critical get
   mapped to error and NOTICE gets mapped to INFO. The methods
   associated with these levels will be removed in a future CR.
2. The log level constants are now sourced from the zap/zapcore packages.
   These values are incompatible wit the go-logging constants. Please be
   sure to use the appropriate constant when necessary.

The existing go-logging package is still used by the chaincode shim to
reduce the risk of negatively impacting legacy chaincode.

Change-Id: Iaf5fac807244883a8285892ccd350c5256959782
Signed-off-by: Matthew Sykes <sykesmat@us.ibm.com>

This changeset:

1. Defines consenter metadata for the etcd/raft plugin, creates the
associated metadata message factory, and registers it with the metadata
factory map in protos/orderer for protolator to work.

2. Adds supports for etcd/raft configuration to the channel
configuration encoder.

3. Adds an "etcdraft" section to the configtx sample YAML and the
backing config structure.

4. Adds a "SampleDevModeEtcdRaft" profile preset to the configtx sample
YAML. This will be consumed by integration tests.

Change-Id: I7460c89fdb91e68e4217932aeed5ca41a131d47b
Signed-off-by: Kostas Christidis <kostas@christidis.io>

Go vet attempts to detect missing format arguments or format strings in
non-format contexts.  However, the default list of functions does not
include many of our logging functions (like Warningf, Debugf, etc.).
This CR enables the additional vetting and fixes the errors that it
catches.

Change-Id: I4eefdf84d9349f2ae08da9bb0c9b829226219c44
Signed-off-by: Jason Yellick <jyellick@us.ibm.com>

This CR adds a -configPath argument to configtxgen,
which can optionally be set to specify the location
of the desired configtx.yaml.

Change-Id: I5b3b273dc91a3349db3062f9f25ed516a47b3a27
Signed-off-by: Will Lahti <wtlahti@us.ibm.com>

- Add variadic argument for extra search paths to configtxgen/localconfig
  Load and LoadTopLevel functions. This is primarily intended to be used
  in tests.
- Create configtxgentest package that loads configurations and profiles
  from the development sampleconfig directory.
- Add method to configtest to setup and reset FABRIC_CFG_PATH.
- Update tests to use configtxgentest and/or configtest as appropriate.

Change-Id: I6a66491bfbc0048f5b38babf6a55fb918a439fe6
Signed-off-by: Matthew Sykes <sykesmat@us.ibm.com>

When generating yaml file using localconfig the
MaxMessageSize label is used instead of MaxMessageCount.
This fixes the issue.

Change-Id: I40ff56be2e777df83a2cd955c056ab6a9b573b02
Signed-off-by: Latitia M Haskins <latitia.haskins@gmail.com>

Add ACLs property to Application in configtx along with
defaults and allow configtxgen to read it.

Change-Id: Ib2cad64db3f272b4760743e3177c092c6b2a31b2
Signed-off-by: Srinivasan Muralidharan <srinivasan.muralidharan99@gmail.com>

This CR adds the ability to specify policies in configtx.yaml.  It is
one half of two required for users to specify their own policies.  The
second half is in FAB-9255 (submitted together) which takes these
additional config file options and encodes them into channel config.

Change-Id: I7ba264a7554d4e3656bf13f2191bd871ba401e21
Signed-off-by: Jason Yellick <jyellick@us.ibm.com>

For all the non-backwards compatible work, including the new chaincode
lifecycle and resources config changes, we must use a new capability to
determine whether the behavior is enabled or not.

This CR adds one for the application portion of the config.  At the time
of this writing there are no features which would introduce
incompatibilities at the orderer or channel level, so those capabilities
are being left in place.

This CR also removes the assorted 'V11' profiles, as maintaining a set
of profiles for each version is unmaintainable and confusing.  For this
and subsequent releases, configtx.yaml will contain only defaults for
the current version of fabric.  Users may of course override these
defaults by modifying the config file.

Change-Id: I08729fef76292044440ebc7e4f39145b0640afc4
Signed-off-by: Jason Yellick <jyellick@us.ibm.com>

In v1.0, chaincode instantiation was always controlled by a policy which
required a single signature.  By default, we wish for chaincode
instantiation to require a quorum of administrators to sign off, but
doing so may break existing workflows.

This CR adds a field to configtx.yaml which specifies the default
mod_policy to use throughout the resources tree.  If set to a single
signature policy like /Channel/Application/Writers, then the old workflow
of a single signer is still possible.  If set to a multi-sig policy like
/Channel/Application/Admins, then the user's workflow will need to be
changed to accomodate the additional signing requirements.

Change-Id: I021801f1540ee5ee25d110682852564650421dbd
Signed-off-by: Jason Yellick <jyellick@us.ibm.com>

This changeset at its core, simply enables embedding a default resource
tree into the output of channel creation transactions which are created
with v1.1 application capabilities.

Unit tests have been added. Additionally, the new channel creation tx
behavior can be confirmed by issuing:

 configtxgen -profile SampleSingleMSPChannelV1_1 \
  -outputCreateChannelTx foo.tx -inspectChannelCreateTx foo.tx

and observing the resources tree in the JSON output.

Because the utility function for creating the channel creation
transaction is used widely throughout the tests, and the function
parameters needed to change significantly, there is a fair amount of
churn in the tests, but none in production code.

Change-Id: Id719a1ae3f309a0a189f566dc0fef409564b519e
Signed-off-by: Jason Yellick <jyellick@us.ibm.com>

This change set introduces an 'MSPType' field to configtx.yaml, so that the
type of MSP can be specified. This permits credentials for other MSP types,
most notably the idemix MSP, to be included in genesis tx/blocks so that they
can be used as well. If no MSPType is specified, the standard one is
automatically assumed.

Change-Id: I5a609b57cfe85ae030f7a8fbecf8cfd71d3074ae
Signed-off-by: Alessandro Sorniotti <ale.linux@sopit.net>

configtxgen is unable to read in the sample V1.1 profiles due to the
period in the profile name as defined in sampleconfig/configtx.yaml
(e.g. SampleSingleMSPChannelV1.1). This CR updates the profile names
to use underscores instead so Viper is able to locate and load them
as expected.

Change-Id: Ic7bf03199c20eca466519c406c702221b071602c
Signed-off-by: Will Lahti <wtlahti@us.ibm.com>

One of the most common questions when attempting to add an organization
to a channel is how to get the JSON definition for that organization.
The only real way to do this at present is to generate a genesis block,
decode the block, then go look for the org definition.

This CR adds a '-printOrg' flag to configtxgen which will print the
organization's definition as JSON to make this process easier.

Change-Id: Id12098f36feae8ca4751b82afaf2914166077e50
Signed-off-by: Jason Yellick <jyellick@us.ibm.com>

With the introduction of FAB-6139 a new more readable and flexible
config encoder is available.  This means all existing usage of the old
deprecated provisional encoder should be removed.  This CR does that.

Change-Id: If64e0e7dc920c709cc39a111547c61c3ae4dc3f0
Signed-off-by: Jason Yellick <jyellick@us.ibm.com>

As a prerequisit to any real usage of the new capabilities support, it
must be possible for an admin to generate channel creation requests and
or bootstrap the ordering system network with the new configuration
keys.

This CR adds support to configtxgen to do this, though there are no
capabilities actually supported yet.

Change-Id: I9ab962727e446f593ccd262cb31ac89357cdf9d9
Signed-off-by: Jason Yellick <jyellick@us.ibm.com>

This changeset adds Kafka-based profiles, based on the existing solo
ones.

It ripples onto more files than it should have because I'm renaming
some existing profile constants for consistency.

Change-Id: Ic85c141a2be0677e46f8f42c9013e25887f07023
Signed-off-by: Kostas Christidis <kostas@christidis.io>

The configtxgen code is currently under fabric/common/configtx/tool
which is really a poor location for it.

For consistency with the other tool components, it should be moved to
common/tools/configtxgen.  This CR does that.

Change-Id: I72a6060c119a906343971fcb10a4dc53e557de85
Signed-off-by: Jason Yellick <jyellick@us.ibm.com>

Change-Id: Ib255d0f1601864bdaac9e6e272fa2fdc89b4f76b
Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

The `Producer.MaxMessageBytes` default we were setting on the Kafka
producers was incorrectly set to Kafka's default value for
`socket.request.max.bytes`. (I actually set it to a value that was just
a byte shorter than this when I pushed in the refactor changeset a few
days ago in order to silence the sarama logs, and added a FIXME note for
this very reason.) This value is incorrect because it doesn't account
for headers. This changeset assumes a too-generous 1MB for headers and
decreates `Producer.MaxMessageBytes` accordingly.

This changeset also updates the default value for AbsoluteMaxBytes in
`configtx.yaml`. The current value is incorrect because it doesn't
account for the wrapping of messages in the container `KafkaRegular`
format. At a minimum, it should be decreased compared to
`Producer.MaxMessageBytes` by an equal padding. This changeset does this
for all `configtx.yaml` files currently in our codebase.

Finally, it sets the default, suggested value for AbsoluteMaxBytes in
the sampleconfig `configtx.yaml` file to a much more reasonable 10 MiB,
well within the constraints we outlined above. The default check in
`localconfig` has been updated accordingly.

See the JIRA item for a detailed explanation of the issue.

Change-Id: I8821982707282c5069c1a66dd2a121b894b3a3c8
Signed-off-by: Kostas Christidis <kostas@christidis.io>

Because configtxgen only collects crypto material, it does not actually
validate it, (ie signing, or checking signatures), the BCCSP information
attached to the MSP definitions is unnecessary.

Its presence complicates the config file for no gain, and there have
been issues reported around failing to set these parms properly.

Change-Id: I9728b69a66fbbc21373eba59693481d1e6156615
Signed-off-by: Jason Yellick <jyellick@us.ibm.com>

There are a few incorrect things about the default configuration
produced for the ordering system channel by the configtxgen tool.

The consortiums definitions and channel creation policy cannot be
modified after creation, and the ordering system channel includes an
Applications section by default which is inappropriate.

This CR fixes both these issues.

Change-Id: Ie6a934198fafbb37065d703edc0bc8fc9208b9ca
Signed-off-by: Jason Yellick <jyellick@us.ibm.com>

This simply helps the panic msg to be more clear to follow.

Change-Id: I04e86a9a6679c612759f5b82440c10648e9a82d2
Signed-off-by: Baohua Yang <baohyang@cn.ibm.com>

This patch simply adds an INFO level log message to indicate
which configuration was ultimately selected.  This may be
helpful to the user since the process for selecting the
config may be automated by the software.  This could be
confusing if the user believes one configuration was in
effect when the system loaded an alternate.

Fixes FAB-3428

Change-Id: I085c4b719c85a9d7d4af86843f89ac819da9b6e4
Signed-off-by: Greg Haskins <gregory.haskins@gmail.com>

There is currently no inspection of the channel creation transaction.
This means that during a channel creation, the creator could
inappropriately specify config, including items like orderer parameters,
and the ordering service would silently accept this incorrect
parameters.  This is especially troublesome from a security perspective,
as a specially crafted config tx could potentially trick another user
into believe they were participating in a channel with one member when
they are actually participating with another.

This CR adds this checking by leveraging the existing underlying
authorization mechanisms of the configuration transaction.  It ensures
that the creating members are not only authorized to create the channel,
but also ensures that all channel config that is set is done so in
accordance with the underlying governance for that config element.  This
means that a collection of authorized ordering organizations may choose
to modifying ordering parameters in a channel creation request while a
set of application orgs would not be able to.

This CR is somewhat large due to the significant invasive nature of this
change, and that the entire end to end must be modified, not just a
single component.  However, given the increase in function with decrease
in complexity, the diff stat is hopefully acceptable.

Change-Id: I9bb64d3f70d07eb8a8a4ea4893c71cf1112b4bf7
Signed-off-by: Jason Yellick <jyellick@us.ibm.com>
Signed-off-by: Kostas Christidis <kostas@christidis.io>

Currently, the configtxgen tool automatically outputs admin policies
which require the signature of an admin certificate in order to satisfy
them.

For some deployment scenarios, especially our current e2e scenarios,
it's not possible to sign with an admin cert, so this CR adds an option
to specify the admin principal to be of type Role.MEMBER.

Change-Id: If0f05fde2f726b88c54d6bdbc81e210c63bfa959
Signed-off-by: Jason Yellick <jyellick@us.ibm.com>
Signed-off-by: Kostas Christidis <kostas@christidis.io>

In order to support returning the list of channels via the ordering
system channel configuration, there needs to be additional configuration
structures representing the groups which may create channels.

We call these groups Consortiums by convention, where each consortium
represents a group of organizations which may form channels together.

Note, this also include FAB-2647 as combined with FAB-2700 it provides
testing coverage for the new code.

Change-Id: Iaa7dada39daa153c95613b47def8169b57f27be4
Signed-off-by: Jason Yellick <jyellick@us.ibm.com>
Signed-off-by: Kostas Christidis <kostas@christidis.io>

Introduction
======================
The primary goal of this patch is to create the notion of a
"config-relative" path reference.  For example, a configuration file
"/etc/foo/bar.yaml" that contains a key "bat" with a value
"baz/blaz/blamo" can be used to specify that "baz/blaz/blamo" should
be considered relative to the configuration file itself.  In this case,
it would be expected to be found at /etc/foo/baz/blaz/blamo.  FAB-2037
does a much more thorough job of explaining the rationale on why
config-relative is considered important/good-form.

This is in stark contrast to what we have today, which is a jumbled
mess of assumed GOPATH relative, CWD relative, ENVVAR absolute and
sometimes even ENVVAR relative.  Therefore, an additional positive
side-effect of this endeavor is that this patch also substantially
cleans up some technical debt that had been accumulating in the tree
for some time related to ad-hoc pathing, DRY violations, and just
general inconsistencies in how configuration files were managed.

Design Details
==========================
This patch refactors the basic configuration system into the notion of
a tree rooted at a configuration-path.  By default, this path is
$GOROOT/..../fabric/sampleconfig during dev/test and
/etc/hyperledger/fabric during runtime.  The root may be overridden
at any time by specifying the environment variable FABRIC_CFG_PATH.
(Note that this variable unifies and replaces the former PEER_CFG_PATH
and ORDERER_CFG_PATH).

The dev/test environment will operate out of the ./fabric/sampleconfig
configuration root.  The build-system will package that root into
/etc/hyperledger/fabric in the runtime context with the intention of
the end-user/admin/deployer replacing parts or all of the sampleconfig
to suit their application.

Since configuration-relative paths are now possible, the configuration
files may reference other relative files and they will behave
appropriately regardless of the context in which they are executed.
For example, consider the files ./sampleconfig/tls/server.[crt|key].
A configuration file may contain a key "tls/server.key" and the system
will properly resolve this relative file even at runtime. This is (IMO)
far more natural than assuming a path is relative to the CWD of where
the command is executed, which is how most of the system behaves today
(or requires awkward and very specific ENVVAR overrides).

This will be conducive to something like a package-installer
(e.g. RPM/DEB) or a docker environment to augment/replace elements
of the configuration root and to freely move the configuration around
as the package/deployer sees fit.

As an example, a deployment on Kubernetes might opt to volume mount
/etc/hyperledger/fabric to replace the entire config, or it might just use
a secrets mount on /etc/hyperledger/fabric/peer/tls.  An RPM packager
might opt to install the configuration files in the default
/etc/hyperledger/fabric, whereas an unprivledged user might install them
in ~/hyperledger.  The point is, it shouldn't matter where they are and the
user shouldn't need a PhD in CORE_* variables to get it to work.

This is part of an overall effort to improve the user-experience as we
march towards a v1.0 release.

Fixes FAB-3169 as part of FAB-2037

Change-Id: I5f47f554c2f956ec2e1afebd9bd82b0bbb62892a
Signed-off-by: Greg Haskins <gregory.haskins@gmail.com>

This CR updates all loggers throughout the code base to use
`flogging.MustGetLogger`. This function wraps `logging.MustGetLogger`
and tracks the logger modules defined in the system. This enables the
ability to set log levels for modules using regular expressions.
and make it easy to change the levels for any module and all its
submodules with a single command (e.g. gossip, ledger, msp).

Change-Id: If5d3229ea2312adb56fc21bfdafbed3d967cf1df
Signed-off-by: Will Lahti <wtlahti@us.ibm.com>