1. 03 Mar, 2019 1 commit
    • Yoav Tock's avatar
      FAB-13264 consensus migration: kafka2raft green path #1 · 316db769
      Yoav Tock authored
      
      
      This is the first of four (1/4) sub-tasks that focus on the
      "green" path of consensus-type migration from Kafka to Raft. 
      
      By "green" we mean that there are no failures or aborts along
      the way. The flow of the green path and the changes made in
      these 4 tasks are described below. The 4 sub-tasks are staged
      in a way that minimizes dependencies between them.
      
      In this sub-task we introduce changes to the 
      orderer/common/bootstrap package (see details below).
      In essence, Just before the last config block of the system
      channel (COMMIT) is written to the ledger, the bootstrap file
      (a.k.a "genesis.block", do not confuse with the first block of
      the ledger) is swapped with the last block of the system
      channel. This sub-task extends package orderer/common/bootstrap
      to support this functionality.
      
      See respective JIRA item for further details.
      
      The "green" path for migration is the following:
      
      1. Start with a Kafka-based ordering service
      2. Send a config update tx (START-TX) on the system channel that:
       - Has ConsensusType.MigrationState=START
       - This will disable the creation of new channels
       - This will disable the processing of normal (standard channel) transactions
      3. Wait until the START-TX is committed and get the block height H of that tx
      4. Send a config update tx (CONTEXT-TX) on each of the standard channels that:
       - Has ConsensusType.MigrationState=CONTEXT
       - Has ConsensusType.MigrationContext=H
       - Has ConsensusType.Type="etcdraft"
       - Has ConsensusType.Metadata=<a marshaled etcdraft metadata: Consenters,
         Options, etc>
      5. Send a config update tx (COMMIT-TX) on the system channel that:
       - Has ConsensusType.MigrationState=COMMIT
       - Has ConsensusType.MigrationContext=H
       - Has ConsensusType.Type="etcdraft"
       - Has ConsensusType.Metadata=<a marshaled etcdraft metadata: Consenters,
         Options, etc>
       - The metadata should be the same as for the standard channels, with the same
         precautions.
       - If committed successfully, no further configuration will be possible
      6. Restart each orderer
       - The orderer will bootstrap into an etcdraft mode
       - Each channel will form a cluster
       - Normal transactions can resume now
      7. In order to configure the channels (system or standard), make sure that
         the first
         config update tx (on any given channel) after migration has:
       - Has ConsensusType.MigrationState=NONE
       - Has ConsensusType.MigrationS=NONE
       - In addition to other changes to the channel's config.
      
      Change-Id: Iccd146bb7260bafa4e4d8c4ee457d2ac19f5a642
      Signed-off-by: default avatarYoav Tock <tock@il.ibm.com>
      316db769
  2. 15 Dec, 2016 1 commit
    • Kostas Christidis's avatar
      [FAB-1364] Switch to provisional bootstrapper · 71a3389f
      Kostas Christidis authored
      https://jira.hyperledger.org/browse/FAB-1364
      
      
      
      All consenters read several of their config settings (think sharedconfig)
      from the genesis block that is generated by a bootstrapper. The only
      bootstrapper available so far is the static one. However, when testing we
      need to be able to modify several of these config values on the fly.
      
      Therefore the bootstrapper should be able to read a config object (which
      is itself created by reading the orderer.yaml file and -if set- its
      associated ENV vars).
      
      An example of that would be the KafkaBrokers value. For unit tests the
      "right" value is "127.0.0.1:9092", whereas for the current Docker
      Compose-based BDD tests the right value is "kafka0:9092".
      
      Since this bootstrapper is no longer static, renaming the package seemed
      appropriate. For production we will need to introduce file-based
      bootstrapper that reads the genesis block created by the genesis block
      tool.
      
      This changeset allows the bootstrapper to generate the appropriate
      genesis blocks (with their own custom keys) for all know consenters
      types: solo, kafka, sbft (though sbft for now just short-circuits to the
      solo case).
      
      The new bootstrapper is built in a way that makes further extensions
      easier, and minimizes code duplication by creating the appropriate
      structs and doing the necessary embeddings.
      
      The test that inspected the block's Data field was removed. It was a
      tautological test, serving no good purpose, and was also hard to
      maintain.
      
      Finally, this changeset updates the bootstrap helper signature by
      removing the error return; while the error return can be useful if the
      expectation is that the caller will be able to resort to a different
      bootstrapper if the first one fails, etc. one could that this amount of
      flexibility is unnecessary, and complicates the code needlessly. The
      bootstrapper should simply panic if the wrong settings have been passed
      to it, and the user should read the fine manual.
      
      Change-Id: I6aef9b19dbf9a39652d2d6b3ccbefc794e3001df
      Signed-off-by: default avatarKostas Christidis <kostas@christidis.io>
      71a3389f
  3. 07 Nov, 2016 1 commit
    • Jason Yellick's avatar
      Merge orderer protos into main protos · c0ba3c23
      Jason Yellick authored
      This commit begins the work of codifying a single set of common protos
      which both the orderer and fabric access, plus a specific set of orderer
      protos which are necessary to interact with the ordering service.  In
      the future, it is expected that the peer protos will go into their own
      directory (importing the common protos).
      
      This changeset looks a little scarier than it is, because the effects of
      changing these paths are far reaching indeed, but should be relatively
      easy to review.
      
      Note that this is the first ever cross-directory-dependent protos
      included in the fabric.  This means the generation incantations change a
      little.  Instead of importing the directory which contains the protos,
      instead, a common root directory must be picked.
      
      The compile_protos.sh has been changed to look for .protoroot files and
      if present use this as the proto compilation root, rather than the
      directory containing the proto.
      
      Change-Id: I619ea2c50ffd450640dafbddc7d7e5f34f809f3b
      Signed-of...
      c0ba3c23
  4. 24 Oct, 2016 2 commits
    • Jason Yellick's avatar
      Move orderer common components to common dir · 053980b0
      Jason Yellick authored
      As the complexity in the orderer grows, there is an increasing need to
      add new component directories, which do not actually correspond to
      different ordering services.
      
      This can be seen with the bootstrap, cauthdsl, and broadcastfilter
      directories, and more are on the way.
      
      This changeset simply creates a new common directory, and moves the
      common components there.  It also makes trivial updates to import paths,
      but otherwise introduces no new features, so does not require new tests.
      
      https://jira.hyperledger.org/browse/FAB-706
      
      
      
      Change-Id: I61e803aa4454926b4ad26610680eb543554346b8
      Signed-off-by: default avatarJason Yellick <jyellick@us.ibm.com>
      053980b0
    • Jason Yellick's avatar
      Define orderer genesis block by config · db22cdcf
      Jason Yellick authored
      The existing rawledger implementations have a hardcoded genesis block
      with non-sense data in it.  In order to support bootstrapping in the
      future, this genesis block will need to contain meaningful data, and
      hardcoding its contents in the rawledgers is not a viable option.
      
      This changeset moves the static definition into a special static
      bootstrap helper package, and removes the references to the static
      genesis block from the rawledgers.
      
      Because the rawledgers now need an additional argument for construction,
      the effects are felt rippling through the tests, but the changes are
      fairly straightforward.
      
      This satisfies JIRA issue:
      
      https://jira.hyperledger.org/browse/FAB-688
      
      
      
      Change-Id: I734a3215998c2977dd16e624869ceeb776767331
      Signed-off-by: default avatarJason Yellick <jyellick@us.ibm.com>
      db22cdcf
  5. 12 Oct, 2016 1 commit
    • Kostas Christidis's avatar
      Add Kafka-based orderer · f6640f22
      Kostas Christidis authored
      Related: http://jira.hyperledger.org/browse/FAB-32
      
      This changeset introduces a simple, solo-like Kafka-based orderer,
      complete with a config file that ties into the orderer config mechanism
      introduced in an earlier changeset, unit and BDD tests. It also provides
      a sample client that broadcasts and delivers counter values.
      
      For a demo of this work please watch this video:
      https://ibm.box.com/s/kqkk12vn18w3s3in2vkioucl9z32jk2h
      
      
      
      This changeset introduces the following abstractions:
      
      - Broker: Provides info on the atomic broadcast seek requests (earliest
      batch available, etc.)
      - Producer: Sends batches to Kafka
      - Consumer: Reads a stream of batches from Kafka
      - Client Deliver: A consumer dedicated to a connected client
      - Deliverer: Handles the deliver part of the Kafka-based orderer, spawns
      a new Client Deliver instance per connected client
      - Broadcaster: Handles the broadcast part of the Kafka-based orderer;
      cuts batches and sends them to Kafka
      - Orderer: Consists of a Deliverer and Broadcaster and, as the name
      suggests, handles all ordering requests (broadcast and deliver RPCs)
      issued by the connected clients.
      
      Change-Id: I09a313e9bf681051ee73b35d8d14401fee234f02
      Signed-off-by: default avatarKostas Christidis <kostas@christidis.io>
      f6640f22
  6. 26 Jul, 2016 1 commit