The aim of this change set is to apply the well-established "Secure by
default" security principle to the way the validator validates transactions
in a block.

The current code behaves as follows: create an array of validation codes, set
by default to "all transactions are valid"; then perform validation which may
mark transactions as invalid. Furthermore, in other parts of the code, if no
array of validation codes is yet persent in the block, a new one is
indiscriminately created (again, marking all transactions as valid). This
approach is a security anti-pattern because it opens up to attacks where an
adversary may force the code through a path where the default "tx is valid"
validation code is maintained even for invalid txes.

This change set ensures that validation code arrays are created and set to a
new value (TxValidationCode_NOT_VALIDATED) which ensures that a transaction
that hasn't been validated cannot be mistaken for a valid one.

Change-Id: I5dbb18dd77af3cd14b168042ae660e4e27bf29dd
Signed-off-by: Alessandro Sorniotti <ale.linux@sopit.net>

This CR allows building of block storage indexes.
For rebuilding the indexes, existing index folder would need to be dropped.
However, please note that this would drop (and rebuild) the indexes for all
the chains because they share the underlying leveldb.

Also, enabled the flush/synch of batch writting to leveldb (statedb, block indexes, and historydb).

Change-Id: I6a926ab765df4bbb6543d6a3960359d95d60fd68
Signed-off-by: manish <manish.sethi@gmail.com>

This CR modifies the tranaction simulation, validation, and commit
code and delivers the end-to-end transaction flow that treats the
private data in a special manner. This CR mainly leverages the earlier
submitted independent CRs for sidedb feature for accomplishing this behavior.

This CR also allows ledger to receive the blocks and the pvt data from
another peer on the same channel (i.e., a peer catching up via state)

This CR is exceptionally large becasue of manily two reasons

1) The way currently the code (and specially the tests) is organized in
simulation/validation/commit flow, its not easy to submit such kind
of changes independently that cuase the change in the whole transaction
processing flow.

2) This CR causes a change in the existing ledger APIs which are used widely
across other packages (specially in the tests) and hence many files are included
for fixing the broken dependencies

Change-Id: Id29575176575f4c01793efd3476b68f8364cb592
Signed-off-by: manish <manish.sethi@gmail.com>

Change-Id: I6722961538f10ea0d8e705d35f84ca3ef72631f5
Signed-off-by: Christopher Ferris <chrisfer@us.ibm.com>

This CR updates all loggers throughout the code base to use
`flogging.MustGetLogger`. This function wraps `logging.MustGetLogger`
and tracks the logger modules defined in the system. This enables the
ability to set log levels for modules using regular expressions.
and make it easy to change the levels for any module and all its
submodules with a single command (e.g. gossip, ledger, msp).

Change-Id: If5d3229ea2312adb56fc21bfdafbed3d967cf1df
Signed-off-by: Will Lahti <wtlahti@us.ibm.com>

Change-Id: I9e33f6a177511f6cbb5a1ff5e20188008d2efae6
Signed-off-by: conghonglei <conghonglei@wanda.cn>

https://jira.hyperledger.org/browse/FAB-2810



This CR:
- Introduces proto messages
- Removes custom types for read-write set in package kvledger/txmgmt/rwset
- Renames the package kvledger/txmgmt/rwset to kvledger/txmgmt/rwsetutil
  and rwset.go to rwset_builder_test.go in order to reflect the purpose
  of the remaining code better
- Modifies the other packages to use rwset from protos
  and utilities from rwsetutil package

Change-Id: Id01be87903ad30f0eea34ef4033195b17bec3e71
Signed-off-by: manish <manish.sethi@gmail.com>

Invalid trans should not be written to history database.
This will prevent writes from invalid trans from
showing up in GetHistoryForKey() results.

Change-Id: I8508094d697febf213098712185b3d6e9aaf9f29
Signed-off-by: denyeart <enyeart@us.ibm.com>

Clean up peer logging for improved serviceability.

INFO level log had a lot of debug information, making it impossible
to find important INFO information and errors through the noise.
Changed most of the debug information to use DEBUG level.

After this change, during normal transaction processing there
will be a single entry in info log per block with messages
like this:

Channel [myc1]: Created block [1] with 43 transaction(s)
Channel [myc1]: Created block [2] with 14 transaction(s)
Channel [myc1]: Created block [3] with 26 transaction(s)

This will make it easy to spot any anomilies in the logs, while
still tracking block progress.

Confirmed there are good INFO messages for important events
such as peer initialization and channel creation.

Removed data content from debug messages for enhanced privacy.

Change-Id: I3a0b2f3a07d5c7dcf388a609d11cfa3bcf7bb065
Signed-off-by: denyeart <enyeart@us.ibm.com>

This change set ensures that the protobuf representation for headers be in
bytes. This makes sure that if ever protobuf marshalling is non-deterministic,
we do not have problems because whenever we hash a header, we do that over
the serialized version we receive.

Change-Id: I838e0d5dec2f79f88fab63d92bdfb51d92c2f069
Signed-off-by: Alessandro Sorniotti <ale.linux@sopit.net>

Made the following changes to fsblkstorage provider:

* Block numbers start at 0, not 1.

* Blockchain height should be last block number + 1.

* A block's Block.Header.Number should be equal to
  the Blockchain height when AddBlock() is called.

* Updated ledger tests to use random temporary dir.

* Update core endorser tests to generate valid block
  numbers.

Change-Id: I0208cd6a4b9da462c29d06126fa651f4c0dcd2fc
Signed-off-by: Luis Sanchez <sanchezl@us.ibm.com>

https://jira.hyperledger.org/browse/FAB-2033



The user facing documentation refers to channels, but the vast majority
of the fabric code refers to chains.  This CR switches the protos to
refer to channels as they are generally the interface between the user
and fabric code.

Change-Id: I869e6d68c1228e638a99e2460274889f18f2543d
Signed-off-by: Jason Yellick <jyellick@us.ibm.com>

Finish the implementation for ledger API GetHistoryForKey().

Lookup the history of key updates by blockNum:tranNum.

For each history record, retrieve the txID and key value
from block storage. This requires passing a block storage handle
into the history query executor, and exposing the block storage
test environment to other packages. Historic txID and key value
is returned to client to enable simple provenance scenarios.

Added tests for GetHistoryForKey() including recovery scenarios.
One test required a fix to block storage syncIndex().
Also tested end-to-end with peer.

Change-Id: I988130e9682f5d8d707c4ec37753bc0e7e297269
Signed-off-by: denyeart <enyeart@us.ibm.com>

https://jira.hyperledger.org/browse/FAB-2022



This changes introduced by this CR
- Moves the block storage code from package
core/ledger/blkstorage to common/ledger/blkstorage

- Splits the ledger_interface.go so as to move common interfaces
and data type to common/ledger package

- Moves some of the util functions to common/ledger package

- Moves core/ledger/ordererledger package to orderer/ledger/fsledger
orderer folks can futher rename/refactor this as seems suitable to them

Change-Id: I759e16f00dc2ec9bb62196121083cf48eae76948
Signed-off-by: manish <manish.sethi@gmail.com>

Getting a panic error when committer receives a config block
and attempts to write the tran to history db.
Fixed by only writing to history db for endorsement transactions.

Change-Id: I32d5e414199d85355eeae84f3e1ae680fe0811f0
Signed-off-by: denyeart <enyeart@us.ibm.com>

- Utilize leveldb for history of keys instead of couchdb

- Single leveldb database serves all chains to optimize
  footprint (same structure as leveldb block index database
  and leveldb state database)

- Block commit writes a history key for each key/value
  that is updated in a tran, in the form:
   ns~key~blocknum~trannum

- History database is an index for history of key writes
  over time.  The key values are not included to reduce
  size of database.

- GetHistoryForKey() finds all records starting with ~ns~key
  and returns the transactions that updated the key.

- Subsequent changeset will lookup and return the txid
  and historic value from the block storage.  Client can
  then GetTransactionById to see the historic transactions.

- Since history db size is much reduced, it is now enabled
  by default in core.yaml.

- Upon crash recovery, ledger initialization will ensure
  that both state db and history db in sync with block storage

Reused existing test logic, therefore the changeset
is relatively large to ensure tests still pass.

Change-Id: I79103aa39957f58d246de5b5295fb40a4b9c033b
Signed-off-by: denyeart <enyeart@us.ibm.com>