Part 3 of ledger servicebility improvement:
Update common/ledger to use error handling framework

Change-Id: I86383b404941c5f922fa2f26ca71af4d91351d4c
Signed-off-by: Wenjian Qiao <wenjianq@gmail.com>

This CR fixes the current behavior of block store index.
In the current implemetation, the indexes that maintain
different pieces of information by txid are overwritten
by a duplicate txid (if any). This CR reverses this behavior
and keeps the first appearance of txid in the index. The future
duplicate txids will not overwrite it.

Though, either of these behaviors (keeping the first tx or
keeping the last tx) are not fully justified, primarily because
the problem itself is a paradox - in the sense that the ids
(that are supposed to be unique by definition) are duplicated.
However, the justification of moving from the current behavior
to the proposed behavior is that, its easy for someone to replay
the transaction or simply use an existing txid in a bogus transaction
just to exploit the current behavior of overwriting the block storage
index and preventing the legitimate user to query about the status
of their transactions.

Change-Id: I3b81ae61c756ef78253b58a94644778716fb0e16
Signed-off-by: manish <manish.sethi@gmail.com>

This CR allows building of block storage indexes.
For rebuilding the indexes, existing index folder would need to be dropped.
However, please note that this would drop (and rebuild) the indexes for all
the chains because they share the underlying leveldb.

Also, enabled the flush/synch of batch writting to leveldb (statedb, block indexes, and historydb).

Change-Id: I6a926ab765df4bbb6543d6a3960359d95d60fd68
Signed-off-by: manish <manish.sethi@gmail.com>

This CR
   - Adds unit tests to enhance the code coverage of the package
   fabric/common/ledger/blkstorage/fsblkstorage
   - Removes an unused function
   - fixes a few comments/debug message

Change-Id: Iab0cbdc6fe8295aa73a0f8a08390ba0f83f24204
Signed-off-by: manish <manish.sethi@gmail.com>

Change-Id: I9e33f6a177511f6cbb5a1ff5e20188008d2efae6
Signed-off-by: conghonglei <conghonglei@wanda.cn>

https://jira.hyperledger.org/browse/FAB-2243



Switch valid flag from a boolean to an enum, so that we can indicate the reason
for the invalidation (txid already exists, vscc validation failure
(endorsement policy failure), mvcc validation failure, etc)

This change adds invalidation reason codes, sets them in the block as a uint8
(byte) array, and returns validation result in GetTransactionById.

Change-Id: I3593bf6aa4f35bb01b14f8fb3d9950b405f84660
Signed-off-by: Ruslan Kryukov <ruslan.kryukov@ru.ibm.com>

Made the following changes to fsblkstorage provider:

* Block numbers start at 0, not 1.

* Blockchain height should be last block number + 1.

* A block's Block.Header.Number should be equal to
  the Blockchain height when AddBlock() is called.

* Updated ledger tests to use random temporary dir.

* Update core endorser tests to generate valid block
  numbers.

Change-Id: I0208cd6a4b9da462c29d06126fa651f4c0dcd2fc
Signed-off-by: Luis Sanchez <sanchezl@us.ibm.com>

https://jira.hyperledger.org/browse/FAB-2086



We need this method for getting information
of block in a ledger by txID.
Also this method helps to check valid tx or not
by getting the validation bit array
from the block.

Change-Id: I4216635097ecd4cf5de9e3442dfbd5660d44292f
Signed-off-by: Ruslan Kryukov <ruslan.kryukov@ru.ibm.com>

Finish the implementation for ledger API GetHistoryForKey().

Lookup the history of key updates by blockNum:tranNum.

For each history record, retrieve the txID and key value
from block storage. This requires passing a block storage handle
into the history query executor, and exposing the block storage
test environment to other packages. Historic txID and key value
is returned to client to enable simple provenance scenarios.

Added tests for GetHistoryForKey() including recovery scenarios.
One test required a fix to block storage syncIndex().
Also tested end-to-end with peer.

Change-Id: I988130e9682f5d8d707c4ec37753bc0e7e297269
Signed-off-by: denyeart <enyeart@us.ibm.com>

History database returns the list of (blockNum,tranNum) that
updated a key.
RetrieveTxByBlockNumTranNum will allow kvledger
to retrieve transaction history for this list of key updates.

Change-Id: I9ff3ed6c273c57b65223b283c4f602910e5982bf
Signed-off-by: denyeart <enyeart@us.ibm.com>

https://jira.hyperledger.org/browse/FAB-2022



This changes introduced by this CR
- Moves the block storage code from package
core/ledger/blkstorage to common/ledger/blkstorage

- Splits the ledger_interface.go so as to move common interfaces
and data type to common/ledger package

- Moves some of the util functions to common/ledger package

- Moves core/ledger/ordererledger package to orderer/ledger/fsledger
orderer folks can futher rename/refactor this as seems suitable to them

Change-Id: I759e16f00dc2ec9bb62196121083cf48eae76948
Signed-off-by: manish <manish.sethi@gmail.com>

https://jira.hyperledger.org/browse/FAB-1686



Change-Id: I0b2d784f52f4f38f6afa03ca750188c369b2cc62
Signed-off-by: grapebaba <281165273@qq.com>

https://jira.hyperledger.org/browse/FAB-1664



This changeset:
- Renames package ledger/util/db to ledger/util/leveldbhelper
- Implements a leveldb provider
  (that enables using same leveldb instance as a multiple logical dbs)
  in util package for being able to reuse across statedb, index,
  and later for historydb
- Implements a provider as a single point of invocation
  for managing multiple block storage
- Uses a single leveldb instance for block storage index
- Makes the structures other than providers as private
  to their respective packages

Change-Id: I5f0b3b9aa8ef3ac1ccdce4f3c6fa6d842b5318c1
Signed-off-by: manish <manish.sethi@gmail.com>

Add a new ledger blockstorage index for History
that will map (blocknum,trannum) to the file storage
location for this block transaction

This index will be used for the API  GetTransactionsForKey()
for (chaincode1,key1).  It will do a key range query on chaincode1~key1
to pick up all chaincode1~key1 records.  Results will indicate
the set of (blocknum,trannum) transactions that updated this key.

Change-Id: I81da09e5526d7e2966634c78a03d34011d514442
Signed-off-by: Mari Wade <mariwade@us.ibm.com>

https://jira.hyperledger.org/browse/FAB-1201



Change-Id: I34f7f56887c25dae712188a50e8bdcd13ae619d2
Signed-off-by: manish <manish.sethi@gmail.com>

https://jira.hyperledger.org/browse/FAB-788
RocksDB has a patent infringement license in it from Facebook
(https://github.com/facebook/rocksdb/blob/master/PATENTS)
Many users may not be comfortable with the term of the license.

The alternatives include
1. LevelDB (https://github.com/google/leveldb) with a go wrapper (https://github.com/jmhodges/levigo),
2. goleveldb (https://github.com/syndtr/goleveldb) - a porting of leveldb in golang
3. BoltDB (https://github.com/boltdb/bolt)

BoltDB is suitable for read heavy workloads (e.g., LDAP)
but has a relatively poor performance for read-write workloads.
Of the other two options, goleveldb is chosen because it is
implemented in golang and hence easy to intergate and maintain.
In addition, as a precedent, ethereum go implementation also
uses this package
https://github.com/ethereum/go-ethereum/blob/master/ethdb/database.go



Change-Id: Ia4fb5a6f9299e613d03d8b414a51bf479bfafd59
Signed-off-by: manish <manish.sethi@gmail.com>

Raw ledger provides basic functionality for storing and
retrieving blocks. This is intended to be used by an orderer service
https://jira.hyperledger.org/browse/FAB-56



Change-Id: I3fb733f5be53b6f630c20554ba4e362540b8f55a
Signed-off-by: manish <manish.sethi@gmail.com>

FAB-853

Ledger is removed from
   . core/peer/peer.go
   . core/rest/api.go

Ledger package itself is removed. "ledgernext" is
replaced with "ledger".

Change-Id: Ie7dfbd9bf94afa0031aef759fc46929e4fb3b400
Signed-off-by: Srinivasan Muralidharan <muralisr@us.ibm.com>

This commit adds the functionality of checkpointing block index progress
and sync-ing (updating) the index during start of the block storage system

Change-Id: Ib1a325add455bce47e510ccfc7af052db51117e6
Signed-off-by: manish <manish.sethi@gmail.com>

(Rocks) DB WAL adds overheads while using the DB for saving checkpoints
for block storage. Avoiding writing to WAL translates the write workload
(appending blocks to the blockfile) into a sequential disk writes.

This commit changes the way checkpoints are saved - checkpoints are
written to DB as before, however since WAL is disabled, the checkpoint
stays in-memory only. The in-memory checkpoint is flushed explicitly to
disk (via DB flush) at the time of new block file creation. The in-memory
checkpoint would implicitly also be flushed to memory at the time of DB
shutdown. However, if a crash takes place, the checkpoint available in the
DB would be behind the actual block file status. In order to handle this
crash scenario, this commit also adds code to perform a scan of the block
file beyond the last saved checkpoint and update the checkpoint.

Change-Id: Ie88646b225abaa50b595833f5e7ed8d4facae920
Signed-off-by: manish <manish.sethi@gmail.com>

This includes structure of the new code conforming to new architecture
and a basic working implementation.

Change-Id: I885e122c0001ff82301cd1448a7ddad80db931b0
Signed-off-by: manish <manish.sethi@gmail.com>