1. 30 Oct, 2017 1 commit
    • manish's avatar
      [FAB-6779] Allow rebuilding block storage indexes · 1daabff7
      manish authored
      
      
      This CR allows building of block storage indexes.
      For rebuilding the indexes, existing index folder would need to be dropped.
      However, please note that this would drop (and rebuild) the indexes for all
      the chains because they share the underlying leveldb.
      
      Also, enabled the flush/synch of batch writting to leveldb (statedb, block indexes, and historydb).
      
      Change-Id: I6a926ab765df4bbb6543d6a3960359d95d60fd68
      Signed-off-by: default avatarmanish <manish.sethi@gmail.com>
      1daabff7
  2. 29 Aug, 2017 1 commit
    • senthil's avatar
      [FAB-5867] Make statedb validator to use bulkread · d97326a5
      senthil authored
      
      
      This CR makes the statedb validator to use bulk
      read optimization APIs such as LoadCommittedVersions() and
      GetVersion() to reduces the number of REST API calls between
      peer and CouchDB to improve the performance.
      
      Statedb validator first retrieves read set (including hashed reads) of all
      transactions in the block (which is to be validated), then preloads the
      committed version number of each key in the read sets into a cache (using
      LoadCommittedVersion()) from the statedb, and finally compares
      version of each key in read set against committed version (which is
      retrieved from cache using GetVersion()) while validating a transaction.
      
      Existing unit-test for validator is adequate to test this CR.
      
      Change-Id: I96cc7fec2d7fcd07c7cffc5cb0aa07635e036164
      Signed-off-by: default avatarsenthil <cendhu@gmail.com>
      d97326a5
  3. 21 Aug, 2017 1 commit
  4. 06 Aug, 2017 1 commit
    • manish's avatar
      [FAB-4977] sidedb:statedb enhancements · 2dec57d8
      manish authored
      
      
      This CR provides support for maintaining public, private, and hashed data
      This includes
         - An interface for managing the three categories of data
      
         - A default implementation that allows the use of either
           leveldb or couchdb. The default implementation uses a
           single logical db and uses different namespaces for
           different categories of data. Alternate implementation
           based on furture need or exploration should be easy to support.
           These may include using different dbs for different categories
           such as using leveldb for hashed data and using separate dbs in
           a couch instance for public and private data
      
         - If the underlying db does not support storing random bytes as key
           (for example couch supports onlu valid utf-8 bytes as key), the
           key for the hashed data is encoded using base64
      
      Change-Id: Ia8ede4f4c0ab392119e59bb7f46e9c20062a411a
      Signed-off-by: default avatarmanish <manish.sethi@gmail.com>
      2dec57d8
  5. 03 Aug, 2017 1 commit
    • Chris Elder's avatar
      [FAB-2960] Transaction Mgr changes batch optimizations · 7d90d014
      Chris Elder authored
      
      
      This is change is part of FAB-2725 CouchDB optimizations
      
      Interactions with CouchDB are currently done individually.
      Need to switch to using bulk operations to get optimal performance
      from CouchDB. Need to performance test and stress test.
      
      This change implements all changes needed by the commit phase to use
      bulk operations in CouchDB.
      
      A subsequent change will implement validation bulk read and caching
      of versions and revisions for CouchDB.
      
      Subsequent changes will also be added to retry errors encountered by
      the bulk update operation and a configuration setting to limit the
      size of the bulk update by number of update documents.
      
      - Add new interface to statedb for bulk operations, the new operations
      will only be implemented for databases that can support bulk operations
      (only CouchDB)
      
      type BulkOptimizable interface {
      	LoadCommittedVersions(keys []*CompositeKey)
      	ClearCachedVersions()
      }
      
      LoadCommittedVersions will bulk load version and revision numbers into
      cache based on the document ID.  Documents missing a version or revision
      will be added to the cache with a nil value.
      
      - Add GetVersion method to VersionedDB to enhance batch
      validation of batches
      
      GetVersion(namespace string, key string) (*version.Height, error)
      
      - Implement the GetVersion method for CouchDB and goleveldb.  The goleveldb
      implementation will be similar to GetState, except it will return the
      version.  CouchDB implementation will attempt a cache lookup, then fall
      back to a database read if not found.
      
      - Enhance the ApplyUpdates method for CouchDB to incorporate the
      ability to retrieve all revision information required for bulk
      updates and deletes for CouchDB.
      
      Change-Id: I65f8525a7a13b1ad7b49fda0e357b9a406fba79d
      Signed-off-by: default avatarChris Elder <chris.elder@us.ibm.com>
      7d90d014
  6. 08 Jun, 2017 1 commit
    • manish's avatar
      [FAB-3556] Throw error for invalid keys at simulation · 19c7670f
      manish authored
      
      
      This CR makes changes in the simulation such that the simulator
      raises an error if an invalid key is passed in. The validity of the
      key is decided based on the configured db. For instance, couchdb allows
      only valid utf-8 strings to be the key.
      
      In the current code, this check is not made during simulation. This causes
      to defer the issue at the commit time in which situation, peer has no option
      other than raising a panic becasue, it simply cannot commit the block.
      
      Change-Id: Id1439c36e3a142f8ca47574d391ee10b366acd69
      Signed-off-by: default avatarmanish <manish.sethi@gmail.com>
      19c7670f
  7. 22 Apr, 2017 1 commit
    • Will Lahti's avatar
      [FAB-2351] Update loggers to flogging.MustGetLogger · 7132dd54
      Will Lahti authored
      
      
      This CR updates all loggers throughout the code base to use
      `flogging.MustGetLogger`. This function wraps `logging.MustGetLogger`
      and tracks the logger modules defined in the system. This enables the
      ability to set log levels for modules using regular expressions.
      and make it easy to change the levels for any module and all its
      submodules with a single command (e.g. gossip, ledger, msp).
      
      Change-Id: If5d3229ea2312adb56fc21bfdafbed3d967cf1df
      Signed-off-by: default avatarWill Lahti <wtlahti@us.ibm.com>
      7132dd54
  8. 27 Feb, 2017 1 commit
    • denyeart's avatar
      Clean up peer logging - serviceability · b39b8a8f
      denyeart authored
      
      
      Clean up peer logging for improved serviceability.
      
      INFO level log had a lot of debug information, making it impossible
      to find important INFO information and errors through the noise.
      Changed most of the debug information to use DEBUG level.
      
      After this change, during normal transaction processing there
      will be a single entry in info log per block with messages
      like this:
      
      Channel [myc1]: Created block [1] with 43 transaction(s)
      Channel [myc1]: Created block [2] with 14 transaction(s)
      Channel [myc1]: Created block [3] with 26 transaction(s)
      
      This will make it easy to spot any anomilies in the logs, while
      still tracking block progress.
      
      Confirmed there are good INFO messages for important events
      such as peer initialization and channel creation.
      
      Removed data content from debug messages for enhanced privacy.
      
      Change-Id: I3a0b2f3a07d5c7dcf388a609d11cfa3bcf7bb065
      Signed-off-by: default avatardenyeart <enyeart@us.ibm.com>
      b39b8a8f
  9. 13 Feb, 2017 1 commit
    • Chris Elder's avatar
      FAB-1985 Scope rich queries to chaincode context · 9da35a27
      Chris Elder authored
      
      
      Motivation for this change:
      Need to inject chaincodeid filter on rich queries called from chaincode.
      Need to add a parameter to QSCC GetQueryResult() to scope queries
      to a certain chaincode.
      
      - This change will add the interface and method signature changes
      required to add the chaincodeid to the execute query methods.
      
      Change-Id: Ia57001d6861862876d5df621a94e2c4b5e288d1d
      Signed-off-by: default avatarChris Elder <chris.elder@us.ibm.com>
      9da35a27
  10. 03 Feb, 2017 1 commit
    • manish's avatar
      Move Blockstorage code under /fabric/common package · 2a16532c
      manish authored
      https://jira.hyperledger.org/browse/FAB-2022
      
      
      
      This changes introduced by this CR
      - Moves the block storage code from package
      core/ledger/blkstorage to common/ledger/blkstorage
      
      - Splits the ledger_interface.go so as to move common interfaces
      and data type to common/ledger package
      
      - Moves some of the util functions to common/ledger package
      
      - Moves core/ledger/ordererledger package to orderer/ledger/fsledger
      orderer folks can futher rename/refactor this as seems suitable to them
      
      Change-Id: I759e16f00dc2ec9bb62196121083cf48eae76948
      Signed-off-by: default avatarmanish <manish.sethi@gmail.com>
      2a16532c
  11. 30 Jan, 2017 1 commit
    • denyeart's avatar
      [FAB-1933] ExecuteQuery on leveldb panic · 6255f8d3
      denyeart authored
      
      
      ExecuteQuery() is not supported on leveldb state
      database.  Currently if called it will panic the peer.
      This changeset changes it to a normal error
      so that it does not kill the peer process.
      
      Change-Id: Ib9f525524c99b107b4fc3935f2d38a5657bace73
      Signed-off-by: default avatardenyeart <enyeart@us.ibm.com>
      6255f8d3
  12. 27 Jan, 2017 1 commit
    • Chris Elder's avatar
      FAB-1818 Create data wrapper for state data in CouchDB · ead6705d
      Chris Elder authored
      
      
      Motivation for this change:
      Maintain the version inline with the document, without changing the
      structure/value of the stored document.  This will prevent unexpected
      data values being returned and will prevent possible name collisions
      with document values.
      
      - Create a data wrapper for ledger JSON data stored in CouchDB.
      
      - Return the version based on block number and transaction id
      
      The wrapper will be implemented as a new key named "data" which will
      contain the JSON data for the state database.
      
      Prior to change example:
      
      "doc": {
       "_id": "2",
       "_rev": "2-8ee0c31b21ad650e5b872c0b98e59ab5",
       "version":"1:2"
       "asset_name": "marble2",
       "color": "red",
       "owner": "tom",
       "size": "25"
      }
      
       Following change:
      
       "doc": {
        "_id": "2",
        "_rev": "2-8ee0c31b21ad650e5b872c0b98e59ab5",
        "version":"1:2"
        "data": {
         "asset_name": "marble2",
         "color": "red",
         "owner": "tom",
         "size": "25"
        }
       }
      
      Change-Id: I59391ea926531c46c346fc8448e3d041ca5f3fdf
      Signed-off-by: default avatarChris Elder <chris.elder@us.ibm.com>
      ead6705d
  13. 26 Jan, 2017 1 commit
    • denyeart's avatar
      [FAB-1762] Refactor couchdb history db to leveldb. · 2c982629
      denyeart authored
      
      
      - Utilize leveldb for history of keys instead of couchdb
      
      - Single leveldb database serves all chains to optimize
        footprint (same structure as leveldb block index database
        and leveldb state database)
      
      - Block commit writes a history key for each key/value
        that is updated in a tran, in the form:
         ns~key~blocknum~trannum
      
      - History database is an index for history of key writes
        over time.  The key values are not included to reduce
        size of database.
      
      - GetHistoryForKey() finds all records starting with ~ns~key
        and returns the transactions that updated the key.
      
      - Subsequent changeset will lookup and return the txid
        and historic value from the block storage.  Client can
        then GetTransactionById to see the historic transactions.
      
      - Since history db size is much reduced, it is now enabled
        by default in core.yaml.
      
      - Upon crash recovery, ledger initialization will ensure
        that both state db and history db in sync with block storage
      
      Reused existing test logic, therefore the changeset
      is relatively large to ensure tests still pass.
      
      Change-Id: I79103aa39957f58d246de5b5295fb40a4b9c033b
      Signed-off-by: default avatardenyeart <enyeart@us.ibm.com>
      2c982629
  14. 24 Jan, 2017 1 commit
    • manish's avatar
      Detect phantom items during validation · 16372176
      manish authored
      https://jira.hyperledger.org/browse/FAB-1668
      
      
      
      Phantom reads problem:
      -----------------------------
      If a transaction executes a key range query (say key_1 - key_n) leveldb
      serves all the keys in the given range in sorted order of keys.
      Between simulation and validation of a transaction, some other transaction
      may insert one or more keys in this range. Now, this change cannot
      be detected by merely checking the versions of the keys present in
      the read-set of the transaction.
      (though, this check can detect updates and deletes).
      
      A serializable isolation level does not permit phantom reads.
      
      Situations where the phantom reads may not be acceptable:
      -----------------------------
       - A chaincode may take a decision based on aggregates of the results of
      the range query such as min/max/avg/sum etc. This would be affected
      if a new entry gets added to the range query results.
      
       - A chaincode may want to change all the marbles
      owned by a specific user to blue in a tran `t1`. The chaincode may do so
      by performing a range query to get all the marbles for the user and
      change their color to blue. Now, if a new white marble get added to
      the user's assets by an another transaction `t2`,
      `t1` should be marked as invalid.
      
      Solution in this change-set:
      -----------------------------
      For solving this, we can capture the details of the range query
      (the query and it's results) during the simulation time and
      re-execute the query during validation time and compare the results.
      If the results (keys and versions) are still the same,
      we can safely commit the transaction.
      
      This changes set introduces
       - Structures for capturing the range query details in rwset
       - Detection of phantom items (insertion/deletion)
         that affect the results of the range query during validation
       - Mark the transactions that are affected by the phantom items
         as invalid to ensure serializability in the presence of range queries
      
      Change-Id: I909841a0234c37795ad7a4ffcca8a9ebd9c9f994
      Signed-off-by: default avatarmanish <manish.sethi@gmail.com>
      16372176
  15. 17 Jan, 2017 1 commit
    • manish's avatar
      SingleLevelDB for block index · 8cdd0f4d
      manish authored
      https://jira.hyperledger.org/browse/FAB-1664
      
      
      
      This changeset:
      - Renames package ledger/util/db to ledger/util/leveldbhelper
      - Implements a leveldb provider
        (that enables using same leveldb instance as a multiple logical dbs)
        in util package for being able to reuse across statedb, index,
        and later for historydb
      - Implements a provider as a single point of invocation
        for managing multiple block storage
      - Uses a single leveldb instance for block storage index
      - Makes the structures other than providers as private
        to their respective packages
      
      Change-Id: I5f0b3b9aa8ef3ac1ccdce4f3c6fa6d842b5318c1
      Signed-off-by: default avatarmanish <manish.sethi@gmail.com>
      8cdd0f4d
  16. 16 Jan, 2017 1 commit
  17. 15 Jan, 2017 1 commit
    • denyeart's avatar
      FAB-1505 and FAB-1337 Refactor CouchDB code · 9df7b0ea
      denyeart authored
      
      
      FAB-1505 refactors CouchDB code to be aligned with LevelDB structure.
      FAB-1337 applies all ledger tests written for LevelDB against CouchDB.
      
      In order to test the CouchDB refactor,
      had to implement FAB-1337 in same changeset to test.
      
      Like most refactors, this changeset touches a lot of files and code,
      since it is not feasible to do a partial refactor.
      
      Tests related to versioning and deleting have been commented out,
      these will be re-enabled in subsequent changeset when CouchDB
      has the support, in order to keep this changeset size more reasonable.
      
      Change-Id: I0d2d6cca89bd0252f6e84317457a9140b3a0d7a5
      Signed-off-by: default avatardenyeart <enyeart@us.ibm.com>
      9df7b0ea
  18. 08 Jan, 2017 1 commit
  19. 14 Dec, 2016 1 commit