Skip to content
  • Chris Elder's avatar
    [FAB-9840] CouchDB safe pagination - statecouchdb · ca152f26
    Chris Elder authored
    
    
    Currently, CouchDB queries are limited to configurable max number of
    results/docs, based on peer queryLimit config option (default 10000).
    The shim/peer supports pagination but CouchDB query iterators do not.
    
    This change will allow additional controls to prevent a large accidental or
    malicious query from causing performance issues with CouchDB.
    
    Changes to core.yaml:
    
    Changed queryLimit to 1000.  This is part of repurposing this parameter.
    QueryLimit is now to be used as a per query limit to CouchDB instead of the
    limit for the query from the shim.
    
    Added totalLimit parameter.  This parameter will be used to cap the maximum
    number of records returned by the query from the shim.
    
    Changes to couchdb:
    
    Added changes to range query to find the next start key (if exists) for the
    specified range.  The next start key is returned with the query results.
    
    Added changes to rich query to find the next bookmark for the query.  The
    bookmark is returned with the query results.
    
    Changes to statedb interface:
    
    Add 2 new methods to allow parameters to be passed to the range and rich
    query methods.
    
    GetStateRangeScanIteratorWithMetadata(namespace string, startKey string,
    endKey string, metadata map[string]interface{})
    (ResultsIterator, error)
    
    ExecuteQueryWithMetadata(namespace, query string, metadata map[string]interface{})
    (ResultsIterator, error)
    
    Changes to statecouchdb (range query):
    
    Add concept of  limit as a query parameter.  If limit is set, then queries
    are executed and returned using limit.  A nextStartKey is also
    returned which will allow the nextStartKey to passed as a parameter to the next
    query.  This implements an "explicit" paging.
    
    If no limit is specified, then the query scanner will use the nextStartKey
    internally to page through the results, using the queryLimit to control the
    query size to CouchDB. This implements an "implicit" paging.
    
    Changes to statecouchdb (rich query):
    
    Add concept of  limit as a query parameter.  If limit is set, then queries
    are executed and returned using limit to control size.  A bookmark is also
    returned which will allow the bookmark to passed as a parameter to the next
    query.  This implements an "explicit" paging.
    
    If no  limit is specified, then the query scanner will use the bookmark
    internally to page through the results, using the queryLimit to control the
    query size to CouchDB.  This implements an "implicit" paging.
    
    Change-Id: I2b2e70b3302120291bd2243d90e6c6402b70e399
    Signed-off-by: default avatarChris Elder <chris.elder@us.ibm.com>
    ca152f26