When the collection config specifies max peers, it might be 0.

Currently, gossip assumes that if the max peers is 0 then it was
omitted, and just sets the max peers to all the known peers.

Since collection membership can change in v1.2, we should be
conservative and enforce 0 max peers as 0.

Change-Id: Idef84b6d5d88a719c737aef05b65a724a3a38468
Signed-off-by: yacovm <yacovm@il.ibm.com>

Change-Id: Ifeb2e5b8350af506411b23d42e891d6f91478989
Signed-off-by: nirro <nirro@il.ibm.com>

The discovery service needs to output identities of known peers.

Gossip stores the identities in an identity mapper,
which could be extended to return a snapshot of its identities.

This change set makes it so that the identity mapper also saves
the organizations of peers upon their insertion, and adds
the needed method.

Change-Id: I94b032781bae5d07a3d9fef586273ac8bfcc2e4d
Signed-off-by: yacovm <yacovm@il.ibm.com>

Update to grpc-go 1.10.0

We were several releases behind and there were
changes in behavior and some APIs which required
tweaking production and test code

Change-Id: I86bc6cd170b5a07777c11db485bb86fdc73e495a
Signed-off-by: Gari Singh <gari.r.singh@gmail.com>

This change set adjusts gossip UT port ranges so that
different packages could run in parallel.

Change-Id: I514623bc1291134cfb11e9f6a1f2439883aa409f
Signed-off-by: yacovm <yacovm@il.ibm.com>

The stateInfo.Metadata field was used in v1.0 for storing the peer's
ledger height.

In v1.1 we added a properties proto structure field that would replace
the metadata for ledger height.
In v1.2 (next release) we deprecate the metadata and as a result, peers
in v1.0 won't be able to gossip in channel context with peers
from v1.2

Change-Id: Ieefa98563500760eb23bf579f93ed786b935acf4
Signed-off-by: yacovm <yacovm@il.ibm.com>

There are few golint warning on gossip module packages, this commit
takes care to address them.

Change-Id: I13a85ad990f981a0da42446962ac0993383cc8b5
Signed-off-by: Artem Barger <bartem@il.ibm.com>

Currently while gossip forward the message during distribution it
selects peers from the current membership view, while not filtering
the peer which delivered the message. As a consequence message might be
send to the source peer one again. This commit extends the peers
selection logic by adding the check which allows to avoid sending same
message to the source peer back again.

Change-Id: I76ceafba2335c27e1186f511d3cdab8a24d600af
Signed-off-by: Artem Barger <bartem@il.ibm.com>

This commit adds a left channel flag to the stateInfo message's properties
so that peers would ignore these peers.

Change-Id: I55e281bf1bd933223da0c8ab253584901e22f6e9
Signed-off-by: yacovm <yacovm@il.ibm.com>

This commit adds a Properties sub-message to the StateInfo message
and makes sure the gossip logic can handle peers with and without
this field set.

Change-Id: Ibe164db1e4dc4c5a1d861e62eb61da01d31778e6
Signed-off-by: yacovm <yacovm@il.ibm.com>

A prior change made fabric compatible
with G0 1.9.  This CR now builds fabric
with Go 1.9 by default by updating the
baseimage version to 0.4.1 which uses
Go 1.9.  Also updated any docs to
indicate Go 1.9 should now be used.

- Fixed linter error with sanitycheck.go
- added check for testing.tRunner for gossip
TestEndedGoroutines.  Not sure why I missed
this earlier

Change-Id: I213cc2f88363c32ccff246b5655afd88b42a7778
Signed-off-by: Gari Singh <gari.r.singh@gmail.com>

This commit adds to the gossip API an ability to send to a set of peers
a message according to some criteria, like:
- Minimum acknowledgement from the peers
- Timeout to wait
- Channel
- Maximum amount of peers to send to
- A predicate, which can be derived from MSP policies

Change-Id: I69ee27ead3532558fa0213594ece71ec54dc5a07
Signed-off-by: yacovm <yacovm@il.ibm.com>

This commit:
1) Makes gossip's identity package use the new method
   Expiration() that was added to the MessageCryptoService
   in the previous commit.
   Now, when an identity is added to the identity mapper -
   a timer is set and the identity is automatically purged
   after the timer expires.
   The timer is set to the time left until the identity
   is expired.
2) Adds some tests to the identity package, and makes its
   code-coverage reach 100%.
3) Moves the logic that periodically purges un-used identities
   from the gossip package to the identity package, to improve
   code cohesion.

Change-Id: I684197ff0f20031e80e9e9341042a5593eaca07a
Signed-off-by: yacovm <yacovm@il.ibm.com>

This commit adds a new method (and its implementation) to
the MessageCryptoService:
	Expiration(peerIdentity PeerIdentityType) (time.Time, error)

In the next commit I would adjust gossip to make use of this method

Change-Id: Idbdf1ba3b238a122e9d2db0b84016eb7cc6490a7
Signed-off-by: yacovm <yacovm@il.ibm.com>

This commit:
1) Moves the identity mapper into the gossip package from the service
   package, since it's not needed in the service package.
2) Extends the constructor of the identity mapper to have a callback
   that is fired whenever an identity is deleted from the identity mapper.

Change-Id: I1c9eaa47c97351518848843a9c4f2e1835f82249
Signed-off-by: yacovm <yacovm@il.ibm.com>

Moving code that store peer state info in msg store to be executed after
state info metadate validated

Fixed all tests in gossip/gossip that update channel metadata to include
correct metadata.

Change-Id: I4cd1a0ebbf8b8a38e61b3e45b3c1d96e2f283218
Signed-off-by: Gennady Laventman <gennady@il.ibm.com>

In the gossip tests there is a waitgroup counter that is
used for the last test (that checks that goroutines ended)
to be able to run only when all the other tests have finished.

Currently the waitgroup counter is initialized to the number of
tests but it should be initialized in a more dynamic way that
shows which tests are disabled and which tests are not.

Change-Id: I491d7d3676aeb79fbe2a30fadc63b27a37f003c9
Signed-off-by: yacovm <yacovm@il.ibm.com>

This patchset helps clean up the code and docs:

* Fix redundant parenthesese.
* Fix mismatching params in func return.
* Fix variable name collides with imported package name and builtin
  function or reserved words.
* Fix word typos.

Change-Id: I30304c233067ead7e74d18e3caf2604b3ed1490a
Signed-off-by: Baohua Yang <yangbaohua@gmail.com>

This commit adjusts gossip files to have
the new SPDX license as required

Change-Id: Ia77ce6f1267a236e0df4b0b80450764534252a8a
Signed-off-by: yacovm <yacovm@il.ibm.com>

This fixes a possible panic in the test, that is
caused by a missing length check

Change-Id: I3afd31aa4ace3ab2237b192ec9ee4090bca77d12
Signed-off-by: Yacov Manevich <yacovm@il.ibm.com>

Currently, the gossip code panics if it's unable to sign a message
or to marshal it bytes, because the assumption was
that if the signing failed then something is terrible wrong with the
peer and it's not recoverable
If turning a message into bytes fails, it's because of something
not recoverable too.

However, a user posted a stack trace in which gossip tried constructing
a message that its envelope has a payload of 2GB.

Now, the cause for a creation of such a message was fixed but in theory
we may somehow have a situation where a large message is constructed.

In order to prevent these kind of problems, and the crash of the peer,
we need to make the Signer return an error instead of panicking.

Change-Id: Ife8f4e137c92c167dadc7bbe1bcdf45f93b2b38e
Signed-off-by: Yacov Manevich <yacovm@il.ibm.com>

There have been many updates to the grpc-go package
since it was vendored in v0.6 and then v1.0.0.

There are several defects / issues related to
inactivity timeouts, connectivity, etc which can
be more easily fixed using new features such as
the client keepalive.

This CR updates grpc-go as well as all of its
dependencies and makes minor updates to the
code where required as well as recompiles all of
the protos using the protobuf compiler plugin
required by this version of grpc-go.

Note that for the updated vendoring, I chose
to delete the old packages and add the
updated ones as govendor is not perfect
when it comes to trying to update packages.

The only "new" code was adding custom server
transport credentials in the comm package as
the default provided by the grpc/credentials
package was changed and breaks the functionality
required for dynamically updating root certs for
running grpc servers and new package config
values for send/receive msg sizes as grpc-go
now enforces them.  Changes were also made
to any grpc clients and servers used in the
code to leverage the new config values.

There is one TODO which is to actually set
the max send/receive msg sizes based on
config.  This will need to be done for
both the peer and the orderer.

Also changed the license header to the new
format for any new or modified files

Change-Id: I41687a54aa21802203b04a339952881349042b70
Signed-off-by: Gari Singh <gari.r.singh@gmail.com>

In gossip we have time-based expiration (purge) of identities that haven't
been used for a long time.
This may cause an expiration of the peer's own certificate if it doesn't
gossip with any peers (i.e if it's alone in the world).

The problem is that when an identity expires from the identity store,
it is also expired from the pull mediator that is used for identities.
This is important because the only way identities are gossiped
transitively is via the pull mechanism.
If a peer's own identity disappears from the pull mediator,
it will never be sent to peers transitively.

Change-Id: Ic2138ebaa60bdf2454d65d3884f141c3736254a0
Signed-off-by: Yacov Manevich <yacovm@il.ibm.com>

The current implementation of what happens when the peer connects
to an anchor peer has a flawed logic.

It assumes that if the probe is successful, the sending is also.
That's too optimistic.
Another problem is that after the handshake is performed, the remote peer
may not distinguish between the closing of the connection of the handshake
and the closing of the new connection that sends the MembershipRequest,
since we only support 1 connection between each peer (to prevent DOS).

I re-implemented the logic to do the following:
1) First, perform a handshake to know whether the remote peer is from
  our org or not.
  Peers in different orgs should not know of our internal endpoints).
  The handshake is performed until it succeeds or too many attempts
  are made.
2) Now, construct a MembershipRequest and send it until you get
   acked from the remote peer, or when attempts are exhausted.

Added a test that simulates the flow.

Change-Id: I72370d9be816105fdc5af7577ce578faf6e5abdc
Signed-off-by: Yacov Manevich <yacovm@il.ibm.com>

While investgating the issue of why tests fails, disabling it
to prevent CI from failing.

Change-Id: I82808581b89adac6021166adab78251cdcdcd226
Signed-off-by: Artem Barger <bartem@il.ibm.com>

This test fails from time to time, and I am going to
restructure the logic that this test tests as part of
https://jira.hyperledger.org/browse/FAB-3109

 so most
likely would also need to change the test itself.

Disabling the test in the meantime,
and it will be introduced later when it will be more stable.

Change-Id: I8b20a308a7ef60b11f9e7965f8b6ee7380fcf1e8
Signed-off-by: Yacov Manevich <yacovm@il.ibm.com>

* Fix the field from inc_number to inc_num, which is aligned to the other
field like seq_num in the same messsage structure.
* Update the generated proto files.
* Update the extensions.go to use the new field.

Change-Id: Ic449e59c893f31ae3f193b2d6f302b2f96a36825
Signed-off-by: Baohua Yang <baohyang@cn.ibm.com>

When a peer receives a data message (contains a block) it doesn't verify
that the sequence of the block matches the sequence inside the marshaled block,
and just calls VerifyBlock.

The implementation already unmarshals the block, so there is no point
in un-marshalling it outside.
I extended the interface VerifyBlock to accept also a sequence number.

Change-Id: I85c6c57f26f5e404fe8e7ece2abffa58eaded625
Signed-off-by: Yacov Manevich <yacovm@il.ibm.com>

The hash field isn't used anywhere but in the protos comparator function
to compare 2 ledger blocks disseminated.
This imposes a security vulnerability:
A malicious peer can send blocks with an arbitrary hash in the hash field
of the DataMsg, and then these messages would enter the in-memory stores
and fill up the memory.

Since we have no use of the block hash of our own message, I am removing it
entirely.

Change-Id: Ic22ed3d06f102795c8f2a74b27063848affc926a
Signed-off-by: Yacov Manevich <yacovm@il.ibm.com>

The current gossip implementation is initialized with a
set of dial options which don't change during the lifecycle
of the peer.

In order to resolve this, the gossip comm implementation
has been modified to accept a function which returns the
appropriate dial options for secure communincations with
other peers.

When used in the peer, the function passed in
leverages the core/comm packages CASupport.

Change-Id: I99cc4204e27c67f93f8f1ba91891fdb97081c99e
Signed-off-by: Gari Singh <gari.r.singh@gmail.com>

In gossip we have peer identities that are stored in-memory.
This commit adds 2 timely checks that deletes the identity
from memory and also closes connections to corresponding peers.
- That the certificate (identity) didn't expire
- If the identity hasn't been used for a long time

Change-Id: I901b60e2a09c11202f305106991b7eadd5f7b29c
Signed-off-by: Yacov Manevich <yacovm@il.ibm.com>

This CR updates gossip-related logging features in light of
the new changes to the `flogging` package. It 1) sets the log
level at peer startup based on the value in core.yaml and 2)
updates the gossip test cases to default to a log level of
`warning`.

Change-Id: I1b254379e8d0d2d0c8521bafcb9ce15ad6420d1a
Signed-off-by: Will Lahti <wtlahti@us.ibm.com>
Signed-off-by: Yacov Manevich <yacovm@il.ibm.com>

When a peer gets a new config block and it contains CRLs,
the gossip layer needs to be notified in order to close existing
connections to peers that their certificates have been expired.

If this is not done, then these peers are still forwarded data
like peer membership and channel membership, because the connection
is already open.

This commit adds an ability to revoke identities by receiving a predicate
function that: given an identity, it returns whether it is suspected of being
revoked (i.e, the SN is found within some CRL of some MSP).
Then- the gossip layer calls ValidateIdentity on the stored identity, and if
it is found to be invalid- it:
- deletes the identity from memory
- closes an active connection to the peer, if such exists.

Currently the implementation of that predicate is the naive/obvious one,
that suspects all identities.
In a future commit I'll (hopefully) add code that uses the CRLs themselves.

Change-Id: I56d995a3720a736b1242b13a193f9a7933299345
Signed-off-by: Yacov Manevich <yacovm@il.ibm.com>

In the current implementation of the join channel message in the
gossip layer, each anchor peer has its own organization, and the
joinChannel invocation enumerates all anchor peers and builds a
list of organizationIDs from them.

When https://gerrit.hyperledger.org/r/#/c/7105/

 will be introduced,
it will be needed to create a channel only with either no anchor peers
or anchor peers of the organization of the channel creator.
This would confuse the gossip layer because it'll derive that each channel
has only 1 organization instead of the original participants.

I decoupled the anchor peers from the organization IDs by changing
the JoinChannelMessage interface in api/channel.go to have
Members() []OrgIdentityType  which would
return the channel members (organizations)
and AnchorPeersOf(org OrgIdentityType) []AnchorPeer which would
return for each organization, its corresponding anchor peers.

Also (of course) updated the tests accordingly.

Change-Id: I90131a0726fc7ca68fd750247f4df558c5cca0fd
Signed-off-by: Yacov Manevich <yacovm@il.ibm.com>

The MessageCryptoService complains:
verifyBlock -> WARN 0cc Received fabricated block from [...]\
 in DataUpdate: Failed casting SignedBlock to []byte on channel [foo]

Because the object that was passed wasn't a []byte as expected but
the object that contains the []byte as a field.
I changed the signature of VerifyBlock to accept []byte instead
of the previous interface type api.SignedBlock and removed the latter type.

Change-Id: I7320c398b117072c0790f77d9c0f9305b1adf5ea
Signed-off-by: Yacov Manevich <yacovm@il.ibm.com>

When the PEM was removed from the anchor peers, the Connect()
method of the discovery layer wasn't updated accordingly.
As a result, all anchor peers were mapped to the same (empty string) key
in the discovery layer

This commit:
1) Changes the logic and semantics of the Connect():
   It now creates a membership request that doesn't
   (or does, up to the caller) have the internal endpoint inside of it,
   according to the param passed.
   This is because we don't have knowledge of whether the
   remote anchor peer is in our organization or not.
2) Changes the test case of the gossip:ConnectToAnchorPeers
   to simulate a situation that would've caught the bug
   had it been tested in the first place:
   Starting 10 peers, waiting a bit and then starting a *random*
   anchor peer out of the 3 anchor peers given to the 10 peers.
   With the bug- all peers would've mapped all anchor peers to the same
   key, thus unable to connect with probability of 2/3.
3) Changes the test of TestConnect in the discovery package
   to test that the when the Connect() is invoked with false,
   meaning- not to send the internal endpoint, the first membershipRequest
   that is sent indeed doesn't contain the internal endpoint.
   and also adds a test for each of the options (true, false) of
   the method createMembershipRequest(includeInternalEndpoint)

Signed-off-by: Yacov Manevich <yacovm@il.ibm.com>
Change-Id: Ibd167188dac214951adedfc2ef050d635c6db79f

https://jira.hyperledger.org/browse/FAB-2473



This is the third and final in a series of changesets to move the gossip
protos to conform with the official proto style guidelines.

This CR is the most invasive, because it changes pkiID to be pki_id,
causing the generated code to become PkiId instead of PkiId.  This is
very similar to the change from ChaincodeID to ChaincodeId, but much
smaller in scope.

Change-Id: I7ebd2db48ef464ac7f1d024a41fbd11514ca5385
Signed-off-by: Jason Yellick <jyellick@us.ibm.com>

BCCSP is logging the following message:
[BCCSP_FACTORY] GetDefault -> WARN 29a Before using BCCSP,\
 please call InitFactories(). Falling back to bootBCCSP.

If it is not initialized manually. Gossip uses BCCSP's hash
so when running the intensive gossip_test, there are lots of
BCCSP log entries logged.
I added a line that initializes the BCCSP to prevent this.

Change-Id: I1dd19794ced995f6cd3699d35aa138511267e9c4
Signed-off-by: Yacov Manevich <yacovm@il.ibm.com>

Background:
------------
The previous commit added support in the discovery layer
for policies that effect the membership response handling logic.
That logic is based on the selfInformation field which is a signed
GossipMessage that contains an AliveMessage and is supposed to represent
the remote peer that sent the membership request.

Even though the message is validated, nothing prevents an attacker
to replay a signedMessage he recorded.

There is a TODO in the code that says:
https://github.com/hyperledger/fabric/blob/master/gossip/discovery/discovery_impl.go#L293


// TODO: make sure somehow that the membership request is "fresh"

This is to prevent replay attacks, and:
1) This needs to be addressed for FAB-2007
2) A replay attack can happen anyway if a malicious peer gets hold
   of such a message early enough befor the attacked peer got the message

What's in this commit?
----------------------
This commit leverages the fact that a membership request is point-to-point,
and checks that the sender of the membership request is the same peer
that is on the other side of the connection.
Also removed the TODO since now it's not needed anymore.

How is this tested?
--------------------
I added a test that creates such a replay attack and spoofs
a membership request, and compares it to a valid membership
request to demonstrate that the attack prevention works.

Change-Id: I8c994b5627189a1d0fb3f6a7d9edbd9a9c021b2c
Signed-off-by: Yacov Manevich <yacovm@il.ibm.com>

Creation of leader election service during channel initialization in gossip
and connecting between election callback to deliver service

Change-Id: Ic2b8b1b5ebc770abcf4b935d39ce06087086b0c9
Signed-off-by: Gennady Laventman <gennady@il.ibm.com>