When adding a block to the message store, if it is added and
not rejected - it is then added to the block puller.

When the block is removed from the message store, a callback
is triggered to remove the block from the block puller.

However, these 2 operations are not atomic.

Since we can add a block to the message store from both AddToMsgStore
(which is invoked by Gossip() ) and from HandleMessage,
we can have the following schedule:

  1) A block with sequence of 100 is gossiped by the upper
    layer of the peer, and AddToMsgStore is called, which
    adds the block to the message store, and the CPU is preempted.
  2) A block message 210 is received via HandleMessage and it causes
     the block 100 to be evicted from the message store, and the
    callback to remove the message from the block puller is called,
    but it is not removed because it is not there yet.
  3) The block 210 is added to the block puller, since it was added
     to the message store.
  4) The CPU is back to perform AddToMsgStore, and adds block 100
     to the block puller.

Now the block puller has block 100, and the message store doesn't
have block 100 anymore - which means it will never be evicted from
the block puller.

To prevent this we need to make these 2 operations be atomic.

Change-Id: I3b7d0d013ce8da5d9a0e40f8b0cdbc3edaed22c9
Signed-off-by: yacovm <yacovm@il.ibm.com>