consensus: change gossip routines sleep and signaling setup

[ValarDragon]

Feature Request

Summary

Currently every gossip routine attempts to gossip a vote or block part. If it has no more block parts to gossip, it sleeps (by default 100ms). This has three issues:

• We get an unnecessary delay for first part gossip. When we are ready to gossip new block parts, we will be unnecessarily sleeping for another rand[0,100]ms.
• If we only have one block part, we gossip it to a peer, then we have nothing else to gossip, we will sleep for 100ms. If another block part comes in, we have to wait that [0,100]ms.
• Wasted CPU when we are not in the relevant gossip phase

Proposal

Make each gossip thread use a channel that "wakes it up".

For block part gossip:

• When we receive a new block part, go to the next block, or detect our peer is on a new block, restart our loop.

For vote gossip:

• When we receive a new vote, go to the next block, or detect our peer is on a new block, restart our loop.

Otherwise keep the current structure for a loop that is "awake".

[cason]

In summary, we need a way to signal to the gossip data routines that new block parts are, or might be, available to be sent. This is a limitation of the current design, where consensus logic and reactor "communicate" via shared memory.

A possibility that we have discussed is to leverage the existing synchronous signal cs.evsw.FireEvent(types.EventProposalBlockPart, msg) between the consensus logic and reactor.

[cason]

For votes, there is also a synchronous event produced by the consensus logic: cs.evsw.FireEvent(types.EventVote, vote).

[nmarcetic]

Maybe is cheaper to KISS, run it in a loop, and just reduce sleep, especially when a peer is behind in gossipDataForCatchup (probably the root cause of slowness; maybe we could handle that in another thread and keep the main loop cleaner 🤷‍♂️ ).
I like the idea of channels, but it could add extra complexity (also more things to change).
For votes, I don't understand the purpose of those sleeps 😢

[cason]

For votes, I don't understand the purpose of those sleeps 😢

We only sleep when we don't find a vote to send to the peer. We sleep because we retrieve votes from the consensus logic via shared memory. The option we have been discussing is to explicitly signalize the reactor when there are new stuff (e.g., votes) available in the shared state to be sent.

While we find votes suitable to be sent, we skip the sleeps in the loops.

[cason]

This can be in conjunction with #3211.