Timeout ticker has incorrect behavior when two timeouts placed in quick succession

[ValarDragon]

Feature Request

Summary

I found that we can get negative timeout duration at least in tests, and subsequently incorrect behavior while working on: #3008 . I have made a PR that can reproduce this failing test (at least locally)

Suppose we have code like (We have effectively this in create empty blocks tests)

                startTime := time.Now()
		negTimeout := timeoutInfo{Duration: -1 * time.Millisecond, Height: height, Round: 0, Step: types.RoundStepNewHeight}
		timeout := timeoutInfo{Duration: 5 * time.Millisecond, Height: height, Round: 0, Step: types.RoundStepNewRound}
		ticker.ScheduleTimeout(negTimeout)
		ticker.ScheduleTimeout(timeout)

		// Wait for the timeout to be received
		to := <-c
                elapsedTime := time.Now().Sub(startTime)

The intended behavior is oneof:

• to := negTimeout, elapsedTime ~= 0
• to := timeout, elapsedTime ~= 5ms

Unfortunately we also get the behavior

• to := timeout, elapsedTime ~= 0

Which is our bug. This should be causing test flakyness. unfortunately I don't know enough about our timeout usages to know if this poses any safety problems.

[ValarDragon]

After understanding the race condition more, this will occur whenever we schedule two timeouts at ~the same time. We can get timeout 1 timer delay, but with timeout 2's value. From skimming the test I wrote, we are definitely at risk of this if the timeouts are scheduled within 100000ns of each other. but the true value for risk starting is likely higher.