Change flowrate.clock() to be based on a variable that is scheduled to update every 20ms

[ValarDragon]

Feature Request

Summary

Currently the flowrate package has every single call to it do a time.Now() call, and then round it to the nearest 20ms, and uses that as a "flowrate epoch" internally for its computations.

This leads to lots of time spent in time.Now() and rounding logic. Result from a 1 hour live osmosis node profile (~1 month ago), that overlapped with the epoch (60s long):

10 seconds spent in time.Now() checking, when were trying to just send the packet, feels very significant. Unfortunately I don't have statistics for the number of packets we sent in this time period to be able to bound it though. Assuming the current flowrate functionality is what we want, we actually have a much more efficient way to implement this function. We can have a global "currentClock" var. Every clock() function is an atomic read of this var. We spin up another goroutine at initialization that atomically writes this var every 20ms. This should change this clock time to then be a tiny fraction of what it is now, with very little consensus delay.

Right now the code is currently

// czero is the process start time rounded down to the nearest clockRate
// increment.
var czero = time.Now().Round(clockRate)

// clock returns a low resolution timestamp relative to the process start time.
func clock() time.Duration {
	return time.Now().Round(clockRate).Sub(czero)
}

so this would change to roughly

var curClockTime time.Duration = time.Duration(0)

func clockTimeUpdateLoop() {
   for {
          time.Sleep(clockRate)
          newClockTime = oldclockfn()
          Atomic.SetInt64(newClockTime, &curClockTime)
    }
}

// clock returns a low resolution timestamp relative to the process start time.
func clock() time.Duration {
	return time.Duration(Atomic.LoadInt64(curClockTime))
}

I don't think we care about high fidelity time keeping of the flow rate, but if we did, then we could do the epoching goroutine do a time.Sleep for say 19.9ms, and check for the final boundary point with a .1ms sleep loop.

Problem Definition

Lowers CPU contention in trying to send packets.

[ValarDragon]

Happy to try PR'ing a change if this sounds acceptable to folks. (Having a single clock variable, and a background process started on init for updating it. Alternatively could make the background process start on the first Flowrate.New)

[sergio-mena]

Sounds reasonable to me. I'd prefer the new goroutine to start upon first use of flowrate