feat: have flightGroups take some time to allow reuse

[moshegood]

Max memory usage on two clusters with/without this change.
The lower line is the one with the change.
Note also, the high memory usage servers were actively shedding load and rejecting connections to stay alive.

Other Approaches

The main other way to do this would be to cache the data from the MakeServerSidePutEvent(...) call in getReplayEvent()
That would require cache-invalidation, which would require the data stores to have some version id to enable the ld-relay to know that the underlying information has changed.

Requirements

• I have added test coverage for new or changed functionality
• I have followed the repository's pull request submission guidelines
• I have validated my changes against all supported platform versions

Related issues

Provide links to any issues in this repository or elsewhere relating to this pull request.

Describe the solution you've provided

Provide a clear and concise description of what you expect to happen.

Describe alternatives you've considered

Provide a clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context about the pull request here.

[moshegood]

This function is only called when we have a new or reconnecting client.
So some small delay here should be acceptable.

[cwaldren-ld]

Hi @moshegood, I like the idea here - trading increased latency for decreased memory usage.

In your use-case, what would you actually use as the configured LD_STREAMING_DELAY_SECONDS?

One downside I'm seeing is that the latency would be introduced even if no one else is being served. A dynamic approach might calculate the incoming RPS and use that to increase the latency, but that might be too complicated.

[moshegood]

Hi @moshegood, I like the idea here - trading increased latency for decreased memory usage.

In your use-case, what would you actually use as the configured LD_STREAMING_DELAY_SECONDS?

One downside I'm seeing is that the latency would be introduced even if no one else is being served. A dynamic approach might calculate the incoming RPS and use that to increase the latency, but that might be too complicated.

I've been testing with 2 seconds.
Our big spikes in connectivity come during deployments and after network reconnects between the relays and LaunchDarkly.

[moshegood]

Would it be better if we only delay when there was another request completed very recently?
For example, pause the flight group only if there was another request completed less than a second ago?

[cwaldren-ld]

I think so, and I'm wondering if we can use something like token bucket to solve this.

For example:

0. Setup token bucket with rate 1 req/sec (can be configurable)
1. Request comes in, check the token bucket.
2. If token available, then serve the request immediately.
3. Otherwise, block and initiate flightgroup.
4. [meanwhile, other requests may come in and join the flightgroup.]
5. Serve the requests all at once. This fills the token bucket back up, go to (1).

[moshegood]

Changed things around so that a single new client connection is responded to immediately.
We only delay the bulk fetch of all data between batches of new clients if we very recently fetched that data, and if the option to do so is set.

[moshegood]

Any updates required on this PR?
Do you need to run additional tests?

[cwaldren-ld]

Hi @moshegood , pardon the delay on my end.

We are still discussing how to approach this internally. Mainly it seems that we might have outgrown the usage of the flightgroup and need a more sophisticated system.