Add option to enforce a minimum latency to improve frame pacing #1139
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
If a user has a poor network connection (high jitter), they may experience instability in frame delivery. This manifests as stutters, and overall is not a great experience. This problem is exacerbated when frames are arriving near the end of the client's vsync period (since there's less time to present the frame before we roll over to the next vsync period).
Solution
To account for jitter in the network connection (or in the host PC capture rate), we can allow the user to purposefully buffer some frames. This trades latency for stability. However, we need to be careful with how the buffering is done. If we simply render a frame once the next is available, that gives us one frame of buffer. However, that doesn't solve the frame pacing issue at all, since the second frame's arrival time dictates the render time. We can't rely on any individual frame's arrival time for timing our frame rendering.
Instead, we need to have a consistent schedule for releasing frames for rendering. We can measure the average time a frame spends in the input queue before it is released for rendering. Based on that measurement, we can gradually adjust our frame release schedule to target a desired minimum latency. Benefits of this approach are:
This "minimum latency" approach is exactly what is implemented in this PR. (I'm also open to different naming... Maybe "buffered latency"?)
Related Items
Idea on Moonlight Ideas board: https://ideas.moonlight-stream.org/posts/251/frame-buffering-option
Testing
I've tested this (and other methods for smoothing frame delivery) extensively, and found this gives me the best experience. My hardware is:
I typically use this option with 6 milliseconds of minimum latency, vsync enabled, and frame pacing disabled (not supported on Steam Deck anyways).
I'm looking for others to test with various hardware configurations to see whether it offers a net improvement.