Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Explicit frame drop signal from VideoEncoder #742

Open
Djuffin opened this issue Nov 6, 2023 · 3 comments
Open

Explicit frame drop signal from VideoEncoder #742

Djuffin opened this issue Nov 6, 2023 · 3 comments
Assignees
Labels
extension Interface changes that extend without breaking. RTC Support for advanced RTC scenarios

Comments

@Djuffin
Copy link
Contributor

Djuffin commented Nov 6, 2023

Currently the spec doesn't specify any way for VideoEncoder to signal that a frame has been dropped due to realtime mode constraints.
I think it would be useful to have such a signal.
Possible options:

  1. the output callback should be called with an empty encoded chunk
  2. the output callback should be called with a new flag set in EncodedVideoChunkMetadata

This came up after WPT test change that accounts for frame drops.

@Djuffin
Copy link
Contributor Author

Djuffin commented Nov 8, 2023

On the other hand, 2 existing implementations drop frames silently, so using the regular output callback as is will be a breaking change. If we ever do this we'll need a VideoEncoderConfig option to turn it on.
Let's wait and see if it actually needed by people building RTC apps.

@Djuffin Djuffin closed this as not planned Won't fix, can't repro, duplicate, stale Nov 8, 2023
@Djuffin Djuffin reopened this May 3, 2024
@Djuffin Djuffin self-assigned this May 3, 2024
@Djuffin
Copy link
Contributor Author

Djuffin commented May 3, 2024

At least Meet is interested in drop frame notifications.

@Djuffin
Copy link
Contributor Author

Djuffin commented May 3, 2024

Currently when a real time encoder drops a frame clients don’t get any signal from VideoEncoder. If they really want to know that a frame was dropped they need to look at the timestamps of the encoded video chunks.

This issue can be addressed by a new flag in the encoder config that would cause the encoder to call EncodedVideoChunkOutputCallback for dropped frames with an empty data buffer in the encoded chunk.

partial dictionary VideoEncoderConfig {
	boolean notifyAboutFrameDrops = false;
}

@Djuffin Djuffin added the RTC Support for advanced RTC scenarios label May 3, 2024
@aboba aboba added the extension Interface changes that extend without breaking. label May 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
extension Interface changes that extend without breaking. RTC Support for advanced RTC scenarios
Projects
None yet
Development

No branches or pull requests

2 participants