-
-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Updating Concentus dependency to speed up Opus decoding #6757
Conversation
…span-out Opus decoding (so we don't have to make temporary buffer copies), returning a more precise error code from the decoder, and automatically linking the native opus library with P/invoke if supported on the current system
I have so far been unable to find a game that actually uses hardware Opus to test, besides MLB The Show '22 which crashes on startup for other reasons (though it does successfully create a decoder) |
I think Pokemon Legends Arceus uses it, allocations in Concentus were topping the profile last time I checked. |
From what I recall, all Pokémon games uses Opus (except maybe Brilliant Diamond and Shining Pearl). There's also Sea of Stars, Unicorn Overlord, etc. |
@lostromb The NuGet package seems to contain the native libraries multiple times at different paths. |
Arceus does hit this code path and the music seems to be working fine there. But I don't get a hit from Scarlet, Shield, Shining Pearl, Sea of Stars, or a dozen other random games. There's no regression, they just don't seem to use the service at all (again, I assume they might have compiled libopus directly into the game engine?) For the packaging, my (maybe incorrect) understanding was that e: also hmmm, if an indirect library uses |
Updated to 2.1.1 which should have better LibraryImport logic and support content files on publish / AOT |
It looks like the macos build isn't happy with the libopus natives right now:
It seems a bit odd putting all the effort into speeding up the C# port of opus decoding, but then using a native library anyways. |
I mean, I guess my intent 8 years ago was to make a pure C# port that had no dependencies and could run anywhere, but then that plan changed to a hybrid approach where it would just give you a wrapper to a native C implementation if your platform supported it (which turns out to be almost all .Net publish targets), keeping the C# code as a fallback. I could theoretically move the native codec wrappers directly into Ryujinx and skip the Concentus package altogether. But then I would probably have to fuss with things like SetDllImportResolver, and make sure everything loaded properly on every platform at runtime, which would take more testing and there's less of a safety net because there's no managed decoder to fall back to if something fails. And then there'd be complaints that Pokemon crashes on startup or whatever.
|
Is it not possible to split the managed implementation and the native wrapper into separate packages? Then we can consume only the managed package, without shipping the opus decoder binary. That would probably also have size benefits as we don't use the encoder at all (in the C# implementation I believe it can be trimmed). If you want to keep the hybrid approach, then perhaps it could be a third library that consumes the other two and uses the native one if possible. |
We can go a few different ways that I'm open to. I'm just not sure what path you would want to prioritize:
I mention "fast performance" but from a quick benchmark and assuming a 30fps game we're looking at a difference of: |
I prefer 1. I'm not sure why the perf gap is so large though, I wouldn't expect the C# version to be that much slower, unless its missing some vectorized code path or something. It might be worth profiling it, and maybe forwarding performance issues to the .NET runtime team once they have been identified. |
…using ExcludeAssets).
…into update-opus-lib
Simple solution:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You could consider moving the pre-compiled libraries to a separate nuget package as well, I have seen some bindings doing that. Something like Concentus.Native
.
|
||
public int SampleRate => _decoder.SampleRate; | ||
public int ChannelsCount => _decoder.NumChannels; | ||
|
||
public Decoder(int sampleRate, int channelsCount) | ||
{ | ||
_decoder = new OpusDecoder(sampleRate, channelsCount); | ||
OpusCodecFactory.AttemptToUseNativeLibrary = false; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe that should be done in a static constructor for the HardwareOpusDecoder
class?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I would normally like to avoid static constructors, but this one seems simple enough. Changed.
Yeah, looking into it more I see the benefit of modern Nuget dependencies in NetCore, namely that on publish and in AoT workflows, it's smart enough to only pull the native binary for the platform you are targeting, rather then just batch copying a big pile of random other .dlls that you won't need. The only problem was I still want to support older .Net Framework installs which can't do that, so I ended up making a few native packages and a metapackage which should support everyone. This PR now just updates the pure C# opus decoder and addresses some minor bugs; not touching native at all. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm, thanks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, nice to see the library using spans now.
Updating to the new Concentus package version which has updated features:
Automatically utilizing a native opus library with P/invoke if supported on the current system(we don't actually care about this as decoding is very fast anyways)This will also have the side effect of copying extra binaries to project output directories and increase the install size by a few megabytes - for files like/runtimes/win-x64/native/opus.dll
. I have also not tested the P/invoke code path on all platforms - just x64 Windows/Linux/Mac. If you want to see if it's working properly, passConsole.Out
as the last parameter to theOpusCodecFactory
methods. And if the native library probe is causing problems, it can be disabled globally viaOpusCodecFactory.AttemptToUseNativeLibrary
Update: Not bothering with native libs at all. This will continue to just use the pure C# decoder.