-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Akka.Streams: Memory Leak with circular GraphInterpreter reference #6947
Comments
is that a fix for this @Zetanova ? |
@Aaronontheweb Most likely the solution would be to fix the callback registration It makes a big difference not only for performance to use Optional would be good to create first a failing memory unload unit test. |
@Aaronontheweb is it poissble to add some special integrations tests somewhere to test for memory leaks ?
But both tests would have side-effects on the testing system itself. |
I've observed behavior like his before as well. I'll note that in addition to the
Yep. We probably need to add some APIs to help with this too (GraphDSL and some of the source/flow variants come to mind.) |
Its the whole materialized graph including the stage instance construct that are not released. The special thing about |
FYI, this does this by design - it's meant to hang onto data indefinitely over long periods. We wrote it for |
I've tracked this down to how peculiar Depending on how busy the supervisor actor is, it can take some time until all of the resources being held by all the logic inside the stream to be released, this is especially true for buffered stream stages where it will hold on to enumerator references even after it completed its job. |
Specific to the |
Version Information
Akka.Streams 1.5.13
Describe the bug
GraphInterpreter <> GraphStageLogic
and/or
GraphInterpreter <> Connections <> GraphStageLogic
and/or
GraphInterpreter <> GraphInterpreterShell
generating a circular reference and the full Graph and its stages
are found under gcdump "dead objects".
the result will be an OOM
To Reproduce
I am getting this issue with an grpc service to akka.remote (streamRef)
but it should be already be present with an local system.
The Graph/Stages instance is relatively small, but if used with
Source.From(IEnumerable<T>)
this Stage will hold the full
IEnumerable<T>
as reference.If it contains >1MB on data it will lead to an OOM faster.
There are no errors with the following code.
But it leaks the full MyRecord entries in memory until OOM.
Expected behavior
All instances of the Enumerable and GraphInterpreter and Stages should be collectable by the GC
after the successful or unsuccessful execution of the stream.
Actual behavior
The GraphInterpreter Stages and the full used Enumerable are found in "dead objects"
with circular references and getting never collected.
(sometimes after the OOM and a successful ActorSystem termination).
Environment
dotnet 6.0
ubuntu-jammy
Docker Desktop and k8n
Additional context
Because the
GraphInterpreter
set theGraphStageLogic.Interpreter
propertyI tried to unset it on
TerminateStage
but this breaks a lot of assumptions/teststhat the
GraphStageLogic.Interpreter
is always available even after stage termination.The one liner can be found here:
Zetanova@9842d4d
The text was updated successfully, but these errors were encountered: