Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multistream projections don't always start properly when maxAge is set on the input stream #4259

Open
hayley-jean opened this issue May 14, 2024 · 0 comments
Labels
good first issue This is a good first issue for anyone that wants to start contributing to EventStoreDB kind/bug Issues which are a software defect subsystem/projections Issues relating to the projections framework

Comments

@hayley-jean
Copy link
Member

Describe the bug
If $maxAge or $maxCount is set on the input stream of a fromStreams projection, the projection can get stuck in the Starting/StateLoaded state when it starts up.

This seems to happen in the specific case where events in the input stream are missing after the checkpoint of the projection, but there are still some events in the input stream. The error logged in the server is coming from MultiStreamMultiOutputCheckpointManager where it will throw an exception even if the read was successful if it is empty

To Reproduce
The easiest way to reproduce this is with $maxCount because it is more deterministic than $maxAge:

  1. Create a projection with the following definition:
fromStreams([
    'test-stream1',
    'test-stream2',
    'test-stream3'
])
.when({
  $any: function(state, event) {
    linkTo('test-output', event);
  }
})
  1. Write 20 events to test-stream1.
  2. Set the $maxCount on test-stream1 to 5. (This can also be achieved by setting the $maxAge and waiting for some of the events to expire)
  3. Restart the node or projections subsystem.
  4. When the server restarts, the projection will be stuck in the Started/StateLoaded state.

Expected behavior
The projection starts up and continues processing events.
Or at the very least, the projection should fault and get into a state which the user can recover it from.

Actual behavior

  1. The projection gets stuck in the Starting/StateLoaded state.
  2. The server logs an error for each expired event:
[20564,25,11:53:47.566,ERR] Error while processing message "EventStore.Core.Messages.ClientMessage+ReadStreamEventsBackwardCompleted" in queued handler '"Projection Core #2"'.
System.Exception: Cannot read 26@test-stream. Error:
   at EventStore.Projections.Core.Services.Processing.MultiStreamMultiOutputCheckpointManager.<>c__DisplayClass16_0.<EnqueuePrerecordedEvent>b__0(ReadStreamEventsBackwardCompleted completed) in C:\Users\hayle\code\EventStore\src\EventStore.Projections.Core\Services\Processing\MultiStreamMultiOutputCheckpointManager.cs:line 165
   at EventStore.Core.Helpers.ReadStreamEventsBackwardHandlers.AdHoc.Handle(ReadStreamEventsBackwardCompleted message) in C:\Users\hayle\code\EventStore\src\EventStore.Core\Helpers\ReadStreamEventsBackwardHandlers.cs:line 83
   at EventStore.Core.Helpers.ReadStreamEventsBackwardHandlers.Tracking.Handle(ReadStreamEventsBackwardCompleted message) in C:\Users\hayle\code\EventStore\src\EventStore.Core\Helpers\ReadStreamEventsBackwardHandlers.cs:line 41
   at EventStore.Core.Messaging.RequestResponseDispatcher`3.Handle(TResponse message) in C:\Users\hayle\code\EventStore\src\EventStore.Core\Messaging\RequestResponseDispatcher.cs:line 88
   at EventStore.Core.Messaging.RequestResponseDispatcher`3.EventStore.Core.Bus.IHandle<TResponse>.Handle(TResponse message) in C:\Users\hayle\code\EventStore\src\EventStore.Core\Messaging\RequestResponseDispatcher.cs:line 78
   at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message) in C:\Users\hayle\code\EventStore\src\EventStore.Core\Bus\MessageHandler.cs:line 24
   at EventStore.Core.Bus.InMemoryBus.Publish(Message message) in C:\Users\hayle\code\EventStore\src\EventStore.Core\Bus\InMemoryBus.cs:line 290
   at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o) in C:\Users\hayle\code\EventStore\src\EventStore.Core\Bus\QueuedHandlerMRES.cs:line 124

EventStore details

  • EventStore server version: 21.10 - 24.2

Additional context
This appears to be similar to #1333

The projection can be recovered by removing the $maxAge or $maxCount metadata on the input streams.
$maxCount can be safely set to a number larger than the projection's checkpoint count.

@hayley-jean hayley-jean added kind/bug Issues which are a software defect subsystem/projections Issues relating to the projections framework good first issue This is a good first issue for anyone that wants to start contributing to EventStoreDB labels May 14, 2024
@hayley-jean hayley-jean changed the title Multistream projections don't always load properly when maxAge is set on the input stream Multistream projections don't always start properly when maxAge is set on the input stream May 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue This is a good first issue for anyone that wants to start contributing to EventStoreDB kind/bug Issues which are a software defect subsystem/projections Issues relating to the projections framework
Projects
None yet
Development

No branches or pull requests

1 participant