Stream and consumer stats

[Inkathu]

Hi,
We have a problem in our project.
We have the stream with the following state:

Information for Stream OrchestrationStream_511e969a754345e994a04362fa5cc24f created 2024-01-17 16:04:19

             Subjects: OrchestrationStream_511e969a754345e994a04362fa5cc24f.>
             Replicas: 1
              Storage: File

Options:

            Retention: Limits
     Acknowledgements: true
       Discard Policy: New
     Duplicate Window: 24h0m0s
    Allows Msg Delete: true
         Allows Purge: true
       Allows Rollups: false

Limits:

     Maximum Messages: unlimited
  Maximum Per Subject: unlimited
        Maximum Bytes: unlimited
          Maximum Age: 1y0d0h0m0s
 Maximum Message Size: unlimited
    Maximum Consumers: unlimited


State:

             Messages: 969,518
                Bytes: 792 MiB
             FirstSeq: 357 @ 2024-01-17T18:34:37 UTC
              LastSeq: 1,055,683 @ 2024-01-23T13:31:50 UTC
     Deleted Messages: 85,809
     Active Consumers: 2
   Number of Subjects: 376

And consumer:

Configuration:

Name: OrchestratorService_OrchestrationStream_Consumer
    Delivery Subject: _INBOX.6-5svvOJMgoW1WwG28Zl6c
      Filter Subject: OrchestrationStream_511e969a754345e994a04362fa5cc24f.>
      Deliver Policy: All
          Ack Policy: Explicit
            Ack Wait: 10m0s
       Replay Policy: Instant
     Max Ack Pending: 1,000
        Flow Control: false

State:

   Last Delivered Message: Consumer sequence: 1,125,774 Stream sequence: 1,023,753
     **Acknowledgment floor: Consumer sequence: 88,353 Stream sequence: 86,668**
         Outstanding Acks: 1,000 out of maximum 1,000
     Redelivered Messages: 943
     Unprocessed Messages: 31,930
          Active Interest: No interest

The consumer has 31930 unprocessed messages and the main question is: how could it be that the last delivered message in the stream is 1 023 753, but the last acked is 8668?
It seems like the Acknowledgment floor is hung at this position :(
Is it normal to have such deviation?
Is it possible that the message pointed with floor is removed and the floor is stopped?

We rely on floor numbers in case of a crash to recover the state of the system, but if it could hang in such a way - it's a problem for us. Maybe you could suggest smth?

Stream sequence in Last Delivered Message - it's delivered, but not acked, right?
Server version: 2.10.9
Thanks.

[ripienaar]

 Outstanding Acks: 1,000 out of maximum 1,000

It appears your application is not acknowleding messages. How do you ack messages when processing them?

[ripienaar]

If you're not acking then the floor doesnt move. The floor is the lowest message before which all other messages are acked.

Not acking is the root cause, fix that then we can look at other options for whats going on.

[Inkathu]

This is the state of stream and consumer after several crashes. And not acked messages could be the result of several crashes.

Could you answer about the deviation of delivery and floor stats?

[ripienaar]

This consumer will not deliver messages until you ack the ones being delivered or redelivered. Everything will stay as it is.

[Inkathu]

How to check which messages are without of ack?
I understand what you mean and I don't ask - why the consumer is not getting new messages. I know that the consumer is stuck.
How delivered stream sequence could point to 1,023,753? and floor sequence is too behind of this pointer.

[ripienaar]

There has been several seemingly related bug fixes in newer versions like #5008.

Suggest you upgrade your server and fix the acks and then we can look more, for now there is a huge elephant in the room, get rif of that so the picture is clearer.