Low default value of StableSequenceThreshold causes premature engagement of skip logic when DB commit is slow

[svrx]

I've recently upgraded to 6.4.1 and have stumbled on an issue where events, projected by an async daemon, where no projected currently.

After investigation we found that this behaviour was due to the fact that transaction experiencing a slow commit, causing daemon to try to process events before they are actually committed and eventually skipping them altogether.

The issue seems to have been introduced by changes made for #2681, where the low 3 seconds threshold was being triggered prematurely if no other changes happen meanwhile.

I'm still trying to understand whether the new logic is necessary as only seems to be engaged when HighestSequence is ahead of LastMark for the stale period and no new events have been entered afterwards, otherwise they are handled by the gap detection logic. Can you clarify which angle is meant to be covered by this logic?

The situation can occur or be compounded by:

• Slow DB (ex: backups, low-spec, network)
• Batched operations
• Heavy inline projections

Can at least the default value be changed to a higher value (30 sec, 60 sec, ...)?
I'm sure it'll save new user some moments of head scratching.

[jeremydmiller]

I'm not comfortable with moving the default all the way up to 30. I'd buy 10 maybe. But without a really compelling reason, I'd prefer not to make any new releases in the 6.* branch if we don't have to.

I'm converting this to a discussion in the meantime

[svrx]

For me I can easily configure a different value on init. So no need for changes on the old release.

My point is only that this logic should be reviewed since it's relatively new and can cause events to be ignored from projections easily due to the low threshold.

As it stands currently after 3 seconds of allocating a new batch of sequence ids for event persistence it'll consider projections stale, attempting to load events that may not be committed yet and skipping them, causing inconsistencies in the respective projections.

All this seems a very aggressive behaviour especially at 3 second default delay for it to trigger.

[svrx]

I may not have a full picture here.
Although while looking at the broader picture for this logic, it does feel that it may only be relevant for a very rare edge case where the head of the event stream has sequence ids that take time or never get committed and no events are inserted afterwards.
If that's the case, skipping forward only avoids log spamming until new events are inserted, which is mostly inconsequential.

In summary this logic seems to address an innocuous scenario by introducing a severe risk for system consistency.

Is there any angle am I missing here?
I'm just trying to understand the full picture here, as I may be able to contribute to improve this area.