Time-sensitive messages

[c16a]

It would be nice if NATS supported time-sensitive message to solve the below use cases

1. Only deliver this message from the stream if it's not expired yet.
2. Defer the delivery of this message until a certain time.

I'm a long time user of NATS, but very new to contributing. So here is how I think we can approach this problem.

Add support for expiring and deferred messages via NATS headers

Two additional headers can be set by clients to configure time sensitivity

1. The Nats-Before header expires the message at the Unix timestamp provided in the header
2. The Nats-Not-Before header prevents the message to be consumed before the Unix timestamp provided in the header

It is possible that users are already building some of this functionality on the client-side using similar headers. So we could make Nats-Before and Nats-Not-Before header keys configurable by user.

Enable Jetstream messages to be evicted if the Nats-Before header passes

Could start a timer and remove the message from the stream. Perhaps KV could also be used?

Enable Jetstream to selectively skip messages until the Nats-Not-Before timestamp passes

Need Jetstream to not push messages to consumers until Nats-Not-Before passes.

Will look into this in detail and post updates here.

[andreib1]

"Could start a timer and remove the message from the stream. Perhaps KV could also be used?"

At the moment, as a workaround, my solution is exactly that; Remove these messages from NATS, and store in redis with a ksuid key that enables me to poll the messages with a cursor to find ones that are due for redelivery. This method is fraught with complexity as injecting the message back into NATS triggers the message deduplication feature which makes it disappear. The only solution that I have found is to make a new message ID, and have an Original-Id header - which is clumsy.

Although canny, the presented Nats-Before solution has the drawback that the need to defer messages often happens after the message is in flight eg. when the message is to be rescheduled for transmission. This would effectively mean that the message headers are then somehow mutable. My preferred option is to leave the message be, and introduce API functions for rescheduling similar to msg.InProgress which instead communicates the requirement to the server.

Wouldn't this discussion be better in the original issue log? #2042

[c16a]

This was actually in response to another issue (#2541) which spoke of something similar. So wanted to get feedback before I jumped into the code.

Also, mind explaining the need for message headers to be mutable? Neither the Nats-Before nor Nats-Not-Before solutions would not need mutable headers. You simply read the header when it's time to decide whether it's safe to deliver to the client or not.

Although, I'm thinking a built-in worker queue implementation could keep track of these milestones (the expiry or deferral timestamps) and run these delivery tasks when the tick is right. Would be great, but would not be simple.

[andreib1]

Sorry, I get you, it's seems like a very similar use case on first glance, but what you are effectively describing here is message embargo/TTL? It is so close in fact, that the underlying engine for embargo should be able to provide both. TTL is a slightly different creature. @ripienaar has TTL for KV data, it may be worth looking at how he's implemented it if applicable.

I was thinking that either way, for embargo, you would need to have mutable headers to reschedule a live message. i.e. once the message is in flight, there is no way to change the time which it gets delivered, as it would require modifying the message delivery time data that is stored in the header.

It's probably better to use message metadata, as those values are mutable.

[ripienaar]

KV TTL is just a Stream with a Max Age set.

[Julienhardis]

Hello,

i have the same need to keep message in nats until a determinated time.

I saw a fork was created here #2733 but it's not merge to the main branch.

Are you considering this feature which means very interesting ?

[derekcollison]

We are considering this feature. Will not make it in the upcoming 2.9 release but is being considered.

[Julienhardis]

Thank your for your quick answer.

I did open an issue concerning this subject yesterday because i thought this subject was sleeping.

For your information : #3403