Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for JDBC & resequencing message stores #89

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

pamod
Copy link

@pamod pamod commented Nov 18, 2023

Purpose

The intention is to allow message store to be more remanent so that the following EIP patterns could be accomplished through synapse

Approach

JDBC user store guide : https://docs.google.com/document/d/1aWIJzi3gUfTbYLxNrXD7K1z4hVb4oY5WMy5sGxFluAo/edit?usp=sharing

Resquencer user store guide :
https://docs.google.com/document/d/1knFw_Twh8jLNaF8-9ObJQ3wMkjXr0FNqnpBv1m3PwVc/edit?usp=sharing

Message Store

Screenshot 2023-12-06 at 05 33 58
  • When a message arrives the message will be placed into the “SeqStore”
  • SequenceIdLookup will extract the sequence id from the given message payload/header based on the specified XPath value
  • Based on the id identified the message will be inserted into the table in DB.

Message Processor

Screenshot 2023-12-06 at 05 34 12
  • Processor would poll to get the message
  • First the last processed message id would be returned from the DB/cache
  • Based on the last processed id the messages would be retrieved from the DB. The top most message would be retrieved and will be validated retrievedMessageId < lastProcessId (this should ensure that the DB has the messages in sorted order)
  • If the validation is successful, the message would be popped from the DB and will be sent to the caller.

Resequence Message Properties

Resequence message store will inherit all properties applicable for JDBC message store. In addition the following

Property Name Description
store.resequence.timeout The count will be a positive integer. The timeout will be specified in seconds. This specifies the timeout the message resequencing order should be ensured. I.e if the current message sequence is ‘3’ it’s predecessor would be 2 and the successor would be 4. If ‘4’ is not present there’s a gap in the sequence. During the gap the processor would wait until the specified timeout is exceeded. If the timeout exceeds it’s limit, the next minimum sequence id available in the store will be selected as its predecessor.If the count is specified as ‘-1’ the processor will wait indefinitely until the correct sequence id is present to fill the gap. Please note that the timeout will be a rough estimate. The durations could vary. Depending on the load of the machine.
store.resequence.id.path Specifies how the store could identify the sequence id from the message content. The path could be either an XPath or a JSON path. Which could be specified in the expression field.  The sequence id should be a positive integer. The store would expect the sequence id to start from 1.


Release note

JDBC message store & Resequencing message store which will support achieving EIP patterns such as guaranteed delivery & Resequencer

Automation tests

Integration tests included. Samples 706 and 707 will be executed as a part of integration tests

@pamod pamod marked this pull request as ready for review December 6, 2023 00:16
@pamod pamod changed the title [WIP] Add support for JDBC & resequencing message stores Add support for JDBC & resequencing message stores Dec 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant