Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nakadi clients resilience to partial outage and partial success #217

Open
adyach opened this issue Sep 8, 2023 · 5 comments · May be fixed by #219
Open

Nakadi clients resilience to partial outage and partial success #217

adyach opened this issue Sep 8, 2023 · 5 comments · May be fixed by #219

Comments

@adyach
Copy link

adyach commented Sep 8, 2023

Nakadi publishing API accepts events in batches. It can fail to publish some events from the batch to underlying storage (Apache Kafka). In that case Nakadi publishing API will return error that batch was partially successful.
It can create problems the following problems, depending on how the Nakadi client and the publishing application deals with this partial success response:

  • increase in traffic on Nakadi publishing API due to Nakadi clients retrying the whole batch over and over
  • the application retries identical batches which prevents application from progressing

The following should be done to decrease the possibility of mentioned problems:

  • Nakadi client should contain a note to developers that publishing can experience partial success. This should be in the client documentation and ideally also within the self contained code documentation, raising awareness for the users, e.g. via docstrings.

  • An optional retry method on batch level can be provided for the whole batch, but the default strategy must contain a backoff - solution in case of continued errors to publish to Nakadi.

  • An optional retry method can be provided that only re-publishes unsuccessful events to Nakadi. This retry must also support a backoff strategy by default.

  • Clients must expose the result of a publishing request in a way that developers can understand that there is the possibility of a partial success for batch publishing.

@gchudnov
Copy link
Collaborator

Hi @adyach
Thank you for reaching out,

At the moment there should be a backoff logic that retries the publishing.
And the result of partial publishing is already exposed to the clients.

I'll update the readme to clarify the behavior.

@adyach
Copy link
Author

adyach commented Sep 12, 2023

@gchudnov docs update is great. thank you!

@adyach
Copy link
Author

adyach commented Oct 9, 2023

@gchudnov any update on this ?

@gchudnov
Copy link
Collaborator

gchudnov commented Oct 9, 2023

@adyach
sorry, being quite busy :(
will finish the doc by the end of tomorrow (Oct 10, 2023)

@gchudnov gchudnov linked a pull request Oct 10, 2023 that will close this issue
@gchudnov
Copy link
Collaborator

@adyach
just made a PR, please take a look: #219

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants