Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[INC-696] Validate Kafka configuration on startup #3404

Closed
jjbayer opened this issue Apr 10, 2024 · 3 comments · Fixed by #3543
Closed

[INC-696] Validate Kafka configuration on startup #3404

jjbayer opened this issue Apr 10, 2024 · 3 comments · Fixed by #3543
Assignees
@iambriccardo
Labels
filler Requires little effort to resolve. Ready to be picked up anytime.

Comments

@jjbayer
Copy link
Member

jjbayer commented Apr 10, 2024

When Relay is configured to publish to a non-existing topic, it currently starts normally and only starts logging errors when traffic for that topic comes in.

Relay should fail startup when a Kafka topic does not exist.

Note: This issue is a narrower version of #2766.
@olksdr olksdr added the filler Requires little effort to resolve. Ready to be picked up anytime. label Apr 15, 2024
@olksdr olksdr mentioned this issue Apr 17, 2024
Merged
@jjbayer jjbayer assigned jjbayer and unassigned jjbayer Apr 19, 2024
@Dav1dde
Copy link
Member

Dav1dde commented May 3, 2024 •
edited

Relay should fail startup when a Kafka topic does not exist.

This is possible but breaks with a cluster that relies on topics to be auto created, which is the default setting.

Probably an issue for self hosted and our own tests also rely on this behaviour.

@Dav1dde Dav1dde mentioned this issue May 6, 2024
Merged
@lynnagara
Copy link
Member

lynnagara commented May 6, 2024 •
edited

Self hosted doesn't rely on topic auto-creation (it uses snuba bootstrap + https://github.com/getsentry/self-hosted/blob/67382fd2abc0357e6d39634c4051cae0240da9c2/install/create-kafka-topics.sh for the rest IIRC)

@Dav1dde
Copy link
Member

Dav1dde commented May 7, 2024

Ah okay, I saw the list and though it was a bit short, if the rest is created elsewhere that's good to know.

Our current plan would be to have config flag which toggles a strict mode in which Relay validates topics, question is whether it should be on by default.

@Dav1dde Dav1dde self-assigned this May 7, 2024
@iambriccardo iambriccardo assigned iambriccardo and unassigned Dav1dde May 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@iambriccardo iambriccardo

Labels
filler Requires little effort to resolve. Ready to be picked up anytime.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ref(kafka): Validate topics on startup getsentry/relay
5 participants
@Dav1dde @jjbayer @lynnagara @olksdr @iambriccardo