Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kafka sink healthcheck is not working with templated topic #10465

Open
fpytloun opened this issue Dec 15, 2021 · 4 comments 路 May be fixed by #20373
Open

Kafka sink healthcheck is not working with templated topic #10465

fpytloun opened this issue Dec 15, 2021 · 4 comments 路 May be fixed by #20373
Labels
sink: kafka Anything `kafka` sink related type: bug A code related bug.

Comments

@fpytloun
Copy link
Contributor

Community Note

  • Please vote on this issue by adding a 馃憤 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Vector Version

0.18.1

Vector Configuration File

    [sinks.out_fluent_kafka]
      type = "kafka"
      inputs = ["in_fluent"]
      encoding.codec = "json"
      bootstrap_servers = "kafka:9092"
      topic = "{{ _topic }}"

Debug Output

Expected Behavior

Healthcheck should use some default topic. Either by adding new parameter, eg. healthcheck.topic or better by having parameter for default topic to catch events where templated topic doesn't match.

Actual Behavior

WARN vector::sinks::kafka::sink: Could not generate topic for healthcheck. error=Missing fields on event: ["_topic"]

Example Data

Additional Context

References
@fpytloun fpytloun added the type: bug A code related bug. label Dec 15, 2021
@spencergilbert spencergilbert added the sink: kafka Anything `kafka` sink related label Dec 15, 2021
@sillent
Copy link
Contributor

sillent commented Dec 29, 2021

Maybe i'm wrong, but 'topic' is a type of 'String' , and cannot be set by the template value

jszwedko added a commit that referenced this issue Dec 29, 2021
@jszwedko
docs(kafka sink): Document that topic is templatable
2aa178f 
Ref: #10465 (comment)

Signed-off-by: Jesse Szwedko <[email protected]>
@jszwedko jszwedko mentioned this issue Dec 29, 2021
Merged
@jszwedko
Copy link
Member

Maybe i'm wrong, but 'topic' is a type of 'String' , and cannot be set by the template value

It is actually templatable but the docs didn't indicate that. I opened #10628 to fix that.

spencergilbert pushed a commit that referenced this issue Dec 30, 2021
@jszwedko
docs(kafka sink): Document that topic is templatable (#10628)
e666b42 
Ref: #10465 (comment)

Signed-off-by: Jesse Szwedko <[email protected]>
@spencergilbert spencergilbert mentioned this issue Jan 12, 2022
Open
@debugmiller
Copy link

I think the issue here is that the healthcheck for the sink emits this warning if you are using the templated value for the topic. (presumably because the healthcheck is generated internally and does not know what topic to use?

@fpytloun
Copy link
Contributor Author

I think the issue here is that the healthcheck for the sink emits this warning if you are using the templated value for the topic. (presumably because the healthcheck is generated internally and does not know what topic to use?

Yes, that's correct. We probably need new parameter to set healtcheck topic.

@yalinglee yalinglee linked a pull request Apr 25, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
sink: kafka Anything `kafka` sink related type: bug A code related bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

enhancement(kafka sink): Make healthcheck topic configurable yalinglee/vector
5 participants
@jszwedko @fpytloun @sillent @spencergilbert @debugmiller