invalid HTTP version error while using http sink

[TertiumOrganum1]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I can't send data to tempo or grafana agent via http sink.
Here is sink config:

    tempo_sink:
      type: http
      inputs:
        - parse_message
      compression: none
      encoding:
        codec: "json"
      method: post
      tls:
        verify_certificate: false
        verify_hostname: false
      #uri: http://tempo.tempo.svc.cluster.local:4318/v1/traces
      uri: http://grafana-agent.grafana.svc:4318

And here what I get:

2024-04-26T23:54:25.807049Z  WARN sink{component_kind="sink" component_id=my-dev-sample-tempo_sink component_type=http}:request{request_id=1}:http: vector::internal_events::http_client: HTTP error. error=invalid HTTP version parsed error_type="request_failed" stage="processing" internal_log_rate_limit=true

If I forward tempo or grfana agent port 4318 to local machine I able to make POST request using curl, but Vector can't.
If it is not a bug, please tell me where I missed something in config?
The same instance of Vector is able to send logs to loki (sink loki) and elasticsearch. Tried with or without TLS and request sections.
Vector is installed as Daemonset.

Configuration

tempo_sink:
      type: http
      inputs:
        - parse_message
      compression: none
      encoding:
        codec: "json"
      method: post
      tls:
        verify_certificate: false
        verify_hostname: false
      #uri: http://tempo.tempo.svc.cluster.local:4318/v1/traces
      uri: http://grafana-agent.grafana.svc:4318

Version

image: "timberio/vector:0.35.0-distroless-libc"

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[jszwedko]

Can you try running with trace logs enabled and share those? Vector should output some information about each request it is making. I believe Vector will be making HTTP/1.1 requests. Maybe your endpoint only supports HTTP/2 or HTTP/3?