-
Notifications
You must be signed in to change notification settings - Fork 169
Index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000) #1694
Comments
@montag could you explain how you are generating and sending the trace data to Promscale? I assume you are using the OpenTelemetry collector? What's the configuration you are using? Are you also sending metrics? From this log message the issue seems to be with metric ingestion and not traces. Maybe you have something that is automatically generating metrics from trace data? |
FTR: this is the same as #1091, which we fixed for tracing tables. The fix should be similar should we decide to go for it. |
Here's my otel collector config
And my app's ENV vars
|
I'm not using otel for app metrics but some of the instrumentation libraries might be sending some. I'm using these libs:
|
I worked around this issue by removing metrics from my otel collector config but at some point, I will need to be able to use them again. |
Describe the bug
Promscale stops writing trace data with an 'Error sending samples to remote storage' error for each sample ingested.
To Reproduce
Attempt to store otel trace data with large SQL or stacktrace tag values.
Expected behavior
Expect traces to be stored and not dropped.
Screenshots
If applicable, add screenshots to help explain your problem.
Configuration (as applicable)
Version
Additional context
Traces are collected without error for a short period of time then the error starts and traces are no longer stored.
Example log entries:
version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:55:31.610Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.053Z level=warn ts=2022-10-13T06:55:36.182Z caller=write.go:244 msg=“Error sending samples to remote storage” err=“copier: writing series: error setting series ids: error filling labels: ERROR: index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:55:36.611Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.053Z level=warn ts=2022-10-13T06:55:41.218Z caller=write.go:244 msg=“Error sending samples to remote storage” err=“copier: writing series: error setting series ids: error filling labels: ERROR: index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:55:41.610Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.053Z level=warn ts=2022-10-13T06:55:46.253Z caller=write.go:244 msg=“Error sending samples to remote storage” err=“copier: writing series: error setting series ids: error filling labels: ERROR: index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:55:46.610Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.053Z level=warn ts=2022-10-13T06:55:51.288Z caller=write.go:244 msg=“Error sending samples to remote storage” err=“copier: writing series: error setting series ids: error filling labels: ERROR: index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:55:51.610Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.053Z level=warn ts=2022-10-13T06:55:56.303Z caller=write.go:244 msg=“Error sending samples to remote storage” err=“copier: writing series: error setting series ids: error filling labels: ERROR: index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:55:56.588Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.053Z level=info ts=2022-10-13T06:55:57.589Z caller=throughput.go:90 msg=“ingestor throughput” metric-metadata/sec=496 level=warn ts=2022-10-13T06:56:01.337Z caller=write.go:244 msg=“Error sending samples to remote storage” err=“copier: writing series: error setting series ids: error filling labels: ERROR: index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:56:01.587Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.053Z level=warn ts=2022-10-13T06:56:06.372Z caller=write.go:244 msg=“Error sending samples to remote storage” err=“copier: writing series: error setting series ids: error filling labels: ERROR: index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:56:06.588Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.053Z level=warn ts=2022-10-13T06:56:11.403Z caller=write.go:244 msg=“Error sending samples to remote storage” err=“copier: writing series: error setting series ids: error filling labels: ERROR: index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:56:11.588Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.05
The text was updated successfully, but these errors were encountered: