Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exporter: The success rate is too low. #3364

Open
winlinvip opened this issue Jan 6, 2023 · 1 comment · May be fixed by #3367
Open

Exporter: The success rate is too low. #3364

winlinvip opened this issue Jan 6, 2023 · 1 comment · May be fixed by #3367
Assignees
@chundonglinlin
Labels
Kubernetes For K8s, Prometheus, APM and Grafana. TransByAI Translated by AI/GPT.
Milestone
5.0

Comments

@winlinvip
Copy link
Member

winlinvip commented Jan 6, 2023 •
edited

Description

When enable prometheus exporter, there are lots of error clients events, so that the success rate is too low.

  1. SRS Version: SRS 5.0

  2. Start SRS origin server:

env SRS_LISTEN=19350 SRS_MAX_CONNECTIONS=1000 \
  SRS_PID=./objs/origin.pid SRS_DAEMON=off SRS_SRS_LOG_TANK=console \
  SRS_EXPORTER_ENABLED=on \
  ./objs/srs -e
  1. Start SRS edge server:
./objs/srs -c conf/edge.conf
  1. Prometheus config prometheus.yml
scrape_configs: 
  - job_name: "srs"
    metrics_path: "/metrics"
    scrape_interval: 15s
    static_configs:
      - targets: ["host.docker.internal:9972"]
  1. Run prometheus:
docker run --rm -v `pwd`/prometheus.yml:/etc/prometheus/prometheus.yml  \
    -p 9090:9090 prom/prometheus

Replay

Please describe how to replay the bug?

Step 1: Publish to SRS

ffmpeg -stream_loop -1 -re -i doc/source.flv -c copy -f flv rtmp://localhost/live/livestream

Step 2: Stop FFmpeg.

Step 3: There is always an error

[2023-01-06 13:35:33.282][ERROR][65245][p4q038w5][4] edge push get server control message failed. err=code=1007(SocketRead)(Socket read data failed) : recv interlaced message : read basic header : basic header requires 1 bytes : read bytes : read
thread [65245][p4q038w5]: recv_message() [./src/protocol/srs_protocol_rtmp_stack.cpp:341][errno=4]
thread [65245][p4q038w5]: recv_interlaced_message() [./src/protocol/srs_protocol_rtmp_stack.cpp:792][errno=4]
thread [65245][p4q038w5]: read_basic_header() [./src/protocol/srs_protocol_rtmp_stack.cpp:887][errno=4]
thread [65245][p4q038w5]: grow() [./src/protocol/srs_protocol_stream.cpp:162][errno=4]
thread [65245][p4q038w5]: read() [./src/protocol/srs_protocol_st.cpp:566][errno=4](Interrupted system call)

Step 4: Query prometheus by:

srs_clients_errs_total

Expect

Ignore some disconnect events.
@winlinvip winlinvip added the Kubernetes For K8s, Prometheus, APM and Grafana. label Jan 6, 2023
@winlinvip winlinvip added this to the 5.0 milestone Jan 6, 2023
chundonglinlin added a commit to chundonglinlin/srs that referenced this issue Jan 6, 2023
@chundonglinlin
Exporter: Filter normal read and write errors.(ossrs#3364)
23dd860
chundonglinlin added a commit to chundonglinlin/srs that referenced this issue Jan 6, 2023
@chundonglinlin
Exporter: Filter normal read and write errors.(ossrs#3364)
e93abb6
@winlinvip winlinvip linked a pull request Jan 7, 2023 that will close this issue
Exporter: Filter normal read and write errors.(#3364) #3367
Open
@chundonglinlin
Copy link
Member

chundonglinlin commented Jan 7, 2023 •
edited by winlinvip

Will place the error code error_code in the metrics section, similar to node_exporter.

# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0

TRANS_BY_GPT3

@winlinvip winlinvip added the TransByAI Translated by AI/GPT. label Jul 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chundonglinlin chundonglinlin

Labels
Kubernetes For K8s, Prometheus, APM and Grafana. TransByAI Translated by AI/GPT.
Projects
None yet
Milestone
5.0
Development

Successfully merging a pull request may close this issue.

Exporter: Filter normal read and write errors.(#3364) chundonglinlin/srs
2 participants
@winlinvip @chundonglinlin