[Detector Support]: Memory usage with openvino

[Pingbo]

Describe the problem you are having

If i active openvino my memory is increasing until the server will crash.
I already limited the frigate container to 4GB and now to 10GB. Currently the RAM usage is around 4.5GB but is still increasing...

I also enabled swap an saw that frigate.detector.ov is swaping some GB.

docker stats:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
d587b6e17194 frigate 23.03% 4.802GiB / 10GiB 48.02% 5.88GB / 302MB 1.41GB / 5.95GB 285

Version

0.12.1-367D724

Frigate config file

mqtt:
  host: 192.168.2.210
  port: 1883
  user: x
  password: x
  topic_prefix: frigate

go2rtc:
  webrtc:
    candidates:
      - 192.168.1.211:8555
      - stun:8555
  rtsp:
    username: "x"
    password: "x"
#  ffmpeg:
#    rtsp: "-fflags nobuffer -flags low_delay -stimeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_transport tcp -i {input}"
  streams:
    Wohnzimmer:
      - rtsp://x:[email protected]:554/h264Preview_01_main
#      - "ffmpeg:Wohnzimmer#audio=opus"
    Wohnzimmer_sub:
      - rtsp://x:[email protected]:554/h264Preview_01_sub
#      - "ffmpeg:Wohnzimmer#audio=opus"
    FeliaCam:
      - rtsp://x:[email protected]:554/ch0_0.h264
#      - "ffmpeg:FeliaCam#audio=opus"
    ElinaCam:
      - rtsp://x:[email protected]:554/ch0_0.h264
#      - "ffmpeg:ElinaCam#audio=opus"
    Buro:
      - rtsp://x:[email protected]:554/stream=1
#      - "ffmpeg:ElinaCam#audio=opus"

ffmpeg:
   hwaccel_args: preset-vaapi
#  hwaccel_args: preset-intel-qsv-h264

cameras:
  Wohnzimmer:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/Wohnzimmer
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/Wohnzimmer_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    live:
      height: 1440
    detect:
      enabled: true
      fps: 5
    objects:
      track:
        - person
        - dog
    snapshots:
      enabled: True
      timestamp: True
      bounding_box: true
      retain:
        default: 3
  FeliaCam:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/FeliaCam
          input_args: preset-rtsp-restream
          roles:
            - record
            - detect
    live:
      height: 720
    detect:
      enabled: true
      width: 1280
      height: 720
      fps: 5
    motion:
      mask:
        - 0,342,0,0,733,0
        - 213,379,393,288,343,192,0,228,0,720,299,720
    objects:
      track:
        - person
    snapshots:
      enabled: True
      timestamp: True
      bounding_box: true
      retain:
        default: 3

  ElinaCam:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/ElinaCam
          input_args: preset-rtsp-restream
          roles:
            - record
            - detect
    live:
      height: 720
    detect:
      enabled: true
      width: 1280
      height: 720
      fps: 5
    motion:
      mask:
        - 772,0,0,389,0,0
    objects:
      track:
        - person
    snapshots:
      enabled: True
      timestamp: True
      bounding_box: true
      retain:
        default: 3

  Buro:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/Buro
          input_args: preset-rtsp-restream
          roles:
            - detect
    detect:
      fps: 5

detect:
  enabled: False

record:
  enabled: True
  retain:
    days: 1
#   events:
#     max_seconds: 300
#     pre_capture: 5
#     post_capture: 5
#     retain:
#       default: 3

detectors:
#  cpu1:
#    type: cpu
#  cpu2:
#    type: cpu
  ov:
    type: openvino
    device: AUTO
    model:
      path: /openvino-model/ssdlite_mobilenet_v2.xml

model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt


# # Optional: Motion configuration
# # NOTE: Can be overridden at the camera level
# motion:
#   # Optional: The threshold passed to cv2.threshold to determine if a pixel is different enough to be counted as motion. (default: shown below)
#   # Increasing this value will make motion detection less sensitive and decreasing it will make motion detection more sensitive.
#   # The value should be between 1 and 255.
#   threshold: 25
#   # Optional: Minimum size in pixels in the resized motion image that counts as motion (default: ~0.17% of the motion frame area)
#   # Increasing this value will prevent smaller areas of motion from being detected. Decreasing will make motion detection more sensitive to smaller
#   # moving objects.
#   contour_area: 100
#   # Optional: Alpha value passed to cv2.accumulateWeighted when averaging the motion delta across multiple frames (default: shown below)
#   # Higher values mean the current frame impacts the delta a lot, and a single raindrop may register as motion.
#   # Too low and a fast moving person wont be detected as motion.
#   delta_alpha: 0.2
#   # Optional: Alpha value passed to cv2.accumulateWeighted when averaging frames to determine the background (default: shown below)
#   # Higher values mean the current frame impacts the average a lot, and a new object will be averaged into the background faster.
#   # Low values will cause things like moving shadows to be detected as motion for longer.
#   # https://www.geeksforgeeks.org/background-subtraction-in-an-image-using-concept-of-running-average/
#   frame_alpha: 0.2
#   # Optional: Height of the resized motion frame  (default: 1/6th of the original frame height, but no less than 180)
#   # This operates as an efficient blur alternative. Higher values will result in more granular motion detection at the expense of higher CPU usage.
#   # Lower values result in less CPU, but small changes may not register as motion.
#   frame_height: 180

live:
  # Optional: Set the encode quality of the live stream (default: shown below)
  # 1 is the highest quality, and 31 is the lowest. Lower quality feeds utilize less CPU resources.
  quality: 8

timestamp_style:
  # Optional: Position of the timestamp (default: shown below)
  #           "tl" (top left), "tr" (top right), "bl" (bottom left), "br" (bottom right)
  position: "tl"
  # Optional: Format specifier conform to the Python package "datetime" (default: shown below)
  #           Additional Examples:
  #             german: "%d.%m.%Y %H:%M:%S"
  format: "%d.%m.%Y %H:%M:%S"
  # Optional: Color of font
  color:
    # All Required when color is specified (default: shown below)
    red: 255
    green: 255
    blue: 255
  # Optional: Line thickness of font (default: shown below)
  thickness: 2
  # Optional: Effect of lettering (default: shown below)
  #           None (No effect),
  #           "solid" (solid background in inverse color of font)
  #           "shadow" (shadow for font)
  effect: shadow

docker-compose file or Docker CLI command

version: '3.9'
services:
  frigate:
    container_name: frigate
    privileged: false # this may not be necessary for all setups
    restart: always
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "128mb" # update for your cameras based on calculation above
#    network_mode: host
    devices:
    #  - /dev/bus/usb:/dev/bus/usb # passes the USB Coral, needs to be modified for other versions
    #  - /dev/apex_0:/dev/apex_0 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
      - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /mnt/user/appdata/frigate/config.yml:/config/config.yml:ro
      - /mnt/cache/frigate/media:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "10000:5000"
      - "1935:1935" # RTMP feeds
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      FRIGATE_RTSP_PASSWORD: "x"
#    cap_add:
#      - CAP_PERFMON
    deploy:
      resources:
        limits:
          memory: 10G

Relevant log output

nothing special

Operating system

UNRAID

Install method

Docker Compose

Coral version

Other

Any other information that may be helpful

No response

[NickM-27]

What version of unraid are you running?

[Pingbo]

Unraid Version is: 6.12.4
Memory usage now:

Docker stats:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
d587b6e17194 frigate 38.67% 5.312GiB / 10GiB 53.12% 7.66GB / 364MB 1.49GB / 7.82GB 285

[NickM-27]

and what CPU is this?

[Pingbo]

Intel(R) Core(TM) i3-6100 CPU @ 3.70GHz

[NickM-27]

Openvino was updated in 0.13 so it may be better if vaapi doesn't fix it. Otherwise I'm not sure what we can do as this could be an issue with the driver.

[Pingbo]

Will try 0.13 tomorrow :) thanks for the hint.
Meanwhile i disabled all detectors in the config aswell as commented the detectors Block.

It seems the issue still persists but is much slower now.

3.4GB in round about 4 hours

Also saw this issue, which Sounds like its the same behavior:
#8463

I also can confirm this was not happen before 0.12., If this will help

[Pingbo]

I also found something intressting...

Current Setup: VAAPI HW Accel, recording and no detectors

When i Check docker stats its reporting 6GB Out of 10GB are used

When checking htop only 3.75GB are used on the whole Host. But some minutes before the Crash i have oom Messages in my syslog from ffmpeg. (Frigate is the only App which is using ffmpeg on the Server)

Edit:
Finally tested without HW Accel... Same behaviour:
After round about 1 hour:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
30b48d5c420a duplicati 0.03% 825MiB / 15.52GiB 5.19% 21.2kB / 9.18kB 1.21GB / 3.65GB 29
d587b6e17194 frigate 54.69% 3.538GiB / 10GiB 35.38% 1.89GB / 57.7MB 467MB / 1.8GB 260

Currently everthing testet with 0.12.1

Edit 2:
Tested with 0.13.0-beta4 same stuff
Probably the same issue:
#8470

[NickM-27]

you'll need to see what process is using the memory inside frigate

[Pingbo]

thats so weird....

Inside frigate:
root@1ba1471620fc:/opt/frigate# ps auxk-%mem
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 108 1.2 1.5 2744924 245272 ? Ssl 12:18 2:13 python3 -u -m frigate
root 320 1.8 0.7 965612 128292 ? Sl 12:18 3:11 frigate.recording_manager
root 354 0.8 0.7 1166948 120444 ? Sl 12:18 1:24 frigate.process:Wohnzimmer
root 357 0.8 0.7 1166948 120444 ? Sl 12:18 1:29 frigate.process:ElinaCam
root 356 0.9 0.7 1166948 120380 ? Sl 12:18 1:33 frigate.process:FeliaCam
root 359 0.7 0.7 1166948 120380 ? Sl 12:18 1:21 frigate.process:Buro
root 365 1.2 0.7 1181508 119864 ? Sl 12:18 2:04 frigate.capture:FeliaCam
root 362 0.4 0.7 1166984 118684 ? Sl 12:18 0:47 frigate.capture:Wohnzimmer
root 367 1.2 0.7 1181508 118396 ? Sl 12:18 2:05 frigate.capture:ElinaCam
root 371 0.4 0.7 1170368 116808 ? Sl 12:18 0:45 frigate.capture:Buro
root 338 0.8 0.7 1098624 114876 ? Sl 12:18 1:23 frigate.output
root 331 0.0 0.6 1034192 113684 ? Sl 12:18 0:00 frigate.detector.cpu
root 302 0.0 0.6 688668 104996 ? S 12:18 0:00 frigate.logger

docker stats:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
1ba1471620fc frigate 55.48% 5.28GiB / 10GiB 52.80% 8.47GB / 284MB 684MB / 7.85GB 270

why does docker have here 5.3 GB RAM Usage and 270 PIDS???
Inside of the container there are one a hand full:
root@1ba1471620fc:/opt/frigate# ps auxk-%mem | wc -l
56

[NickM-27]

Docker stats may be wrong, nothing inside frigate looks to be using much memory

[Pingbo]

But why ist ffmpeg crashing with OOM and shortly afterwards the whole Host?

If frigate is not running, the Host ist running stable without freezing or anything else :/

[NickM-27]

We'd need to know what ffmpeg process is using high memory.

I'd suggest setting a memory limit in docker so you can catch it and debug

[Pingbo]

Already limited it to 10 GB currently, what do you mean with catching the process?

[NickM-27]

if frigate is limited to 10GB of RAM and your system has more I don't see how it would be taking the whole host down. What I mean is after you see memory usage is up get a shell in the frigate container and figure out what process (with the command) is using the memory

[Pingbo]

Thats a good question, If not the question.
After reaching almost the Limit in docker stats it will take some time and than it throws the OOM in syslog and than will freeze.

Why this ist happening and how this can happen is completely unclear to me.

But its the same behaviour every crash/freeze.

The Host has 16GB RAM. If i limit frigate to 2 GB the Host will freeze multiple times a day, at 4 only one freeze per day and so on.

[NickM-27]

Then I'd suggest following #8470 (comment)

[Pingbo]

testing with this config now:

mqtt:
  host: 192.168.2.210
  port: 1883
  user: MQTT
  password: x
  topic_prefix: frigate

go2rtc:
  webrtc:
    candidates:
      - 192.168.1.211:8555
      - stun:8555
  rtsp:
    username: "x"
    password: "x"
  streams:
    Wohnzimmer:
      - rtsp://x:[email protected]:554/h264Preview_01_main

    Wohnzimmer_sub:
      - rtsp://x:[email protected]:554/h264Preview_01_sub

    FeliaCam:
      - rtsp://x:[email protected]:554/ch0_0.h264

    ElinaCam:
      - rtsp://x:[email protected]:554/ch0_0.h264
    Buro:
      - rtsp://x:[email protected]:554/stream=1

cameras:
  Wohnzimmer:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/Wohnzimmer
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/Wohnzimmer_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    live:
      height: 1440

  FeliaCam:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/FeliaCam
          input_args: preset-rtsp-restream
          roles:
            - record
            - detect
    live:
      height: 720

  ElinaCam:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/ElinaCam
          input_args: preset-rtsp-restream
          roles:
            - record
            - detect
    live:
      height: 720

  Buro:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/Buro
          input_args: preset-rtsp-restream
          roles:
            - detect
    detect:
      fps: 5

detect:
  enabled: false

record:
  # enabled: true
  enabled: false
  retain:
    days: 1

If the issue still persists i will remove the go2rtc block. Thats still possible in 0.13.0, right?

Removed everything with detect, hw accel, recording and so on

Edit:
Looks good so far:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
1ba1471620fc frigate 36.21% 774.9MiB / 10GiB 7.57% 4.14GB / 225MB 2.35GB / 385kB 268

I assume it could be recording...
Will active HW Accel now and Test it

Edit 2:
With VAAPI (done stable 1GB)
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
1ba1471620fc frigate 24.21% 1.062GiB / 10GiB 10.62% 2.44GB / 194MB 2.55GB / 4.01MB 266

With QSV (done stable 1GB)
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
1ba1471620fc frigate 32.36% 1.039GiB / 10GiB 10.39% 6.56GB / 343MB 1.06GB / 10.3MB 291

With HW Accel (VAAPI) and OpenVINO: (stable 1.2GB):
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
1ba1471620fc frigate 37.13% 1.189GiB / 10GiB 11.89% 1.83GB / 104MB 989MB / 23.3MB 291

With HW Accel, OpenVINO and Recording (as expected, memory is increasing)
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
1ba1471620fc frigate 39.07% 3.371GiB / 10GiB 33.71% 2.97GB / 96.4MB 190MB / 2.75GB 296

[Pingbo]

Testing Results:

Without anything: (stable 779MB)

CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
1ba1471620fc frigate 36.21% 774.9MiB / 10GiB 7.57% 4.14GB / 225MB 2.35GB / 385kB 268

With VAAPI (done stable 1GB)

CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
1ba1471620fc frigate 24.21% 1.062GiB / 10GiB 10.62% 2.44GB / 194MB 2.55GB / 4.01MB 266

With QSV (done stable 1GB)

CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
1ba1471620fc frigate 32.36% 1.039GiB / 10GiB 10.39% 6.56GB / 343MB 1.06GB / 10.3MB 291

With HW Accel (VAAPI) and OpenVINO: (stable 1.2GB):

CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
1ba1471620fc frigate 37.13% 1.189GiB / 10GiB 11.89% 1.83GB / 104MB 989MB / 23.3MB 291

With HW Accel, OpenVINO and Recording (as expected, memory is increasing)

CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
1ba1471620fc frigate 39.07% 3.371GiB / 10GiB 33.71% 2.97GB / 96.4MB 190MB / 2.75GB 296

[Pingbo]

Testing Results:

Without anything: (stable 779MB)

CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
1ba1471620fc frigate 36.21% 774.9MiB / 10GiB 7.57% 4.14GB / 225MB 2.35GB / 385kB 268

With VAAPI (done stable 1GB)

CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
1ba1471620fc frigate 24.21% 1.062GiB / 10GiB 10.62% 2.44GB / 194MB 2.55GB / 4.01MB 266

With QSV (done stable 1GB)

CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
1ba1471620fc frigate 32.36% 1.039GiB / 10GiB 10.39% 6.56GB / 343MB 1.06GB / 10.3MB 291

With HW Accel (VAAPI) and OpenVINO: (stable 1.2GB):

CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
1ba1471620fc frigate 37.13% 1.189GiB / 10GiB 11.89% 1.83GB / 104MB 989MB / 23.3MB 291

With HW Accel, OpenVINO and Recording (as expected, memory is increasing)

CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
1ba1471620fc frigate 39.07% 3.371GiB / 10GiB 33.71% 2.97GB / 96.4MB 190MB / 2.75GB 296

@NickM-27:
It seems the issue is related to recording.

The issue persists since 12.X not sure if 12.0 or 12.1 as i updated from 11.X to 12.1 directly.
As far as i know and i read it in the Changelogs in 12.0 there where some changes how recording is handled. Perhaps there is something what will cause this issue.

[NickM-27]

How much memory is used by the recordings in /tmp/cache? This doesn't seem unusual especially if your recordings are slower to be moved to the host storage

[Pingbo]

How much memory is used by the recordings in /tmp/cache? This doesn't seem unusual especially if your recordings are slower to be moved to the host storage

Wasn't able to check yesterday, sorry. But as stated in the Documentation i only have a 1GB /tmp/cache as tmpfs and Host Storage is a SATA SSD so it should be fast enought in my opinion. But can test it and let you know the results.

Meanwhile my Host crashed again, without recording, but with OV, HW Accel and Snapshots. Therefore i will test again anything and let it run some days to see if its stable or not. Probably as you already statet it's just a docker stats issue and it not directly involved into the crash.

[NickM-27]

Users have found in the past that updating the kernel and intel drivers on the host fixed crashes

[Pingbo]

Users have found in the past that updating the kernel and intel drivers on the host fixed crashes

So after 2 Days without Recording and OV everything running stable, will activate Recording today and let it run some days.
Are there any informations which kernels might be "broken"?

Will check if there is a possibilty to update the intel drivers in unraid. But HW Accel is also using the GPU and this setup is running stable. Am i right, that this shouldn't be the case if there are driver issues on the host?

[NickM-27]

Not necessarily, OpenVINO and hwaccel access different parts of the GPU and use them in different ways, hwaccel working while OpenVINO doesn't is not uncommon.

Here are a couple references to similar issues where driver or kernel fixed:

• [HW Accel Support]: Random out of memory errors #7934 (comment)
• [Support]: System crashes #5799 (comment)

[Pingbo]

That makes sense!
Anyway tried another gpu driver on the host: https://forums.unraid.net/topic/108650-plugin-intel-gvt-g/
Without success, do you think it might work to change the default driver within frigate? As documented here:
https://docs.frigate.video/configuration/hardware_acceleration/

But its regarding HW accel so i'm not sure if it also counts for openvino...

Changing kernel will be a little bit challenging in unraid, so this is not a real option.
I currently thinking about to buy a USB Coral, but already read that there is also an issue open, where the whole host is crashing. So i'm a little bit scared.

[NickM-27]

The same driver is always used for openvino. I've used a coral in unraid for multiple years without any issues

[Pingbo]

@NickM-27
Is it possible that the issue could be related to the "ssdlite_mobilenet_v2" model ????
Running since two days the following config with yolov8s model and there is no crash:

detectors:
  ov:
    type: openvino
    device: GPU #AUTO
    model:
      # path: /openvino-model/ssdlite_mobilenet_v2.xml
      path: /config/yolov8s/yolov8s.xml

model:
#  width: 300
#  height: 300
 width: 416
 height: 416
 input_tensor: nchw  # nhwc
 input_pixel_format: bgr
 model_type: yolov8
 labelmap_path: /config/coco_80cl.txt  #/openvino-model/coco_91cl_bkgr.txt

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.