New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Config Support]: Whole Machine Crashing - looking for some tips #8470
Comments
There's no info provided here so there is nothing to go off of. You first need to figure out why the machine is actually freezing (is it memory issue, kernel panic, etc.) |
nothing is shown on the console. I'll check to see if there is anything in syslog. The last time i checked there was nothing - the whole machine was just frozen. |
it can happen for many different reasons, if there is no information that can be provided then there's not really much that can be done on the frigate side. There are plenty of solutions like having a log written to a file so the cause can be seen in the logs after restarting the machine. Also, you can try putting a memory limit on the frigate container |
The next steps would be to back down frigate to a bare minimum config and slowly add parts back until you can see what is causing the issue. |
Thanks - i'm following the other thread also. I also added some more debugging to Ubuntu to see if I can capture anything in the logs before the freeze. Do you have any suggestions on where to start removing the config from? us the hwaccel param a place to start? ffmpeg: |
This comment is not very helpful but I had the exact issue running the containers on kubernetes (microk8s on ubuntu). I ended up removing my coral (m.2) and switching to CPU/VAAPI detection for now, It's been a few weeks without issue. It's a long shot but could be worth trying the same to rule it out? I have not gone back to the coral as I have only a 1 camera doing detection / CPU usage is not high. |
I have a similar issue, running an i5-6500T, no external accelerator, and so far I've been able to ascertain the following:
Here's my config using three random camera feeds from the Internet that I use for debugging, currently the hardware acceleration for decoding/encoding is commented out:
I tried to grab kernel crashdump via kdump, and also tried out kernel netconsole (dmesg) logging to another server running on the same network, but neither resulted in any output, which makes me think it's a driver issue that affects the CPU itself, not even a kernel crash. Running the beta2 image in docker-compose, the beta3 image has an issue with go2rtc failing to parse the camera feed URLs. If you have any ideas for any further troubleshooting I could do, please do let me know. |
@ggidofalvy-tc After round about 12h-24h the Host is crashing when using OpenVINO. Tried different drivers in the Host, but didnt Help. Currently thinking about to buy a Coral... |
I have same issue on K3s on Debian with i3-6100U. VAAPI HW encode/decode + OpenVINO setup in config. with obj detection turned off its rock solid. if I turn on obj detection for a single object on a single camera, whole node hangs within 3 days. Those of us using the official helm chart cant update go2rtc or ffmpeg with custom versions. |
Adding onto my previous comment: Running Ubuntu 22.04, tried both the GA (5.15) and HWE (6.2) kernels, both exhibited the same crash behaviour. |
#8338 (comment) may be relevant with a couple suggestions (and other linked issue) |
@ggidofalvy-tc @madasus especially if your frigate machine is headless, I would recommend removing the often-default It's certainly suspicious that what I reported in #8338 is also using a i7-6600(U) / Skylake GPU - same generation as you both - wondering if there is a driver bug / hardware quirk that other generations don't have that the i915 driver isn't handling |
@kevin-david my host is headless so i'll give this a try. Will the debug then be written to syslog? how are you grabbing it? Can you point me in the direction of where you made this change in your linux distro? (i'm using Ubuntu). I'm glad i opened this thread as it appears this is not an isolated problem - and while not a Frigate issue but likely something that Frigate exposes due to load in the underlying hardware/software. Thanks M |
@madasus sure - I am using proxmox, so it should be similar. In my case the message never appeared in syslog, only on the physically connected screen - I guess because the machine was hung, it wasn't able to be written to syslog. this might mean you need to temporarily connect a monitor to the machine. To do what I was talking about, you'll want to change This describes it a little more: https://askubuntu.com/a/19487. Again in my case, I removed |
I gave This is what I got in dmesg:
I'll keep a look out for more messages in the netconsole destination now that I rebooted again and set the |
I probably found a solution... Running a yolov8s model since some days and currently >48h stable without any crash. Perhaps you can try this aswell? |
@Pingbo can you share your model and detector config.yml snippets? Sorry for the mild derail, I would like to see if this might be a model-specific issue, not an OpenVINO-related one. Running the beta2 branch, since beta3 has issues with go2rtc with my config. I've been trying to get yolov8n/yolov8s running on my setup based on the notebook linked in this comment: #5184 (comment) But I keep getting an error when the detector starts up:
My config.yml bits, attempting to run the yolov8n model:
(all 3 output files are mounted /config/openvino-model, I'm reusing the labelmap from the original mobileSSD model used) |
when mine freezes I managed to check the console this time and there were no messages at all being written to the console before the crash. @Pingbo can you elaborate on how to use the model you are suggesting? is this being used instead of the Coral? |
Thats how i have done it:
@madasus |
Thank you for the detail @Pingbo The only thing I would clarify for others is 1. you want to put all 3 files in the .zip file from the yolo model generation in the model folder, and 2. the files that were generated for me were yolo8n.xml, so make sure your file path is correct. Hopefully this is the fix. Edit: 2 weeks running with solid person detection using the yolov8n model on a single camera. Looks like CPU usage dropped significantly for me. Enabling it on the rest of my cameras now. |
@Pingbo Thank you for the help and the detailed instructions! I've been using yolov8n for nearly two weeks now without any crashing on beta2. I think the issue might indeed be caused by the combination of the bundled ssdlite_mobilenet_v2 model and Skylake-gen OpenVINO -- is this perhaps worth documenting somewhere? |
Wanted to chime in here, im a new frigate user as of about two weeks ago. My hardware is an i7-7700 kaby lake. I am running frigate and wyze-bridge together. Wyze bridge is correctly using Intel QSV with ffmpeg and Frigate will use it fine on ffmpeg as well. However, if i tried to use any openvino detector, it would crash the container everytime. If i set a detector as cpu (not openvino cpu), the container would start and detect fine. Today i followed these steps by @Pingbo and finally my openvino detector will start with GPU selected. My inference speed went from 45ms (cpu) to 15ms (ov gpu). The only error i could make out from the container was Anyway, the yolov8 model from the above comment seems to have resolved my issue for now. Ive been stable for a few hours (where previously i was unable to even start the containers). I will continue to monitor. (thanks @Pingbo !!) edit: i am on frigate version 0.12.1-367D724 |
@Pingbo i'm trying generate a YOLO model via the link, but i have zero to no clue what i'm doing. The scripts are giving me errors. Is it possible for you to help me? Example from first script: ImportError: cannot import name 'is_exact_shape_match' from 'pandas.core.indexers' (/usr/local/lib/python3.10/dist-packages/pandas/core/indexers/init.py) Example from second script: (with some other lines and then ending with..) WARNING: The following packages were previously imported in this runtime: By that i understand that i just have to refresh the page and try again, but it produces the same error. |
@Strux-DK its not you, you just picked a bad day. I literally just ran this yesterday to generate an 8s model to upgrade from the 8n I was using and it worked flawlessly. Today, I'm getting the same errors you are. (tried 3x) It looks like @aeozyalcin 's awesome colab may have broken? That said, since I just generated it yesterday, maybe give this one a try? |
I have Frigate 0.13 running on a Dell Optiplex 3070 Micro i3-9100t, with a single USB coral. It runs Debian Bookworm and Frigate only. I've had probably one hang per month on average, but recently I made quite a few config changes (mainly around go2rtc streams), and it now seems to crash around once a week. I'm in the process of experimenting with kernel crash dumps to see if I can catch any more info. As has already been said, this is highly unlikely to be a Frigate bug. I suspect the issue is either in Linux drivers, or (particularly in the case of cheap no-name mini PCs, or even micro form factor desktops like my Optiplex) hardware - not necessarily faulty hardware, but hardware that isn't designed to run full-tilt 24/7 on the CPU/iGPU while also supplying 1A power to a Coral (or two). I never had any problems on my Dell R220 with Xeon E3s and ECC RAM, and I don't think I've seen many examples of people reporting crashes on server-class hardware. I think Frigate is actually quite an interesting test-case of a 24/7 demanding application often running on cheap consumer gear. The only other tasks most people might assign to this gear - transcoding video or gaming - doesn't run 24/7. |
I had the same issues with a Poweredge T20 Xeon E3 1225 V3 CPU, hardware acceleration and coral usb active. Therefore I upgraded to a Dell PowerEdge T320 without hardware acceleration and with coral usb. |
Another +1 here with similar config, Skylake (i3-6100) generation CPU with OpenVINO (No Coral) in case it helps as a further reference point. The combination of mobilenet_v2 and vaapi for hwaccel causes hard system crashes within 24 hours every time. The workarounds I've found that work are:
And the things I've tried that I tried in isolation and do NOT work (Including because other issues referenced these as possible solutions):
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
Issue still exists. |
Same problem in unraid with 0.13.2-6476f8a. I have setup a syslog to try to catch the what it is happening when server gets stuck. |
Having this issue as well on unRAID... Attaching unRAID diagnostics (includes syslog and other logs/info) in case it helps track this down. Observed out of memory errors and the reaper process killing ffmpeg within the syslog. Sometimes the system runs for weeks other times it dies in a matter of days but works great otherwise. Usually, the system will hard lock and I'm unable to get any further diagnostic information but one of my plugins detected the OOM problems and this diagnostic zip was captured at that time. and full props to yayitazale for helping me get this far over on the unRAID forums. |
I had this happen on unraid and it was due to their network implementation. Switching to ipvlan fixed the issue and hasn't happened to me in months https://forums.unraid.net/topic/151441-macvlan-crashes-6126/ |
I have switched to ipvlan and I'm still having crashes. |
Confirmed I am already on ipvlan and having this issue. Fought that battle a couple years ago. |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
I can confirm that using yolov8n has most likely fixed the issue on my HP ProDesk G3, before that the GPU would crash every ~2 weeks causing the whole PC to hang. |
Switched from Cuda (Quadro RTX 4000) to Coral/Intel 12th Gen QSV for processing. Consumes about 100watts less power and generates a lot less heat but still crashes. If there's an improvement in uptime I haven't noticed. Though limiting the memory of this container to 4GBs seems to keep it in check. |
In my case, everything up to date and running with no issues for 1 month. |
I have this same thing happening, Running Ubuntu on a SFF pc, with a PCI Coral. I get maybe 3 days before it goes unresponsive. My setup is 5 unifi cameras, gortc, and QSV for decode. |
I have same problem, hard hang with no access at console or remotely. I’ve moved from RPI4 with 4GB to Mac mini (2012) with 16GB on Ubuntu both having same issue. Hangs within 2-3 days normally but sometimes a couple of times a day. No logs and doesn’t seem related to excess activity. I have 2 coral tpu usb devices but have moved one off to another server to limit this server to just frigate. Am running 4 cams. |
Describe the problem you are having
I have two docker hosts and both have a coral. I find that Frigate seems to cause the whole host to freeze completely (console is not responsive) at frequent intervals - right now I would say on average every 48 hours but its not consistent. I've moved the docker container to the other host and cleared out all the other dockers and the freeze follows Frigate.
Its likely Frigate is pushing the hosts much harder than any other docker and perhaps its finding a bug somewhere in the hardware or OS. The Devices are BeeLink devices running the latest Ubuntu.
Looking for some advice - has anyone seen this sort of behavior and identified the cause?
This has been happening for many months so it is not related to the beta Frigate or any particular Frigate (and likely this is NOT a Frigate bug)
Version
0.13 Beta 3
Frigate config file
Relevant log output
Frigate stats
No response
Operating system
Other
Install method
Docker Compose
Coral version
USB
Any other information that may be helpful
No response
The text was updated successfully, but these errors were encountered: