Creating an LXC (Linux Container) image of Frigate for use on Proxmox etc

[leccelecce]

Discussed in a couple of places, and I thought useful to have a dedicated topic. I am using Proxmox but in theory this is relevant to any other hypervisor that doesn't natively support Docker.

Given that:

1. Frigate is packaged using Docker
2. Proxmox (and similar hypervisors) have the following options to get a Docker runtime: a) install Docker in a VM, b) install Docker in LXC (specifically not recommended by Proxmox), c) install Docker directly on the Proxmox host (not appropriate for a hypervisor)
3. Unfortunately Docker-in-a-VM is the only Proxmox recommended option, but this can be slow/unreliable if you have a Coral USB due to Qemu's USB emulation
4. LXC containers on Proxmox are closer to bare-metal and avoid USB emulation
5. Frigate does not come with instructions to do a manual Linux installation and is unlikely to do so given the reasonably complex Docker multi-stage build process
6. LXC can import Docker OCI images as the root filesystem for an LXC container

...theoretically there could be a simple process to take a pre-built Frigate Docker image and convert it into an LXC template.

I have made a start on attempting this and will see how it goes. Key areas of interest I can think of are going to be architectures, hardware support, upgrading between releases.

[kloknibor]

I was thinking to start on this as well, how far did you come? What did you try?

[andorardo]

Very interested in this. I tried the Docker-in-a-VM with USB port pass-through, but hit the problem of the Coral dongle randomly disconnecting and bringing down all of Frigate.

For both Edge TPU and Quicksync the recommendation is to run as close to the metal as possible, rather than attempt a VM pass-through. Would LXC accomplish this?

[hvanoch]

I currently am succesfully running frigate inside a docker container in an lxc container.

Before I had the same issue with the coral dropping out. It is now running smooth with the coral. Also my integrated gpu is easier to pass through.

[andorardo]

Good to know!

Any idea why Proxmox recommends against docker-in-LXC?

[nicoleise]

Good to know!

Any idea why Proxmox recommends against docker-in-LXC?

To my knowledge, one major issue is the Docker images and containers disappearing when restoring a backup and often also simply when rebooting the host machine. The images and container files will remain on the system storage, but Docker somehow looses awareness that they exist.

It will typically work without issue until a host reboot/power loss or a backup/restore action is performed.

There are also performance issues due to incompatible storage drivers in some/many Proxmox and Docker combinations.

All this in addition to the general concern that is adding a layer of complication for "no reason"/benefit.

[pbvdven]

Very interested in this. I tried the Docker-in-a-VM with USB port pass-through, but hit the problem of the Coral dongle randomly disconnecting and bringing down all of Frigate.

For both Edge TPU and Quicksync the recommendation is to run as close to the metal as possible, rather than attempt a VM pass-through. Would LXC accomplish this?

I needed to move frigate away from my home assistant machine because i have some issue. Now my issues are still there but the usb coral works fine. I run docker in lxc with usb passthrough the only issue i have is that on reboot of my proxmox server the usb hub adres changes and i need to change that in the lxc config. Still looking for a fix for that.

[Dougiebabe]

lxc.mount.entry: /dev/bus/usb/004 dev/bus/usb/004 none bind,optional,create=dir

This is what I use in the conf file and it seems to work OK

[pbvdven]

@hvanoch this is what i have right now in my lxc config for the usb pass through. also the last time it changed from 004/something to /002/002 i thought that the first 3 meant the actual fysical port but probably not because i never changed port.
lxc.cgroup2.devices.allow: c 189: rwm*
lxc.mount.entry: /dev/bus/usb/002/002 dev/bus/usb/002/002 none bind,optional,create=file
and i the frigate stack i have
/dev/bus/usb:/dev/bus/usb
that works fine for me.

now i changed it to
lxc.cgroup2.devices.allow: c 189: rwm*
lxc.mount.entry: /dev/bus/usb/002 dev/bus/usb/002 none bind,optional,create=file
then i can run frigate container but the coral doesnt get fount i get this error in frigate No EdgeTPU was detected. If you do not have a Coral device yet, you must configure CPU detectors.

[NickM-27]

Make sure to run multiple times. It's common for the first try to fail as the usb device loads its internal driver

[pbvdven]

lxc.mount.entry: /dev/bus/usb/004 dev/bus/usb/004 none bind,optional,create=dir

This is what I use in the conf file and it seems to work OK

ill probably need to change =file to =dir

[pbvdven]

yes now it works as it should /dev/bus/usb/ it needed to be =dir instead of =file. that's nice thanks.

[blakeblackshear]

I would be curious if something like this would work: https://bobcares.com/blog/convert-docker-image-to-lxd/

[GrumpyMeow]

I've managed to convert the Frigate Docker container into a fully working Proxmox LXC container, without using Docker.
I have a detailed description of the process.

Here are my notes: https://github.com/GrumpyMeow/proxmox-tips/blob/main/frigate-docker-lxc.md
Things I did:

• used lxc-create to download the Frigate image from gcr.io
• Installed packages like “init” and also other networking-packages “chroot” on top of the lxc-rootfs.
• Created a (Proxmox Isabelle) template from the rootfs.
• Create a LXC container from the template
• Started the container, entered the container, modified the “/init” file to create ramdisk, set env-vars (like path)
• Created config and media folders
• Created a config file
• Stopped the container
• Configured the “lxc.init.cmd” to point to “/init”
• Started the container

[hkrob]

Hi - nice work! I followed your instructions and got it working easily.

One question, probably an obvious one, but... How do I update it in the future?

[Dvalin21]

@blakeblackshear is it possible to add this to install documents?

[m-d-brown]

I have #10759 pending that provides some of this. It provides more of a baseline and can be improved in later PRs

[shewless]

I think @GrumpyMeow 's instructions above are ideal for proxmox. You build a lxc image directly from the docker image and run frigate directly as an lxc container on proxmox. I've tested it and it works well.

[andorardo]

Nicolai, thanks for the elaborate explanation! Question: in the backup/LXC mode, how does the *user* perform an upgrade? Your explanation of how the maintainer updates is clear, but the user will have additional custom config files and media. Would the idea be to use externally mounted volumes, like with docker?

…

On Sat, May 20, 2023, 12:36 PM Blake Blackshear ***@***.***> wrote: Are you aware of any other open source projects that provide a Proxmox compatible LXC like this? — Reply to this email directly, view it on GitHub <#5448 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMFGJLEGHYNXPIRGTKHTGDXHDXILANCNFSM6AAAAAAUYE4KBM> . You are receiving this because you commented.Message ID: ***@***.*** com>

[nicoleise]

Nicolai, thanks for the elaborate explanation! Question: in the backup/LXC mode, how does the user perform an upgrade? Your explanation of how the maintainer updates is clear, but the user will have additional custom config files and media. Would the idea be to use externally mounted volumes, like with docker?

Personally, I would not care about the recordings, since they'll be gone come end of retention anyways, and it's a planned situation where I can first make sure nothing of interest is lost.

If the user is like me, then the simplest solution is simply to copy/paste/transfer the config file directly from within Frigate web UI and let the rest perish.

If not, then the user would need a separate storage volume to maintain data between backups. This is often the case, I would assume. For example, as outlined above, I don't care, but I do care that Frigate doesn't wear out my main enterprise SSDs, both for the cost and the massive inconvenience. So I have an array of HDDs dedicated to Frigates recordings, and as such would maintain data across an update.

Frigate itself runs off the SSDs to keep things snappy, and store the recordings on the HDDs. From what I've read, it seems to be a pretty common approach for users "advanced enough" to look at Proxmox to begin with. 🙂

[petervk]

Is there anything we can do to advance this? I would love an official LXC method of installing frigate.

[Dvalin21]

Um...I've been running frigate in docker on lxc in proxmox for about a month or more with no issues. Using Google Coral and AMD GPU.

[petervk]

Ok, that's what I figured. I was going to try using the Docker LXC script here: https://tteck.github.io/Proxmox/ to get the LXC setup and then install the frigate docker image in that. I do have a m.2 2030 coral that I will need to install in my new server and get working, but that should be straightforward in a LXC, right?

[Dvalin21]

Ok, that's what I figured. I was going to try using the Docker LXC script here: https://tteck.github.io/Proxmox/ to get the LXC setup and then install the frigate docker image in that. I do have a m.2 2030 coral that I will need to install in my new server and get working, but that should be straightforward in a LXC, right?

Thats exactly what I used to get started. Post script, and the docker script Also, wrote a script ( need to refined it, haven't had time) to setup the lxc with coral, and gpu once created. https://github.com/Dvalin21/ProxHeader_Nvidia_Coral and this one for enabling iommu https://github.com/Dvalin21/enable_iommu_proxmox at that point, just go in create the folders and run docker.

[Bettman66]

Attention, after the kernel update to 6.5.11-4-pve, the Coral accelerator no longer runs.

[Bettman66]

It works with the Unprivileged = 0 option, but I would like to avoid that.

[alienatedsec]

@Bettman66 It means that it has privileged access to your host's resources, and it needs to work this way for hwaccel and Coral support.

[alienatedsec]

@arevindh I am still experiencing performance issues even after the latest changes I made to the frigate config.

[Edit] The same settings are stable on the QNAP container (yes, I have another Coral installed there)

It means that I got to test the VM way as described here #1837 (comment)

[distante]

I have managed to pass iGPU for Hardware acceleration to LXC containers, the main problem is to know the correct mix of permissions. I am still looking at how to be able to pass coral.

[aburow]

I had to physically disconnect and reconnect the coral to get it recognised but still wasn't happy so I pulled and recreated the docker image. Not sure what it resolved but it worked after I took those 2 actions.

[alienatedsec]

I currently am succesfully running frigate inside a docker container in an lxc container.

Before I had the same issue with the coral dropping out. It is now running smooth with the coral. Also my integrated gpu is easier to pass through.

@hvanoch
Are you ok with sharing your setup, and how you did it? I got some performance issues with memory and CPU being maxed out after some time.

[dlasher]

Ever-so-slight bump on this issue.

Same boat, been running frigate on docker inside KVM on proxmox. Works very well, but would prefer to not run a full KVM to do this.

You can export your running docker container like so:

docker export container_name > output.tar

which leaves you with a tar file you can, in theory, turn into an LXC template, or copy over the top of an existing ubuntu LXC, but I suspect more work is needed.

Anyone closer to a full "template" that can be used?

(these are fantastic : https://github.com/tteck/Proxmox/ -- was hoping to either find one there, or help create one that lives there)

[alienatedsec]

Following my latest comment and a few months of working with the bare-metal, I decided to go the cluster route and repurpose my mini PC. I used a Coral A+E in the WLAN slot on the Ubuntu, which served me well over time. This was tested on Dell Optiplex 3050 (i5 6500) and HP Prodesk (i5 7500). It was a long journey, contributions outside of this repo, and a stubborn idea of this is going to work, which I want to share with everyone.

I hope this short guide will save you time.

In short, I replaced Ubuntu with Proxmox, joined the cluster, and followed the below to make it run the first time.

There are host and LXC parts to follow:

The host part

• Installing Coral drivers - go to the shell of the host and type the following commands

apt update
apt upgrade
apt install pve-headers-$(uname -r)
apt install proxmox-default-headers
apt install dh-dkms devscripts git
git clone  https://github.com/google/gasket-driver
cd gasket-driver
debuild -us -uc -tc -b -d
cd ..
dpkg -i gasket-dkms_1.0-18_all.deb
reboot

Once rebooted, ensure the apex_0 device is in /dev/.

ls -alh /dev/apex_0 
crw-rw---- 1 root root 120, 0 Mar 20 14:51 /dev/apex_0

The above is crucial and we need some information e.g., 120 for our cgroup setting.

I also use Intel graphics to serve decoding.

ls -alh /dev/dri/renderD128 
crw-rw---- 1 root render 226, 128 Mar 20 14:51 /dev/dri/renderD128

The same as above 226 will be used for our cgroup setting.

• Install LXC by executing the following.

bash -c "$(wget -qLO - https://github.com/tteck/Proxmox/raw/main/ct/docker.sh)"

Follow the onscreen instructions (can be default at this point), install portainer and docker-compose and ensure the unprivileged is set to No when prompted.

It will create a docker with the [ID number] e.g. 105 (docker).

Edit the config file of the LXC using nano

nano /etc/pve/lxc/105.conf

Add the following at the bottom of the file

lxc.cgroup2.devices.allow: c 226:* rwm
lxc.cgroup2.devices.allow: c 120:* rwm
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file 0, 0
lxc.mount.entry: /dev/apex_0 dev/apex_0 none bind,optional,create=file 0, 0

Reboot your LXC.

The LXC part

• Go to console of the LXC - thats the one with the ID - e.g. 105 (docker)

mkdir /opt/frigate
mkdir /opt/frigate/config
nano /opt/frigate/config/config.yml

My sample frigate config for the hardware

detectors:
  CoralPCIe:
    type: edgetpu
    device: pci:0
ffmpeg:
  hwaccel_args: 
    preset-vaapi  
    hwaccel_device - /dev/dri/renderD128

• Now we must create docker-compose.yaml to run Frigate.

cd /opt
nano docker-compose.yaml

Paste the below:

version: '3.9'

services:

  frigate:
    container_name: frigate
    privileged: true
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:0.13.2 #version at the time of writing this
    shm_size: "256mb" # update for your cameras based on the calculation
    devices:
      - /dev/apex_0
      - /dev/dri/renderD128
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /opt/frigate/config:/config:rw
     # - /media/cctv_data/config:/config:ro
     # - /media/cctv_data:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp

    environment:
      FRIGATE_RTSP_PASSWORD: "yourpassword"

docker compose up

The above is useful so you will see the output and verify if Frigate executed successfully.

[alienatedsec]

Apologies @BradleyFord I just made a small adjustment - I corrected my mistake as previously I described reinstall instead of replaced. I am no longer using Ubuntu as the OS.

In short, I replaced Ubuntu with Proxmox, joined the cluster, and followed the below to make it run the first time.

Docker LXC is using defaults, so I guess its Debian.

Debian GNU/Linux 12 docker tty1

docker login: root (automatic login)

Docker LXC provided by https://tteck.github.io/Proxmox/

[BradleyFord]

Ok, thanks for that. I might try getting it going again over the next week or so.

My current version is running well, except it is not using the Coral which is quite frusterating. Which was linked with the Python library version issues, hence really hoping to copy someone smarter than me on this topic!

[alienatedsec]

@BradleyFord please let me know if you require a USB Coral linked with your container as it will be a slightly different cgroup setting.

[BradleyFord]

@alienatedsec Ok, I managed to run through it all last night.
It seems for the unprivileged setting that can only be done in the advanced flow rather than defaults. At the moment I have unprivileged set to No.
The Coral seems to be loading perfectly this time which is great news!

For some reason none of my cameras are working, even though I directly copied my previous Frigate config file. It seems to be something with go2rtc, when I click the "Streams info" button it is now just showing the xml where previously it showed the go2rtc WebUI which I would then be able to start the fault finding process

The go2rtc cameras are configured to connect directly to the camera; with the frigate cameras using 127.0.0.1:8554. From my knowledge that still should be correct.
But the logs of go2rtc are quite boring:
2024-03-25 12:23:52.945176315 [INFO] Preparing new go2rtc config... 2024-03-25 12:23:53.438721999 [INFO] Starting go2rtc... 2024-03-25 12:23:53.503204985 12:23:53.503 INF go2rtc version 1.8.4 linux/amd64 2024-03-25 12:23:53.503739021 12:23:53.503 INF [api] listen addr=:1984 2024-03-25 12:23:53.503934449 12:23:53.503 INF [rtsp] listen addr=:8554 2024-03-25 12:23:53.504193341 12:23:53.504 INF [webrtc] listen addr=:8555 2024-03-25 12:24:02.838030037 [INFO] Starting go2rtc healthcheck service...

If there anything special I need to do for this following your steps?

[alienatedsec]

@BradleyFord
The Streams Info needs to expose a port to access the page you mentioned. Read #10316

I would need to see your config.yml to advise further. What is or what else is in the Frigate logs?

[BradleyFord]

I figured out my issue after some deeper searching, I have removed "#hwaccel_device - /dev/dri/renderD128" and it has all come to life ok. Thank you very much, my CPU is no longer pegged to 100% now that it is using the Coral finally :)

…

On Mon, 25 Mar 2024 at 18:30, alienatedsec ***@***.***> wrote: @BradleyFord <https://github.com/BradleyFord> The Streams Info needs to expose a port to access the page you mentioned. Read #10316 <#10316> I would need to see your config.yml to advise further. What is in the Frigate logs? — Reply to this email directly, view it on GitHub <#5448 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACOZ72BC2VYP3EPGYSTIF3Y2ADELAVCNFSM6AAAAAAUYE4KBOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DSMBRGI2DQ> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[alienatedsec]

"#hwaccel_device - /dev/dri/renderD128"

@BradleyFord what exactly did you remove? Is it the # at the beginning of the line or the entire line #hwaccel_device - /dev/dri/renderD128?

The # at the beginning of the line makes the setting inactive so you either reenabled it by removing # or done nothing as something else has potentially fixed it.

[BradleyFord]

I added a # in front of this line #hwaccel_device - /dev/dri/renderD128, essentially disabling it.

[aburow]

You use the /dev/dri/renderD128 for hwaccel ffmpeg for encode and decode of H264/H265:

In this section I'm using vaapi as my case I'm using the Intel GPU of a 10th gen i7:

ffmpeg:
hwaccel_args: preset-vaapi

[aburow]

Same here, no issues with the setup of docker in lxc but I had to make the container privileged to ensure everything runs smoothly.
I have the storage connected to a NAS as an SMB3 mount at container level. This made a significant improvement on the responsiveness of the system as opposed to using one of the local disks. I have a seperate VLAN MTU@9216 and interfaces on the storage pointing interfaces at 9000 for the NAS/SAN side of my systems. SMB and NFS being the preferred option so that I can access data directly via the share from another PC.

[distante]

Just to shine in for new readers.

You CAN run Frigate on an unprivileged LXC container. But will need to fix some problems with hardware configuration if you want to use Google Coral or iGPU. I had make some notes for myself but maybe it helps others:

• Passing Google Coral USB Edge TPU to an unprivileged container in proxmox
• Pass Intel iGPU to an Unprivileged LXC Container (Proxmox)

Also I moved the media folter to a mounted NAS directory to exclude them from my LXC backups.

[aburow]

Yes you absolutely can. I chose not to do it when I set things up as I didn't have something quite right with the setup. Using unprivileged container is best practice.

Thanks for the notes, I'll review to figure out what was bugging me out.

[dimatx]

There's now a Proxmox unprivileged LXC creation script for Frigate available in https://github.com/tteck/Proxmox/ with an active discussion thread: tteck/Proxmox#2711

Haven't tried it myself but others are reporting success, Will probably move from my own docker setup if I can confirm that everything works.

Nick / Blake, if it works, could be a good way to help folks get up and running with Frigate. (side note: Scrypted is currently recommending LXC as preferred install option.)

[imellor]

I've been following the guide and I have frigate working, thank you :-)
I have a Nvidia gpu, which I believe is installed OK, both in host and container.
However if I use:

environment_vars:
  LIBVA_DRIVER_NAME=nvidia

The log shows:

*************************************************************
*************************************************************
***    Your config file is not valid!                     ***
***    Please check the docs at                           ***
***    https://docs.frigate.video/configuration/index     ***
*************************************************************
*************************************************************
***    Config Validation Errors                           ***
*************************************************************
1 validation error for FrigateConfig
environment_vars
  value is not a valid dict (type=type_error.dict)
Traceback (most recent call last):
  File "/opt/frigate/frigate/app.py", line 645, in start
    self.init_config()
  File "/opt/frigate/frigate/app.py", line 112, in init_config
    user_config = FrigateConfig.parse_file(config_file)
  File "/opt/frigate/frigate/config.py", line 1346, in parse_file
    return cls.parse_obj(config)
  File "pydantic/main.py", line 526, in pydantic.main.BaseModel.parse_obj
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for FrigateConfig
environment_vars
  value is not a valid dict (type=type_error.dict)

*************************************************************
***    End Config Validation Errors                       ***
*************************************************************

I've down some googling, but cannot find what the valid arguments for LIBVA_DRIVER_NAME are?