Dual TPU M.2 Coral Accelerator

[jayburkard]

Can't seem to find the USB edge accelerator in stock anywhere. Saw some unraid guys had success using the mini pcie module. Anything conceptually wrong with using the new dual TPU unit with an M.2 -> pcie adapter and passing it all through?

[hudsonsecurity]

Doesn't work with most setups to get dual tpu. Stock with the single. Read the github issue open for dual tpu on Google corals page. My opinion you're better off getting a dedicated device with tpu integration.

[didi767]

So you think that even if I switch to Ubuntu instead of the dedicated vmdk I would still be having issues with the pass through?
I have like 5 vms currently on ESXi and just thinking about transition to proxmox gives me a headache :(
But then again I really want to utilize this dual TPU for frigate

[digiblur]

I don't have the dual anymore, abandoned it as I haven't needed to jump to two yet.

[didi767]

Yeah, I think I jumped too high as well.
So there is no way I can pass it through on ESXi with Ubuntu 20 server?
Have to go with the proxmox for that?

[felipecrs]

I bought the dual TPU, it boots correctly and if I set like:

detectors:
  coral1:
    type: edgetpu
    device: pci:0

Frigate seems to use it too:

But I have no clue if the two TPUs are being used, or how I would configure Frigate to use both. Any clue on how to configure Frigate to make use of both?

I have a BlitzWolf BW-MPC1 by the way. Don't buy one, because NVME SSDs are not supported and the fan is too noisy, but it at least seems to work with the Coral M.2 card (I replaced the Wi-Fi card with it).

[tim-lukacik]

It's only using one TPU in the screenshot you are showing there. For the second TPU it would show up as another independent device like "pci:1" and you would add that entry to frigate as well.

[markmghali]

Following this as I just bought the dual tpu as it is the only thing in stock. I am on unraid, hoping there is a soltuon?

[rlrl]

bought the usb & dual m.2 from arrow about a month ago. their website says 359 parts available for usb (was over 1300 when i ordered)

https://www.arrow.com/en/products/g950-06809-01/google-corporation

[richardkuw]

I have the m.2 dual TPU working through Proxmox (since that was the only TPU actually available). I spent a couple of days trying to get pcie passthrough working on Frigate with esxi 7.0. There are issues with MSI-X interrupts that I couldn't resolve so I migrated my server and all my VMs to Proxmox (after running esxi for 12 years!). Proxmox is working fine passing the dual TPU through to a KVM virtual machine. I'm using a Fenvi wifi adapter from Amazon - note there is more than one, the one that works is the 'M.2 NGFF A+E key to PCI Express adapter'. Of course only one TPU is recognized.

I've seen some suggestions to run Frigate on an LXC container in Proxmox but in order to run Docker (a container inside a container) you need to disable ALL of the security features. Docker runs as root. You need to run the container in privileged mode with apparmor disabled. So Docker root is the same as Proxmox root. It is a hideously insecure setup. I tried it briefly but went with a KVM virtual machine and followed the instructions for passthrough. Performance is OK, inference speed is around 9.

I tend to document things so if anyone wants my notes on migrating from esxi to Proxmox and setting up passthrough let me know.

[didi767]

I finally received my dual TPU adapter and I'm happy to report it's working great! It took over a month to get here from Russia, and shipped just before the war started. Not sure what impact the war will have on shipments now but it appears the seller is working on getting more distribution channels up.

@tim-lukacik that's incredible! Could you please where did you order this from?

Thanks!

[tim-lukacik]

@didi767

https://github.com/magic-blue-smoke/Dual-Edge-TPU-Adapter

[tim-lukacik]

@felipecrs

my output

lspci | grep Edge
05:00.0 System peripheral: Global Unichip Corp. Coral Edge TPU
06:00.0 System peripheral: Global Unichip Corp. Coral Edge TPU

[felipecrs]

Thanks a lot, @tim-lukacik. That's the easiest way to confirm that my system only recognizes one of the two TPUs.

By the way, HassOS 8.0 mentions:

Google Coral support is now using Google’s latest driver. This enables additional Coral device support such as PCI Dual Edge TPU.

https://www.home-assistant.io/blog/2022/05/16/home-assistant-os-release-8/#other-changes

[didi767]

@felipecrs but I think we still have to have a 2 lane m.2 or this adapter that splits the pci into two lanes in order to have both TPU, regardless of the driver update.

[Yonny24]

Also looking at alternatives for the usb coral.

Is it easier to set up with the standard M.2 accelerator and/or single mini pcie accelerator?
Do they have the same performance as the coral usb?

I have no issues wiping proxmox and installing generic linux and frigate.
Looking at some mini pc's with m.2 expansion slots or mini pcie slots.

Thanks

[richardkuw]

I suggest you find a supplier first. I went with the dual m.2 TPU because it was actually available. Now I don't see stock of any of them (at Mouser). None are expected until March.

I don't think m.2 vs. mini pcie should make any difference as long as you have the appropriate adapter for your motherboard.

[tmjpugh]

Is it easier to set up with the standard M.2 accelerator and/or single mini pcie accelerator? Do they have the same performance as the coral usb?

I’m my experience PCIe was slightly faster

My server not have USB C so I used PCIe USB port card to add support. I lost connection with usb coral at times. PCIe coral was more stable in this regard as well

single and dual coral setup was same. For me only single shows up with dual coral as well. Hoping to find working adapter in future

[rchiileea]

I could not get hte dual tpu to work, when using the config setup like this

detectors:
coral1:
type: edgetpu
device: pci:0
coral2:
type: edgetpu
device: pci:1

frigate would not load properly and terminate so gave up and ran with just the 1, then I thought I would try this

detectors:
coral1:
type: edgetpu
device: pci:1
coral2:
type: edgetpu
device: pci:0

and it worked lol, go figure

[2023-01-28 18:10:42] frigate.edgetpu INFO : Attempting to load TPU as pci:0
[2023-01-28 18:10:42] ws4py INFO : Using epoll
[2023-01-28 18:10:42] frigate.edgetpu INFO : Attempting to load TPU as pci:1
[2023-01-28 18:10:42] frigate.app INFO : Output process started: 236
[2023-01-28 18:10:42] frigate.edgetpu INFO : TPU found
[2023-01-28 18:10:42] frigate.edgetpu INFO : TPU found

[Genieplumb]

this is the card I have on order (has a fan mount) https://www.makerfabs.com/dual-edge-tpu-adapter.html

[NickM-27]

If you use it in a spot where the motherboard has a heat sink then it shouldn't be an issue. I have one of each of these (1 PCIe with heatsink / fan and 1 m.2 with motherboard m.2 heatsink on it) and both are roughly the same temp

[digiblur]

Haven't had issues with the dual overheating either but I do recall documents from Google saying that some cooling could be required.

[rchiileea]

I actuall just bought a set of 20mm heatsinks with some 2mm thick thermal adheasive tape and stuck than on the top of the whol device its fine.

[FrigateEnthusiast]

May sound "hacky" but I've have put small 15mm axial fans (like used on 3D printers extruders) inside cases, directly blowing on components that needed extra cooling help such as a coral pice devs, and were not in high air flow areas. Easy to power from a MB fan connector or tap into a the 5v or 12v power from a sata drive connector from the PS.

[Saboral]

Got my dual TPU working used this adapter:
https://www.makerfabs.com/dual-edge-tpu-adapter.html

Had to blacklist the apex and gasket drivers in modprobe.d on Proxmox, update initramf, reboot, and manually add the PCI passthru to the VM using the shell command. Once that was done I had zero issue passing through to my VM.

Once I went in the VM and added the TPU Runtime and Pycoral I was in business. Currently getting a 11.09 inference speed, but I'm still tweaking my config.

I have a USB Coral as well, but I found it was having a lot of connectivity issues with Proxmox passthru. I'm convinced the USB on the Dell r720 I use wasn't supplying sufficient power for the TPU under load.

[jlarnal]

At this point in time (Q4 2023), I end up wondering if the Google Coral team ever had a brainstorming session about the M.2 E-key choice for the dual TPU module. There's so few end products (other than the Frigate NVR) that offer enough (populated) lanes on their M.2 expansion slots !
And all the industrial SFF motherboards that do (have those lanes), cost a leg. Plus, at their price range, why would an engineer bother with int8 inferences anyway ? They'd go with FPU-able NPUs, as most of these board come with NPUs integrated with the onboard SOC.

Really, the Coral M.2 dual edge TPU module is a head-scratcher to me. No compatibility whatsoever with the run-of-the-mill iot sbcs the Coral is aimed to supplement ( in terms of power figures ). 🤔

I'm weeping over the fact that I won't ever use this nifty little module. 😭

[rchiileea]

they done it like this as its the nature of the beast, the product and uses for need of a dual tpu are mainly industrial with the end user market probably representing about 1% of sales.

[jlarnal]

Yes, that was my point : any motherboard or SBC that would offer a fully-complemented M.2 Key-E interface are industrial products (in the $3K price range, obviously, it's a rugged class of appliances ). And that's not a bad thing, imho.

But that's where the rubber meet the road : most of these SBCs already have NPUs... So the Coral would be a bit... redundant ?
Unless the point is to use the energy-efficient Coral rather than the onboard, often power-hungry NPU ? In an industrial application ? Meh ? Too much discrepancies in this line of toughs.

[linuxdep]

Hi, it is possible to add the M.2 board to WIFI M.2 connector on HP ProDek 600 G4?

[rchiileea]

it will but if its the dual version, you wont make much use of it, due to the nature of the way it needs the bandwidth for dual tpu, your best bet would be if dual is to purchase the m.2 adaptor for it and use it in m.2 port and then use a normal sata ssd for the main drive.

[linuxdep]

Thanks, I'm not that deep into the subject, but a TPU is certainly enough for me for now. The two M.2 slots are fitted with SSDs, which I can't and don't want to remove, but if the slot for the WIFI card also works, I think it's better than USB, don't you?

[noelhibbard]

I am using the Dual Edge in Frigate with no issues other than my adapter (made for an M.2 WiFi NIC) is only giving me access to one TPU. The key to getting the Dual Edge to work is by getting the right adapter. This $30 adapter will give you access to both TPUs. https://www.makerfabs.com/dual-edge-tpu-adapter.html

[dataslayermedia]

The Dual Edge TPU is working well for me with my Zima Blade and Zima Board...

I'm able to access both TPU's from the Coral Dual PCIe accelerator using the Maker Fab Dual TPU Low Profile adapter on the Zima Blade and Zima Board.

Running lspci -nn | grep 089a and ls /dev/apex_*

Operating System: Debian 12

Also works fine with Frigate just needed to pass both apex devices to the docker container.

docker run -d \
  --name frigate \
  --restart=unless-stopped \
  --mount type=tmpfs,target=/tmp/cache,tmpfs-size=1000000000 \
  --device /dev/apex_0:/dev/apex_0 \
  --device /dev/apex_1:/dev/apex_1 \
  --shm-size=512m \
  -v /frigate/storage:/media/frigate \
  -v /frigate/config.yml:/config/config.yml \
  -v /etc/localtime:/etc/localtime:ro \
  -p 5000:5000 \
  -p 8555:8555/tcp \
  -p 8555:8555/udp \
  ghcr.io/blakeblackshear/frigate:stable

config.yml

detectors:
  coral1:
    type: edgetpu
    device: pci:0
  coral2:
    type: edgetpu
    device: pci:1

Full Tutorial Blog Post

YouTube Tutorial

[didi767]

Hi, I have the adapter but still only one TPU works. (it doesn't add the name 089a to each of the TPUs, does it matter?)
output for the commands:
.
sudo lspci:
00:10.0 System peripheral: Global Unichip Corp. Coral Edge TPU
00:11.0 System peripheral: Global Unichip Corp. Coral Edge TPU

user@frigate-242:~$ sudo ls /dev/apex_*
/dev/apex_0 /dev/apex_1

frigate only recognize pci0

detectors:
coral_pci1:
type: edgetpu
device: pci:0

coral_pci2:

type: edgetpu

device: pci:1