OrangePi5Plus32G cannot bind RTL8125 2.5GbE to vfio-pci

[rnovikov]

root@ubuntu:/sys/bus/pci/drivers/vfio-pci# echo 0004:41:00.0 > bind
bash: echo: write error: No such device

Device OrangePi5Plus32G
Ubuntu 24.04

[Joshua-Riek]

Can you modprobe vfio-pci?

[rnovikov]

I had done it,before trying to bind. vfio-pci probbed successifull. Do you need some additional traces?
FYI: i am trying to build and run dpdk and spdk application on your ubuntu. (Thank you very much for port on orangepi5plus!)

[Joshua-Riek]

Hmmm, I'm not very familiar with vfio-pci, I will need to do some research.

But have you confirmed that the device "0004:41:00.0" exists? I assume this can be confirmed by the output of lspci.

[rnovikov]

MODINFO
name: vfio_pci
filename: (builtin)
description: VFIO PCI - User Level meta-driver
author: Alex Williamson [email protected]
license: GPL v2
file: drivers/vfio/pci/vfio-pci
parm: ids:Initial PCI IDs to add to the vfio driver, format is "vendor:device[:subvendor[:subdevice[:class[:class_mask]]]]" and multiple comma separated entries can be specified (string)
parm: nointxmask:Disable support for PCI 2.3 style INTx masking. If this resolves problems for specific devices, report lspci -vvvxxx to [email protected] so the device can be fixed automatically via the broken_intx_masking flag. (bool)
parm: disable_idle_d3:Disable using the PCI D3 low power state for idle, unused devices (bool)
parm: disable_denylist:Disable use of device denylist. Disabling the denylist allows binding to devices with known errata that may lead to exploitable stability or security issues when accessed by untrusted users. (bool)

BINDING PROCESS
UNBIND from driver
cd /sys/bus/pci/drivers/r8169
ls
0003:31:00.0 0004:41:00.0 bind module new_id remove_id uevent unbind
echo 0004:41:00.0 > unbind
ls
0003:31:00.0 bind module new_id remove_id uevent unbind
cd ../vfio-pci/
ls
bind module new_id remove_id uevent unbind
echo 0004:41:00.0 > bind
echo: write error: No such device
dpdk-devbind.py out
Other Network devices
0004:41:00.0 'RTL8125 2.5GbE Controller 8125' unused=r8169,vfio-pci

DMESG BOOT ERRORS (if it is needed)

9.869775] Goodix-TS 7-0014: supply AVDD28 not found, using dummy regulator
[ 9.871393] Goodix-TS 7-0014: Looking up VDDIO-supply from device tree
[ 9.871401] Goodix-TS 7-0014: Looking up VDDIO-supply property in node /i2c@fec90000/touchscreen@14 failed
[ 9.871434] Goodix-TS 7-0014: supply VDDIO not found, using dummy regulator
[ 9.891880] pwm-fan pwm-fan: Looking up fan-supply from device tree
[ 9.891887] pwm-fan pwm-fan: Looking up fan-supply property in node /pwm-fan failed
[ 9.964599] Goodix-TS 7-0014: Error reading 1 bytes from 0x8140: -6
[ 9.991643] Goodix-TS 7-0014: Error reading 1 bytes from 0x8140: -6
[ 10.016224] [BT_RFKILL]: bt shut off power
[ 10.017912] Goodix-TS 7-0014: I2C communication failure: -6
[ 11.347913] RTL8226B_RTL8221B 2.5Gbps PHY r8169-3-3100:00: attached PHY driver (mii_bus:phy_addr=r8169-3-3100:00, irq=MAC)
[ 11.538013] r8169 0003:31:00.0 enP3p49s0: Link is Down
[ 11.554575] rk_pcie_establish_link: 228 callbacks suppressed
[ 11.554583] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[ 11.581246] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[ 11.607843] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[ 11.634506] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[ 11.661173] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[ 11.687839] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[ 11.714506] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[ 11.741173] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[ 11.767836] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[ 11.794506] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[ 13.261172] rk-pcie fe170000.pcie: PCIe Link Fail, LTSSM is 0x3, hw_retries=1
[ 14.324508] rk-pcie fe170000.pcie: failed to initialize host

[rnovikov]

@Joshua-Riek If you have time - we can have a debug meeting in google meet with share screen or i can give you a ssh access directly to device. (I am not expecting that we will fixed it, but i am sure we will have fun )

[Joshua-Riek]

From my understanding you need to unbind the pcie network interface first as it's currently in use by the system.

[rnovikov]

in "BINDING PROCESS", as i first step
cd /sys/bus/pci/drivers/r8169
ls
0003:31:00.0 0004:41:00.0 bind module new_id remove_id uevent unbind
echo 0004:41:00.0 > unbind
After this i can sucsesifull bind it back to r8169, but cannot bind to vifo

root@ubuntu:/sys/bus/pci/drivers# echo 0004:41:00.0 > vfio-pci/bind no work
bash: echo: write error: No such device
root@ubuntu:/sys/bus/pci/drivers# echo 0004:41:00.0 > r8169/bind work

[rnovikov]

I have progress with this issue.
cat config-6.1.0-1012-rockchip | grep IOM
CONFIG_VIRTIO_IOMMU is not set
CONFIG_ROCKCHIP_IOMUX is not set
could you give me manual, how to make iso image locally

[Joshua-Riek]

I can enable this in the kernel and push an update tonight. There are a few fixes I already have staged.

[rnovikov]

Thank you very much!

[Joshua-Riek]

Sorry for the delay, I'm waiting on the kernel to build on Launchpad, hopefully it will be done in a few hours.

https://launchpad.net/~jjriek/+archive/ubuntu/rockchip/+build/28434319

[Joshua-Riek]

The kernel upgrade is now ready, any luck using the VFIO drivers?

[rnovikov]

Sorry, long weekend. No access to device till tomorrow.

[rnovikov]

Sorry for delay. Still no work
root@ubuntu:/home/ubuntu# echo 0004:41:00.0 > /sys/bus/pci/drivers/r8169/unbind
root@ubuntu:/home/ubuntu# echo 0004:41:00.0 > /sys/bus/pci/devices/0004:41:00.0/driver_override
root@ubuntu:/home/ubuntu# echo vfio-pci > /sys/bus/pci/devices/0004:41:00.0/driver_override
root@ubuntu:/home/ubuntu# echo 0004:41:00.0 > /sys/bus/pci/drivers/vfio-pci/bind
bash: echo: write error: Invalid argument
root@ubuntu:/home/ubuntu# echo 0004:41:00.0 > /sys/bus/pci/drivers/vfio-pci/bind
bash: echo: write error: Invalid argument

[rnovikov]

I dig a bit dipper. Same problem on ubuntu 22.04. with NIC (0004:41:00.0) and with SSD (0000:01:00.0)