Add support for USB Knx Interfaces

[farmio]

It would be nice to support USB interfaces in addition to KNX/IP.
Next to supporting users that don't own IP Interfaces Xknx could be used as USB - IP bridge in the future.

See KNX specification 9 - 3: Basic System and Components - Couplers §3 KNX USB Interface

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Please make sure to update to the latest version of xknx (or Home Assistant) and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[kistlin]

For the not yet so familiar ones like myself. The specification can be downloaded for free from KNX Specifications. A free account can be created and then the mentioned item has to be put in the basket. After the order is placed (free again) you can download the specification.

KNX uses the USB HID device class.

For those that want to just have a quick look, here are some online resources.

KNX protocol description

• How to implement KNX USB Interface device?

C++ implementation of the USB interface

• knx

USB HID specification

• Device Class Definition for Human Interface Devices (HID)
• HID Usage Tables FOR Universal Serial Bus (USB)

To talk to KNX USB devices there are different possibilities.

The python library pyusb uses libusb and there is activity trying to integrate it into asyncio in pyusb/pyusb#63. There is also the python library hidapi which is more HID specific.

@farmio, do you already have thoughts on how to integrate all this?

[farmio]

Hi 👋!

If one is more comfortable in looking at Java there is also the Calimero project having Knx USB support.
https://github.com/calimero-project

@kistlin I personally do not have any thoughts on how to implement this, nor any plans to do it myself. If you like to do it, you are very welcome! Feel free to join the xknx Discord Server if you want to chat about implementation details.

[kistlin]

For now I have done some isolated work in the examples folder. It can be found in the following commit kistlin@c6f70a6
Branch feature/add_usb_interface_support examples/usb

As a USB interface I used the Siemens OCI702 USB - KNX Serviceinterface.

Sending bytes from a recorded valid KNX frame, seems to work.
Also parsing of a USB HID frame into a more meaningful object seems straight forward.
I feel comfortable in the lower parts of the communication.

@farmio do you know where in the specification I find information on the application layer?
I got in my view a non-trivial KNX device, which has lots of KNX objects in the manual.
I would be interested how these objects map to actual bytes on a lower level.

And maybe a description (reference to a document) of the workflow of typical first steps.
For example assigning an address to a device. What bytes do we need to send? So that I don't have to reverse everything.

[farmio]

Hi!

For Application layer information I'd look at 03_03_07 Application Layer v01.06.02 AS - in xknx we have the xknx.telegram.apci module for the different services. This can be passed to a xknx.telegram.Telegram object as payload. See https://github.com/XKNX/xknx/blob/main/xknx/knxip/cemi_frame.py for how we use it to encode/decode a Telegram object to and from KNX/IP. Afaik USB doesn't necessarily use CEMI-Frames so you maybe would have to add the USB-pendant for this.

To map KNX group objects to bytes you need to know the used DPT and decode it accordingly. See the xknx.dpt module. We use xknx.device to abstract that.
If what you are looking for is to reprogram the device, I'm afraid I can't help you. Afaik this is vendor/device specific.

Assigning an address to a device is currently not supported from xknx. There is currently a PR open for this.

From https://www.cs.hs-rm.de/~werntges/proj/knxusb/KSC2005-LinuxUSB.pdf I read there is an Application note KNXonUSB (AN0037) for KNX over USB. Maybe you'll have to requeset it from KNXA (my.knx.org support ticket).

[kistlin]

Ok, maybe back to focusing on integrating USB support.

What helped me was document 03_06_03 EMI_IMI v01.03.03 AS.pdf in combination with this image

in 09_03 Basic and System Components - Couplers v01.03.03 AS.pdf.

There in the KNX USB Transfer Protocol Body is the cEMI/EMI1/EMI2 frame. And from your comment and the code, xknx implements atm. only the cEMI frame right?

If I look for example in 03_06_03 EMI_IMI v01.03.03 AS.pdf chapter 4.1.5.3.3 L_Data.req.

This is the higher level interpretation of the KNX USB Transfer Protocol Body.

So the integration in a perfect world would then be, receive USB packets on a defined interface and concatenate the data packets together. Once a complete frame is received, check what frame type it is and pass it on to the already existing xknx implementation.

All the higher levels are transport independant, as it should be. Am I missing something?

[farmio]

Yes sounds reasonable. The L_Data.req example here is a CEMI Frame - so this is what you get by creating a xknx.telegram.Telegram and encapsule it in a cemi frame xknx.knxip.cemi_frame.CEMIFrame (with CEMIFrame.init_from_telegram or setting the telegram property).

    telegram = Telegram(
            destination_address=GroupAddress("1/0/15"),
            payload=GroupValueWrite(DPTBinary(0)),
        )
    )
    cemi = CEMIFrame.init_from_telegram(xknx, telegram, code=CEMIMessageCode.L_DATA_REQ, src_addr=xknx.own_address)

Then append cemi_frame.to_knx() to your payload - this should make your whole KNX USB Transfer Protocol Body.
To parse received frames pass the KNX USB Transfer Protocol Body (cut the header) to CEMIFrame.from_knx()

cemi = CEMIFrame(xknx)
try:
    cemi.from_knx(raw[header_length:])
except UnsupportedCEMIMessage as err:
    logger.warning("CEMI not supported: %s", err)
    # handle error

(see eg https://github.com/XKNX/xknx/blob/main/xknx/knxip/routing_indication.py)

As far as I know Cemi support is mandatory, EMI1 adn EMI2 are optional, so going with Cemi schould be fine.

[kistlin]

What do you think of an integration where a KNXIPInterface or USBInterface object, depending on the ConnectionConfig passed in, is instantiated.

To test this integration I created a ConnectionConfigUSB.
The changes in xknx/xknx.py are pretty minimal.

@@ -114,14 +116,19 @@ class XKNX:
     async def start(self) -> None:
         """Start XKNX module. Connect to KNX/IP devices and start state updater."""
         self.task_registry.start()
-        self.knxip_interface = KNXIPInterface(
-            self, connection_config=self.connection_config
-        )
-        logger.info(
-            "XKNX v%s starting %s connection to KNX bus.",
-            VERSION,
-            self.connection_config.connection_type.name.lower(),
-        )
+        if isinstance(self.connection_config, ConnectionConfig):
+            self.knxip_interface = KNXIPInterface(
+                self, connection_config=self.connection_config
+            )
+            logger.info(
+                "XKNX v%s starting %s connection to KNX bus.",
+                VERSION,
+                self.connection_config.connection_type.name.lower(),
+            )
+        if isinstance(self.connection_config, ConnectionConfigUSB):
+            self.knxip_interface = USBInterface(self, connection_config=self.connection_config)
+            logger.info("XKNX start logging on USB device (idVendor: 0x%04x, idProduct: 0x%04x)",
+                        self.connection_config.idVendor, self.connection_config.idProduct)
         await self.knxip_interface.start()
         await self.telegram_queue.start()
         if self.start_state_updater:

The switch example with USB would look like this. Only the instantiation of XKNX needs a change.

...
async def main():
    xknx = XKNX(connection_config=ConnectionConfigUSB(USBVendorId.SIEMENS_OCI702, USBProductId.SIEMENS_OCI702))
    await xknx.start()
    switch = Switch(xknx, name="TestOutlet", group_address="1/1/11")
...

All the existing code with default initialization would still work as expected.

[farmio]

Sure for an initial proof of concept implementation this sounds reasonable.
Later we can always rename knxip_interface to knx_interface and maybe consolidate the ConnectionConfig classes somehow so the KNXInterface class handles connections to IP and USB in the io module.

To communicate with xknx devices it just needs to register as an xknx.io.Interface subclass at xknx.knxip_interface.interface(so send_telegram is called, and register the callback xknx.knxip_interface.telegram_received to put received GroupValue* telegrams into xknx.telegrams asyncio.Queue.

Is there a tunnelling-like ConnectRequest - ConnectResponse process in USB or is it connectionless?

[kistlin]

In 09_03 Basic and System Components - Couplers v01.03.03 AS.pdf it says

3.4.2 KNX tunnelling

For communication between a tool on a PC and a KNX device connected to the KNX network, KNX
frames are tunnelled on the USB link using one of the EMI formats.
The time-out for a KNX tunnelling is 1 s. This is, a KNX USB Interface Device shall be able to receive a tunnelling frame, transmit it on the KNX medium and send the local confirmation back to the tool within 1 s. This is a recommended value which shall be complied to under normal bus load.

3.5.3.2 Device feature services

Figure 21 – Device Feature Service
...

For a possible use, please refer to the features proposed in 3.5.3.3 “Device features” below. Please note that only the Device Feature Get service is confirmed by a Device Feature Response service. The Device Feature Set- and the Device Feature Info services shall not be answered by the receiver.

...

The time-out for the Device Feature Get service is 1 s. This is, a KNX USB Interface Device shall reply with a Device Feature Response frame within 1 s.

For me the relevant parts sound like request/response only. Or just sending something.

[farmio]

Yes it seems no TunnelingACK is sent on USB, which is good. I guess L_DATA.con confirmation CEMI frames will be received (and should be waited for) just like on IP Tunneling?

Is it possible to identify a USB device as KNX interface? see https://wiki.debian.org/HowToIdentifyADevice/USB do they share some common device protocol or class or description?

[kistlin]

As for the first question, the only thing I found atm. is in 03_06_03 EMI_IMI v01.03.03 AS.pdf

4.1.5.1 Flow Control

During treatment of a request that is not yet confirmed to the cEMI client, the cEMI Server shall accept a
new request from the cEMI client. This is used e.g. for management requests (see clause 4.1.7) during an
L_Data.req/L_Data.con cycle.

Which indicates that the server could receive multiple requests, but I guess it makes it easier to implement (maybe?) if we just wait on the response first.

To the second question in 09_03 Basic and System Components - Couplers v01.03.03 AS.pdf

3 KNX USB Interface

3.1 Introduction

3.1.1 Scope

Discovery and self-description mechanisms on USB level are not in the scope of this clause as well as
mechanisms for configuring and establishing of a communication link between the USB host (PC) and
the KNX USB Interface Device. These mechanisms are managed by the USB host protocol. Please refer
to the corresponding USB specification documents.

And the USB HID specification defines that

4.1 The HID Class

The USB Core Specification defines the HID class code. The bInterfaceClass
member of an Interface descriptor is always 3 for HID class devices.

So in my understanding there is no real possibility to scan the USB devices and be sure it is a KNX device.

A possibility is to enumerate all HID devices and check for a known set of vendor/product id's. These would probably grow over time.

[farmio]

See #323 (and the link there if you speak German) for the L_Data.con flow control.
This was just recently added to xknx. Worked before with the rate_limiter but now we can be (I hope) sure we never flood the receivers buffers even on high Bus load (or disabled rate_limit).

[kistlin]

Yes the reference in your ticket to 4.1.5.1 Flow Control in 03_06_03 EMI_IMI v01.03.03 AS.pdf seems to indicate, that we should wait for a con.

4.1.5.1 Flow Control

cEMI Client
To keep the flow control for Data Link Layer services as simple as possible (this allows a simple flow
control state machine in the cEMI client), it is recommended that:
• a cEMI client sends a new Data Link Layer request only when the confirmation of the preceding
request is received, or
• a request-to-confirmation timeout is recognised; the recommended time-out for the cEMI client is
3 seconds.
A cEMI client shall at any time be able to accept an indication message from the cEMI Server.

FYI your assumption in the forum about the meaning of con was right, by accident I read that part. 03_03_04 Transport Layer v01.02.02 AS.pdf

1.1 Communication Modes

Every communication mode shall provide specific Transport Layer Service Access Points (TSAPs)
accessible via different Transport Layer services. Each of these services shall consists of three service
primitives, the request(req), the confirm (con) and the indication (ind).

And just for me, we are speaking about this, so that a possible USB implementation doesn't make the same mistake? :)
Also do you consider a refactoring so we don't have twice the logic and twice the tests for the same problem? Or not yet?

Because for now I focused on the sending of a telegram. I hoped to just put a received telegram into the queue an we are good :).

[farmio]

Also do you consider a refactoring so we don't have twice the logic and twice the tests for the same problem? Or not yet?

Not yet, since not every sent telegram needs this (eg. when using routing).
The confirmation waiting function is currently not more than
https://github.com/XKNX/xknx/blob/0.18.15/xknx/io/tunnel.py#L282-L296
(you wouldn't need the Tunnelling class since no ACK is required)

I hoped to just put a received telegram into the queue an we are good :).

I think this will work fine. 👍

[farmio]

Hey! Small update for

Also do you consider a refactoring so we don't have twice the logic and twice the tests for the same problem? Or not yet?

I did move some code in #841 that may ease the implementation of a USB-Tunnel (using the _Tunnel class. USB seems very similar to what is needed for TCP tunnels). Its not perfect as it still has some IP-specific attributes, but I guess its a start.

Marvin is working on a refactoring of the ConnectionConfig class - we'd like to use separate Dataclasses for the connection types with some minor config validation baked into it - let's see.

[kistlin]

Hello,

recently I wasn't that active. I'll try to take a look into the changes next week.

And if one of you is working with KNX over USB, I have commited a Wireshark KNX USB dissector.

This helped me big time in quickly understanding the message flow.
Put the script in the B.4. Plugin folders and it should work. Maybe you have to select the protocol in the GUI.

[mikimikeCH]

Hi kistlin,

I' am trying to use your USB implementation of xknx. When I use your example_usb.py I get:

INFO:xknx.log:XKNX start logging on USB device (idVendor: 0x0908, idProduct: 0x02dc)
WARNING:xknx.usb:TODO: load libusb dll on Windows
INFO:xknx.usb:found 1 device(s)
INFO:xknx.usb:device 1
INFO:xknx.usb:    manufacturer  : Siemens (idVendor: 0x0908)
INFO:xknx.usb:    product       : OCI702 KNX Interface (idProduct: 0x02dc)
INFO:xknx.usb:    serial_number : 00FD10D01DB7
DEBUG:xknx.telegram:<Telegram direction="Outgoing" source_address="0.0.0" destination_address="1/1/11" payload="<GroupValueWrite value="<DPTBinary value="True" />" />" />
DEBUG:xknx.state_updater:StateUpdater stopping
DEBUG:xknx.telegram:<Telegram direction="Outgoing" source_address="0.0.0" destination_address="1/1/11" payload="<GroupValueWrite value="<DPTBinary value="False" />" />" />
DEBUG:xknx.log:Stopping TelegramQueue```

It seems to find the USB KNX Interface and passes the telegram. I have a BUS Monitor running in ETS 6 with an older Siemens OCI700 Interface, but there I don't receive any Telegram. Is the problem that I have a Warning "load libusb dll"?  

[kistlin]

Hello @mikimikeCH,

as long as it finds the device the warning shouldn't matter that much. I just didn't run it on Windows yet.

The bigger problem is that the implementation is not in a usable state. I never finished it. I don't have an actual setup that works, just the programmer as you have. And I ran out of holidays :).

Are you familiar with debugging? Else you could follow the code and look where sending fails. If it even reaches USBClient.send_telegram in xknx/io/usb_client.py.

[Golpe82]

Hi there,
i changed a bit the code of @kistlin for using my usb iface of jung:

util.py

class USBVendorId(IntEnum):
    """ Vendor ID's """
    SIEMENS_OCI702 = 0x0908
    JUNG_2130USBREG = 0x135e


class USBProductId(IntEnum):
    """ Product ID's """
    SIEMENS_OCI702 = 0x02dc
    JUNG_2130USBREG = 0X0023 

......

example_switch.py

........
async def main():
    #xknx = XKNX(connection_config=ConnectionConfigUSB(usb_util.USBVendorId.SIEMENS_OCI702, usb_util.USBProductId.SIEMENS_OCI702))
    xknx = XKNX(connection_config=ConnectionConfigUSB(usb_util.USBVendorId.JUNG_2130USBREG, usb_util.USBProductId.JUNG_2130USBREG))
    await xknx.start()
    #switch = Switch(xknx, name="TestOutlet", group_address="1/1/11")
    switch = Light(xknx, name="TestOutlet", group_address_switch="1/2/30")
    await switch.set_on()
    await asyncio.sleep(10)
    await switch.set_off()
    await xknx.stop()


if __name__ == "__main__":
    asyncio.run(main())

and i have similar output like @mikimikeCH

DEBUG:asyncio:Using selector: EpollSelector
INFO:xknx.log:XKNX start logging on USB device (idVendor: 0x135e, idProduct: 0x0023)
INFO:xknx.usb:found 1 device(s)
INFO:xknx.usb:device 1
INFO:xknx.usb:    manufacturer  : ALBRECHT JUNG GMBH & CO. KG (idVendor: 0x135e)
INFO:xknx.usb:    product       : KNX-USB Data Interface (idProduct: 0x0023)
INFO:xknx.usb:    serial_number : None
DEBUG:xknx.telegram:<Telegram direction="Outgoing" source_address="0.0.0" destination_address="1/2/30" payload="<GroupValueWrite value="<DPTBinary value="True" />" />" />
DEBUG:xknx.state_updater:StateUpdater stopping
DEBUG:xknx.telegram:<Telegram direction="Outgoing" source_address="0.0.0" destination_address="1/2/30" payload="<GroupValueWrite value="<DPTBinary value="False" />" />" />
DEBUG:xknx.log:Stopping TelegramQueue

Let´s find time for debugging...

Also a rebase with main is needed from the branch of @kistlin , there are small conflicts

[kistlin]

@Golpe82 I pushed changes that integrate the existing implementation into the current main of XKNX. And some minor changes. When I now run the example my KNX interface at least blinks :). Does not mean it works yet.

[farmio]

Oh that is great! Still looking forward to having support for USB from xknx directly 😃

We did some changes allowing to directly return list[Telegram] from a received Telegram - to answer incoming request-response queries directly (for management).

Unfortunately we still have no perfectly clean separation of (OSI)Layers - so I guess there will still be some duplicated code in USB and IP branches...

If one of you has any direct questions regarding implementation details or such, don't hesitate to join our Discord server! 👍

[Golpe82]

Hi, I was playing a bit today.
I made a project 2 years ago that translate http to KNX frames using a raspi and a kberry.
For it I needed to build cemi frames but embedded in ft1.2 protocol.
I suppose for usb the ft1.2 protocoll is also needed, not only the cemi frames:
https://drive.google.com/file/d/1-B_LsezclvBC3x42pkxy9pLOOp0ll1YZ/view?usp=drivesdk

[Golpe82]

I mean I was commuticating with the kberry over tty serial port and usb is also serial.
I suppose xknx access the bus over network layer but serial connection succeed over link layer I think and the frame needs the ft1.2 protocoll for it so far I understand

[Golpe82]

Take a look to apendixE (this time English):
https://www.google.com/url?q=https://weinzierl.de/images/download/documents/baos/knx_baos_binary_protocol_v2_0.pdf&sa=U&ved=2ahUKEwjSiJHE6v34AhVagv0HHfhnBwAQFnoECAAQAg&usg=AOvVaw0yEF3lHYytLXW9xje22UoO

Or maybe am I confusing things?
Ft1.2 is for uart. The question is if the usb interface transforms dataframe to uart itsself? Aka. Embeds the cemi in a ft1.2 alone?

[farmio]

Eg. Calimero lists FT1.2 and KNX USB as separate protocols.
https://github.com/calimero-project/calimero-core

I couldn't find any mention of "FT1.2" in the Knx specifications pdfs 🧐

[Golpe82]

ok, so far i understand ft1.2 is only for TP/KNX BAOS Module 830 of Weinzierl (also kberry)
https://www.knx.org/wAssets/docs/downloads/Marketing/Flyers/KNX-Development-Solutions/KNX-Development-Solutions_en.pdf

[mikimikeCH]

Hi @farmio

If i remember right i had to "Install a device filter" when i wanted to access my USB-KNX Interface from Python.

Maybe on macOS you have to do it as well.

[kistlin]

Hey @farmio,

great to hear. A few weeks back I added a bunch of unit tests for assembling/splitting frames. I would also like to get an initial merge soon.

I switched regularly between Windows and Linux, but not on a Mac. On Linux you shouldn't forget udev rules if you want to use it as normal user and Windows should work once the right drivers are installed.

But that problem looks similar to pyusb/pyusb#208.
They mention to use HIDAPI and the Cython binding. Maybe a migration is required to get it to work.

If you can, you could try on Windows or Linux first. To run some basic tests.
And then either of us could look into making it work on a Mac. I have one and I see the same error.

[farmio]

So after searching the web for some hours I think it is a limitation / security feature of macOS that can't be worked around that easily. See libusb/libusb#1014
Looking at libusb documenation it even recommends to use the hidapi rather than libusb directly. https://github.com/libusb/libusb/wiki/FAQ#user-content-Does_libusb_support_USB_HID_devices which also has up-to-date python bindings which look quite promising.

• https://pypi.org/project/hidapi/
• https://pypi.org/project/hid/

I think I'll take a stab on testing these.

[kistlin]

Ok. Then I'll leave it up to you for now :).
It might also simplify the code around USB.

[farmio]

Allright! I have copied your branch to https://github.com/XKNX/xknx/tree/usb-interface-support
@kistlin Feel free to open a (draft-)PR against main 🙂

[farmio]

So I did some testing on Windows 10 and had no luck - even with different drivers.
libusb1 times out with usb:Operation not supported or unimplemented on this platform

libusb1

DEBUG:asyncio:Using proactor: IocpProactor
INFO:xknx.log:XKNX start logging on USB device (idVendor: 0x0000, idProduct: 0x0000)
INFO:xknx.log:XKNX v2.2.0 starting tunneling connection to KNX bus.
INFO:xknx.usb:found 1 device(s)
INFO:xknx.usb:device 1
INFO:xknx.usb:    manufacturer  : Gira Giersiepen GmbH & Co. KG (idVendor: 0x135e)
INFO:xknx.usb:    product       : KNX-USB Data Interface (idProduct: 0x0022)
INFO:xknx.usb:    serial_number : None
INFO:root:Using device:
DEVICE ID 135e:0022 on Bus 002 Address 008 =================
 bLength                :   0x12 (18 bytes)
 bDescriptorType        :    0x1 Device
 bcdUSB                 :  0x110 USB 1.1
 bDeviceClass           :    0x0 Specified at interface
 bDeviceSubClass        :    0x0
 bDeviceProtocol        :    0x0
 bMaxPacketSize0        :    0x8 (8 bytes)
 idVendor               : 0x135e
 idProduct              : 0x0022
 bcdDevice              :  0x103 Device 1.03
 iManufacturer          :    0x1 Gira Giersiepen GmbH & Co. KG
 iProduct               :    0x2 KNX-USB Data Interface
 iSerialNumber          :    0x0
 bNumConfigurations     :    0x1
  CONFIGURATION 1: 50 mA ===================================
   bLength              :    0x9 (9 bytes)
   bDescriptorType      :    0x2 Configuration
   wTotalLength         :   0x29 (41 bytes)
   bNumInterfaces       :    0x1
   bConfigurationValue  :    0x1
   iConfiguration       :    0x0
   bmAttributes         :   0x80 Bus Powered
   bMaxPower            :   0x19 (50 mA)
    INTERFACE 0: Human Interface Device ====================
     bLength            :    0x9 (9 bytes)
     bDescriptorType    :    0x4 Interface
     bInterfaceNumber   :    0x0
     bAlternateSetting  :    0x0
     bNumEndpoints      :    0x2
     bInterfaceClass    :    0x3 Human Interface Device
     bInterfaceSubClass :    0x0
     bInterfaceProtocol :    0x0
     iInterface         :    0x0
      ENDPOINT 0x81: Interrupt IN ==========================
       bLength          :    0x7 (7 bytes)
       bDescriptorType  :    0x5 Endpoint
       bEndpointAddress :   0x81 IN
       bmAttributes     :    0x3 Interrupt
       wMaxPacketSize   :   0x40 (64 bytes)
       bInterval        :    0x2
      ENDPOINT 0x2: Interrupt OUT ==========================
       bLength          :    0x7 (7 bytes)
       bDescriptorType  :    0x5 Endpoint
       bEndpointAddress :    0x2 OUT
       bmAttributes     :    0x3 Interrupt
       wMaxPacketSize   :   0x40 (64 bytes)
       bInterval        :    0x2
DEBUG:xknx.usb:Operation not supported or unimplemented on this platform
DEBUG:xknx.telegram:<Telegram direction="Outgoing" source_address="0.0.0" destination_address="5/1/20" payload="<GroupValueRead />" />
DEBUG:xknx.log:sending: <Telegram direction="Outgoing" source_address="0.0.0" destination_address="5/1/20" payload="<GroupValueRead />" />
DEBUG:xknx.log:write 64 bytes: 0113130008000b010300001100bce000002914010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Exception in thread USBSendThread:
Traceback (most recent call last):
  File "C:\Users\meti\AppData\Local\Programs\Python\Python310\lib\threading.py", line 1009, in _bootstrap_inner
    self.run()
  File "c:\Users\meti\dev\xknx\xknx\usb\usb_send_thread.py", line 39, in run
    self.usb_device.write(hid_frame.to_knx())
  File "c:\Users\meti\dev\xknx\xknx\usb\util.py", line 209, in write
    write_count = self._ep_out.write(data)
  File "C:\Users\meti\AppData\Local\Programs\Python\Python310\lib\site-packages\usb\core.py", line 408, in write
    return self.device.write(self, data, timeout)
  File "C:\Users\meti\AppData\Local\Programs\Python\Python310\lib\site-packages\usb\core.py", line 989, in write
    return fn(
  File "C:\Users\meti\AppData\Local\Programs\Python\Python310\lib\site-packages\usb\backend\libusb1.py", line 855, in intr_write
    return self.__write(self.lib.libusb_interrupt_transfer,
  File "C:\Users\meti\AppData\Local\Programs\Python\Python310\lib\site-packages\usb\backend\libusb1.py", line 938, in __write
    _check(retval)
  File "C:\Users\meti\AppData\Local\Programs\Python\Python310\lib\site-packages\usb\backend\libusb1.py", line 602, in _check
    raise USBTimeoutError(_strerror(ret), ret, _libusb_errno[ret])
usb.core.USBTimeoutError: [Errno 10060] Operation timed out
WARNING:xknx.log:Error: KNX bus did not respond in time (2.0 secs) to GroupValueRead request for: 5/1/20
Value: None - took 2.008 seconds
DEBUG:xknx.state_updater:StateUpdater stopping
DEBUG:xknx.log:Stopping TelegramQueue
DEBUG:xknx.log:stopping thread USBSendThread
DEBUG:xknx.log:stopping thread USBReceiveThread
DEBUG:xknx.log:USBSendThread stopped
DEBUG:xknx.log:USBReceiveThread stopped

whereas libusb0 ´[Errno None] b'libusb0-dll:err [claim_interface] could not claim interface 0, invalid configuration 0\n'´

libusb0

DEBUG:asyncio:Using proactor: IocpProactor
INFO:xknx.log:XKNX start logging on USB device (idVendor: 0x0000, idProduct: 0x0000)
INFO:xknx.log:XKNX v2.3.0 starting tunneling connection to KNX bus.
ERROR:xknx.usb:No USB backend found. Set XKNX_LIBUSB environment variable pointing to libusb-1.0.dll or install it to C:\Windows\System32
INFO:xknx.usb:found 1 device(s)
INFO:xknx.usb:device 1
INFO:xknx.usb:    manufacturer  : Gira Giersiepen GmbH & Co. KG (idVendor: 0x135e)
INFO:xknx.usb:    product       : KNX-USB Data Interface (idProduct: 0x0022)
INFO:xknx.usb:    serial_number : None
INFO:root:Using device: 
DEVICE ID 135e:0022 on Bus 000 Address 001 =================
 bLength                :   0x12 (18 bytes)
 bDescriptorType        :    0x1 Device
 bcdUSB                 :  0x110 USB 1.1
 bDeviceClass           :    0x0 Specified at interface
 bDeviceSubClass        :    0x0
 bDeviceProtocol        :    0x0
 bMaxPacketSize0        :    0x8 (8 bytes)
 idVendor               : 0x135e
 idProduct              : 0x0022
 bcdDevice              :  0x103 Device 1.03
 iManufacturer          :    0x1 Gira Giersiepen GmbH & Co. KG
 iProduct               :    0x2 KNX-USB Data Interface
 iSerialNumber          :    0x0
 bNumConfigurations     :    0x1
  CONFIGURATION 1: 50 mA ===================================
   bLength              :    0x9 (9 bytes)
   bDescriptorType      :    0x2 Configuration
   wTotalLength         :   0x29 (41 bytes)
   bNumInterfaces       :    0x1
   bConfigurationValue  :    0x1
   iConfiguration       :    0x0
   bmAttributes         :   0x80 Bus Powered
   bMaxPower            :   0x19 (50 mA)
    INTERFACE 0: Human Interface Device ====================
     bLength            :    0x9 (9 bytes)
     bDescriptorType    :    0x4 Interface
     bInterfaceNumber   :    0x0
     bAlternateSetting  :    0x0
     bNumEndpoints      :    0x2
     bInterfaceClass    :    0x3 Human Interface Device
     bInterfaceSubClass :    0x0
     bInterfaceProtocol :    0x0
     iInterface         :    0x0
      ENDPOINT 0x81: Interrupt IN ==========================
       bLength          :    0x7 (7 bytes)
       bDescriptorType  :    0x5 Endpoint
       bEndpointAddress :   0x81 IN
       bmAttributes     :    0x3 Interrupt
       wMaxPacketSize   :   0x40 (64 bytes)
       bInterval        :    0x2
      ENDPOINT 0x2: Interrupt OUT ==========================
       bLength          :    0x7 (7 bytes)
       bDescriptorType  :    0x5 Endpoint
       bEndpointAddress :    0x2 OUT
       bmAttributes     :    0x3 Interrupt
       wMaxPacketSize   :   0x40 (64 bytes)
       bInterval        :    0x2
DEBUG:xknx.usb:is_kernel_driver_active
WARNING:xknx.log:[Errno None] b'libusb0-dll:err [claim_interface] could not claim interface 0, invalid configuration 0\n'
DEBUG:xknx.telegram:<Telegram direction="Outgoing" source_address="0.0.0" destination_address="5/1/20" payload="<GroupValueRead />" />
DEBUG:xknx.cemi:Outgoing CEMI: <CEMIFrame code="L_DATA_REQ" src_addr="IndividualAddress("0.0.0")" dst_addr="GroupAddress("5/1/20")" flags="1011110011100000" tpci="TDataGroup()" payload="<GroupValueRead />" />
DEBUG:xknx.log:sending: <CEMIFrame code="L_DATA_REQ" src_addr="IndividualAddress("0.0.0")" dst_addr="GroupAddress("5/1/20")" flags="1011110011100000" tpci="TDataGroup()" payload="<GroupValueRead />" />DEBUG:xknx.log:write 64 bytes: 0113130008000b010300001100bce000002914010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Exception in thread USBSendThread:
Traceback (most recent call last):
  File "C:\Users\meti\AppData\Local\Programs\Python\Python310\lib\threading.py", line 1009, in _bootstrap_inner
    self.run()
  File "c:\Users\meti\dev\xknx\xknx\usb\usb_send_thread.py", line 33, in run
    self.usb_device.write(hid_frame.to_knx())
  File "c:\Users\meti\dev\xknx\xknx\usb\util.py", line 214, in write
    write_count = self._ep_out.write(data)
  File "C:\Users\meti\AppData\Local\Programs\Python\Python310\lib\site-packages\usb\core.py", line 408, in write
    return self.device.write(self, data, timeout)
  File "C:\Users\meti\AppData\Local\Programs\Python\Python310\lib\site-packages\usb\core.py", line 986, in write
    intf, ep = self._ctx.setup_request(self, endpoint)
  File "C:\Users\meti\AppData\Local\Programs\Python\Python310\lib\site-packages\usb\core.py", line 113, in wrapper
    return f(self, *args, **kwargs)
  File "C:\Users\meti\AppData\Local\Programs\Python\Python310\lib\site-packages\usb\core.py", line 229, in setup_request
    self.managed_claim_interface(device, intf)
  File "C:\Users\meti\AppData\Local\Programs\Python\Python310\lib\site-packages\usb\core.py", line 113, in wrapper
    return f(self, *args, **kwargs)
  File "C:\Users\meti\AppData\Local\Programs\Python\Python310\lib\site-packages\usb\core.py", line 178, in managed_claim_interface
    self.backend.claim_interface(self.handle, i)
  File "C:\Users\meti\AppData\Local\Programs\Python\Python310\lib\site-packages\usb\backend\libusb0.py", line 447, in _check
    raise USBError(errmsg, ret)
usb.core.USBError: [Errno None] b'libusb0-dll:err [claim_interface] could not claim interface 0, invalid configuration 0\n'
WARNING:xknx.log:[Errno None] b'libusb0-dll:err [claim_interface] could not claim interface 0, invalid configuration 0\n'
WARNING:xknx.log:Error: KNX bus did not respond in time (2.0 secs) to GroupValueRead request for: 5/1/20
Value: None - took 2.002 seconds
DEBUG:xknx.state_updater:StateUpdater stopping
WARNING:xknx.log:[Errno None] b'libusb0-dll:err [claim_interface] could not claim interface 0, invalid configuration 0\n'
WARNING:xknx.log:L_DATA_CON Data Link Layer confirmation timed out for <CEMIFrame code="L_DATA_REQ" src_addr="IndividualAddress("0.0.0")" dst_addr="GroupAddress("5/1/20")" flags="1011110011100000" tpci="TDataGroup()" payload="<GroupValueRead />" />
DEBUG:xknx.log:Stopping TelegramQueue
DEBUG:xknx.log:stopping thread USBSendThread
DEBUG:xknx.log:stopping thread USBReceiveThread
DEBUG:xknx.log:USBSendThread stopped
DEBUG:xknx.log:USBReceiveThread stopped

Can you give me a hint of how to get that running on Windows? What are the right drivers?

I'll refactor to use CEMIFrame instead of Telegram meanwhile - but I guess this will be more straight forward if I could test it.

Edit: under Ubuntu VM I don't get the UDEV rule working. But with sudo, at least I don't get any error, the interface gets found and according to logs it seems a telegram is sent. Nothing hits the bus though 🫤

USB is simple and reliable, they said 🤣

[kistlin]

So I did some testing on Windows 10 and had no luck - even with different drivers. libusb1 times out with usb:Operation not supported or unimplemented on this platform

libusb1
whereas libusb0 ´[Errno None] b'libusb0-dll:err [claim_interface] could not claim interface 0, invalid configuration 0\n'´

libusb0
Can you give me a hint of how to get that running on Windows? What are the right drivers?

Did you see _get_usb_backend in xknx\usb\util.py?
It expects libusb1 to be at C:\Windows\System32\libusb-1.0.dll or you can set it with an environment variable XKNX_LIBUSB.

Yeah all these quirks. But HID might help.

[farmio]

Yes I saw that and changed the path to my .DLL that was provided by libusb_package and also a downloaded version from libusb.info.
I'm a Windows noob - I probably forgot something obvious.

[kistlin]

If that doesn't help then it was me probably using Zadig and install WinUSB for my usb device.
I just had to remove that, else hidapi would no longer find my device.

I would say let's forget about pyusb/libusb and just use hidapi. Too many inconveniences.

[kistlin]

@farmio
I quickly hacked some hidapi support in. You might try it on your system. It's in my branch under usb-interface-support.
Using the monitor example I don't get what I expect. But running the switch example, it seems to communicate.

[farmio]

I did also try Zadig 🤣
Ok, just tested your branch - hidapi doesn't raise any errors on macOS and Windows - so that's good. But it also doesn't actually send anything to the bus - so thats to improve 😬

[farmio]

Ok, so unfortunately it seems it's more work to get this running with my interface than I thought... according to this Device Feature Get - Response

hidapitester --vidpid 135e/0022 --open --send-output 1,0x13,0x09,0x00,0x08,0x00,0x01,0x0F,0x01,00,00,01 --read-input
Opening device, vid/pid: 0x135E/0x0022
Writing output report of 64-bytes...wrote 64 bytes:
 01 13 09 00 08 00 01 0F 01 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Reading 64-byte input report 0, 250 msec timeout...read 64 bytes:
 01 13 0B 00 08 00 03 0F 02 00 00 01 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Closing device

it looks to me that my interface doesn't support CEMI, but only EMI1 😭

[farmio]

We should probably implement a Device Feature Get - Response cycle on start - raising an error if the device doesn't support CEMI or do a Device Feature Set if it supports more than CEMI.

Currently the thread-encapsulation is not great... we call xknx.cemi_handler.handle_cemi_frame (which calls xknx.telegrams.put_nowait(telegram) or xknx.management.process(telegram)) from the receive thread - which should be handled in main thread. For a proof of concept, I don't think this is a problem, but in production code this should probably be avoided.

Also, do you think it is possible to do send and receive in a single thread? I could imagine something like this to work (untested pseudo code)

self._usb_device.set_nonblocking(True)
while self._is_active.is_set():
    if self.out_queue.qsize(): # Return the approximate size of the queue (not reliable!). <- not reliable doesn't sound promising
        self.usb_device.write(self.out_queue.get_nowait())  # maybe `try: except Empty:` - maybe even without qsize test
    usb_data = self._usb_device.read()
    if usb_data:
        self._knx_to_telegram.process(usb_data)

[kistlin]

Implementing a feature check makes sense.

True asyncio.Queue is not thread safe. And in the CEMIHandler multiple thread contexts would be at work.
With the proposed approach it evolves into timer based code, which I'm not a fan of.
As soon as it is blocking at read, the thread cannot write. Now some time has to be chosen to timeout in the read to allow more writes.

Alternatives to check out could also be:
awaitable loop.run_in_executor(executor, func, *args)
For example whenever something was read and the executor returns the result, queue a new read task which contains the blocking read call.
Write should just be a one off call.

Or janus

Mixed sync-async queue, supposed to be used for communicating between classic synchronous (threaded) code and asynchronous (in terms of [asyncio](https://docs.python.org/3/library/asyncio.html)) one.

https://stackoverflow.com/a/32894169/15862303

[farmio]

As soon as it is blocking at read

hence the set_nonblocking(1), or am I misunderstanding that?
http://hidapi-d.dpldocs.info/hidapi.bindings.hid_set_nonblocking.html

As stated in the SO answer: Maybe a call to loop.call_soon_threadsafe could probably do it too.
This would eliminate the need for another dependency. We already use that for IP - and I guess we could just use USB from the same class.

xknx/xknx/io/knxip_interface.py

Line 514 in e031588

self._main_loop.call_soon_threadsafe(super().cemi_received, cemi)

[kistlin]

As soon as it is blocking at read

hence the set_nonblocking(1), or am I misunderstanding that? http://hidapi-d.dpldocs.info/hidapi.bindings.hid_set_nonblocking.html

True. But then the thread would be needlessly spinning.

Yes loop.call_soon_threadsafe sounds like the best option at this point. I agree to keep dependencies at a minimum.

[kistlin]

@farmio

it looks to me that my interface doesn't support CEMI, but only EMI1

Did you see that in a data sheet? I tried to quickly look it up but couldn't find anything.
I compared the data exchange of my device when using ETS6.
There it asks the interface which EMI version it understands and then configures it to use cEMI for example.

So maybe yours supports more too, it just defaults to EMI1?

[farmio]

I just looked up the datasheet of my Gira 107000 USB interface (here in german). As expected there is no word about EMI or CEMI 🫥

I should have been more verbose about that output of hidapitester i posted above. There I did a Device Feature Get Request with Service Identifier 01 - Supported EMI Type.
See 09_03 Basic and System Components - Couplers v01.03.03 AS.pdf §3.5.3.2

The response was a 0x01

So it seems only EMI1 is supported with my device.

If multiple types were supported it would require to set the preferred type before further communication. See §3.5.3.3.2

[martinmoravek]

Hi all, is there any progress on supporting USB interfaces?
thanks

[farmio]

@martinmoravek afaik, none apart from what you can read here in this issue or see here https://github.com/XKNX/xknx/tree/usb-interface-support

I don't have a cemi compatible usb interface, so I moved to different topics again for now.

Feel free to chime in if you like.

[kistlin]

Hi all, is there any progress on supporting USB interfaces? thanks

As to me, in recent times I did not invest time worth mentioning. And it is not planned in the near future. Maybe a few days within the next two to three months.

The current state is

• basic sending/receiving over USB works for me (:)) (receiving button presses and temperature values)
• the initial communication with my USB device is not working, only after a couple of runs or using ES6 to monitor the bus

The last part is holding me up currently.
I tried to send what was mentioned by @farmio recently

If multiple types were supported it would require to set the preferred type before further communication. See §3.5.3.3.2

But it didn't seem to fix it. So I'm currently when I work on it comparing some captures.

After querying and setting the EMI type to use, I see a lot of M_PropRead.req/M_PropWrite.req.
Does someone know how to read those messages?
I see the message described in 03_06_03 EMI_IMI v01.03.03 AS.pdf but cannot make much sense out of Object Instance and Property ID and its data. Once I read a bit more on that there is maybe progress in having a stable communication from the start.

@martinmoravek if you are willing to test things or even develop that would be welcome. I only have one USB programmer and one KNX device. So no real variety to say with confidence if it works or not.

[farmio]

Since #1210 we should be able to decode these M_Prop messages, and there is also a module for decoding property ids 😃

[kistlin]

Ok thanks for the information.

And in the document it was right at the end of 03_06_03 EMI_IMI v01.03.03 AS.pdf in section 4.2.2 Generic management based on Interface Objects.

And while at it. The PID_COMM_MODE was by default FFh “no layer” on my device. Setting it to 00h Data Link Layer starts the communication. It can be checked with the Bus connection status (Bus Access Server Feature Service - Device Feature Get), which goes from 0 to 1.

So the next steps would be to integrate

• check the supported EMI type
• check the communication mode
• check the bus connection status

and after that start normal communication.

[martinmoravek]

Hi all, is there any progress on supporting USB interfaces? thanks

As to me, in recent times I did not invest time worth mentioning. And it is not planned in the near future. Maybe a few days within the next two to three months.

The current state is

• basic sending/receiving over USB works for me (:)) (receiving button presses and temperature values)
• the initial communication with my USB device is not working, only after a couple of runs or using ES6 to monitor the bus

The last part is holding me up currently. I tried to send what was mentioned by @farmio recently

If multiple types were supported it would require to set the preferred type before further communication. See §3.5.3.3.2

But it didn't seem to fix it. So I'm currently when I work on it comparing some captures.

After querying and setting the EMI type to use, I see a lot of M_PropRead.req/M_PropWrite.req. Does someone know how to read those messages? I see the message described in 03_06_03 EMI_IMI v01.03.03 AS.pdf but cannot make much sense out of Object Instance and Property ID and its data. Once I read a bit more on that there is maybe progress in having a stable communication from the start.

@martinmoravek if you are willing to test things or even develop that would be welcome. I only have one USB programmer and one KNX device. So no real variety to say with confidence if it works or not.

@kistlin not sure if I am able to develop (never ever developed in python, only c/c++, sql), but testing is no problem (I have usb, ip as well as rs232 interface). with some advice and help to get the dev environment running I might try some dev as well

[farmio]

Setting up a dev env is straightforward. Provided you have a supported version of Python (>=3.9) installed, it should suffice to clone the repo, switch the branch and install dependencies (see root Readme).

RS232 is out of scope imho (it's not even supported by ETS anymore 😬).

[kistlin]

@martinmoravek
No problem I can help you setup everyting. If you need help with a specific IDE/OS just tell me.

Additional steps before you install dependencies are creating a virtual environment. This prevents you from globally installing packages which will lead to version conflicts of packages, when you work on multiple projects.

To create one you can run
python -m venv env
in the root of the repository. (full path to Python if it is not in PATH)

And every time you launch a new terminal activate it with

• source ./env/bin/activate (Linux/Mac)
• .\env\Scripts\activate.bat (cmd) or .\env\Scripts\Activate.ps1 (powershell) (Windows)

You can also add as a second remote my fork https://github.com/kistlin/xknx. Works happens on usb-interface-support.

After you created your environment and installed the packages as mentioned in the README.md you can try and run

python examples/usb/example_telegram_monitor.py --filter "0/0/*"

and already check if you see something happening. (you might want to change KNX address to match your setup)

[rnixx]

Hi @farmio, what's the status of this issue and https://github.com/XKNX/xknx/tree/usb-interface-support? What's missing to get this done and upstream?

[rnixx]

I just tried to merge recent main to usb-interface-support here https://github.com/rnixx/xknx/tree/rnixx-usb-interface-support

knx_hid_helper_test.TestKNXtoCEMI.test_process fails now, looks like some new information from CEMIMessageCode is considered in the meantime. Either test frames have improper format or some hid related special handling is missing.

[farmio]

@rnixx Hi 👋!

What's missing to get this done and upstream?

Tbh I don't remember the exact last state of this, but you may read it in the previous comments. It didn't work on my interface at all since it doesn't support CEMI, so this would be needed to check and raise an exception (or translate CEMI to EMI1 if this is possible).

I don't know if anyone is still working on this.