mqttwarn does not like py39-paho-mqtt-2.0.0 #694

dlangille · 2024-03-02T12:27:59Z

I've been unable to get mqttwarn to start today after updating py39-paho-mqtt: 1.6.1 -> 2.0.0

Reverting to 1.6.1 gets it running again.

Seem related to https://stackoverflow.com/questions/77984857/paho-mqtt-unsupported-callback-api-version-error?noredirect=1

[12:20 mqtt01 dvl /usr/local/etc/rc.d] % sudo /usr/local/bin/mqttwarn --config-file=/usr/local/etc/mqttwarn/mqttwarn.ini
2024-03-02 12:20:58,853 INFO     [mqttwarn.commands         ] Starting mqttwarn
2024-03-02 12:20:58,853 INFO     [mqttwarn.commands         ] Log level is DEBUG
2024-03-02 12:20:58,853 DEBUG    [mqttwarn.core             ] Trying to load built-in service "file" from "file"
2024-03-02 12:20:58,854 DEBUG    [mqttwarn.core             ] Trying to load service "file" from file "/usr/local/lib/python3.9/site-packages/mqttwarn/services/file.py"
2024-03-02 12:20:58,855 INFO     [mqttwarn.core             ] Successfully loaded service "file"
2024-03-02 12:20:58,855 DEBUG    [mqttwarn.core             ] Trying to load built-in service "pushover" from "pushover"
2024-03-02 12:20:58,855 DEBUG    [mqttwarn.core             ] Trying to load service "pushover" from file "/usr/local/lib/python3.9/site-packages/mqttwarn/services/pushover.py"
2024-03-02 12:20:58,985 INFO     [mqttwarn.core             ] Successfully loaded service "pushover"
2024-03-02 12:20:58,986 DEBUG    [mqttwarn.core             ] Trying to load built-in service "log" from "log"
2024-03-02 12:20:58,986 DEBUG    [mqttwarn.core             ] Trying to load service "log" from file "/usr/local/lib/python3.9/site-packages/mqttwarn/services/log.py"
2024-03-02 12:20:58,986 INFO     [mqttwarn.core             ] Successfully loaded service "log"
Traceback (most recent call last):
  File "/usr/local/bin/mqttwarn", line 8, in <module>
    sys.exit(run())
  File "/usr/local/lib/python3.9/site-packages/mqttwarn/commands.py", line 93, in run
    run_mqttwarn(configfile=options["--config-file"])
  File "/usr/local/lib/python3.9/site-packages/mqttwarn/commands.py", line 155, in run_mqttwarn
    subscribe_forever()
  File "/usr/local/lib/python3.9/site-packages/mqttwarn/core.py", line 688, in subscribe_forever
    mqttc = connect()
  File "/usr/local/lib/python3.9/site-packages/mqttwarn/core.py", line 642, in connect
    mqttc = paho.Client(cf.clientid, clean_session=cf.cleansession, protocol=cf.protocol)
  File "/usr/local/lib/python3.9/site-packages/paho/mqtt/client.py", line 766, in __init__
    raise ValueError(
[12:20 mqtt01 dvl /usr/local/etc/rc.d] % sudo /usr/local/bin/mqttwarn --config-file=/usr/local/etc/mqttwarn/mqttwarn.ini
2024-03-02 12:20:58,853 INFO     [mqttwarn.commands         ] Starting mqttwarn
2024-03-02 12:20:58,853 INFO     [mqttwarn.commands         ] Log level is DEBUG
2024-03-02 12:20:58,853 DEBUG    [mqttwarn.core             ] Trying to load built-in service "file" from "file"
2024-03-02 12:20:58,854 DEBUG    [mqttwarn.core             ] Trying to load service "file" from file "/usr/local/lib/python3.9/site-packages/mqttwarn/services/file.py"
2024-03-02 12:20:58,855 INFO     [mqttwarn.core             ] Successfully loaded service "file"
2024-03-02 12:20:58,855 DEBUG    [mqttwarn.core             ] Trying to load built-in service "pushover" from "pushover"
2024-03-02 12:20:58,855 DEBUG    [mqttwarn.core             ] Trying to load service "pushover" from file "/usr/local/lib/python3.9/site-packages/mqttwarn/services/pushover.py"
2024-03-02 12:20:58,985 INFO     [mqttwarn.core             ] Successfully loaded service "pushover"
2024-03-02 12:20:58,986 DEBUG    [mqttwarn.core             ] Trying to load built-in service "log" from "log"
2024-03-02 12:20:58,986 DEBUG    [mqttwarn.core             ] Trying to load service "log" from file "/usr/local/lib/python3.9/site-packages/mqttwarn/services/log.py"
2024-03-02 12:20:58,986 INFO     [mqttwarn.core             ] Successfully loaded service "log"
Traceback (most recent call last):
  File "/usr/local/bin/mqttwarn", line 8, in <module>
    sys.exit(run())
  File "/usr/local/lib/python3.9/site-packages/mqttwarn/commands.py", line 93, in run
    run_mqttwarn(configfile=options["--config-file"])
  File "/usr/local/lib/python3.9/site-packages/mqttwarn/commands.py", line 155, in run_mqttwarn
    subscribe_forever()
  File "/usr/local/lib/python3.9/site-packages/mqttwarn/core.py", line 688, in subscribe_forever
    mqttc = connect()
  File "/usr/local/lib/python3.9/site-packages/mqttwarn/core.py", line 642, in connect
    mqttc = paho.Client(cf.clientid, clean_session=cf.cleansession, protocol=cf.protocol)
  File "/usr/local/lib/python3.9/site-packages/paho/mqtt/client.py", line 766, in __init__
    raise ValueError(
ValueError: Unsupported callback API version: version 2.0 added a callback_api_version, see migrations.md for details

The text was updated successfully, but these errors were encountered:

jpmens · 2024-03-02T12:31:37Z

To be expected; paho mqtt has broken most programs with their 2.0; quite sad. I don’t think we’ve yet added support for it.

adequacy1 · 2024-03-25T08:28:31Z

core.py:

- mqttc = paho.Client(cf.clientid, clean_session=cf.cleansession, protocol=cf.protocol)
+ mqttc = paho.Client(paho.CallbackAPIVersion.VERSION1, cf.clientid, clean_session=cf.cleansession, protocol=cf.protocol)

jpmens · 2024-03-25T08:42:57Z

Yes, Paho broke that very badly, and it affects hundreds of peoples' programs. If you care to produce a PR we'll take it.

amotl · 2024-03-27T23:49:44Z

omg. Thanks for letting us know, and apologies for the late reply. We will take corresponding steps to remedy the problem in one way or another.

Paho™ MQTT Python v2.0.0, released on Feb 10 2024, includes breaking changes. See migration instructions for more details on how to upgrade. The release also introduces new documentation available online.

-- https://github.com/eclipse/paho.mqtt.python/releases/tag/v2.0.0

amotl · 2024-03-28T00:43:25Z

At least the dependency version pinning in project metadata is right. So, when package managers are taking them into account, nothing should break from one day to the other accidentally?

mqttwarn/setup.py

Line 21 in e096289

"paho-mqtt<2",

NB: I am just trying to assess the seriousness and impact of the issue. @dlangille: Did you upgrade py39-paho-mqtt to the next major version deliberately, or did it happen automatically? Is it possible for BSD to have both versions at the same time, or are you currently destined to exclusively run the previous version of paho mqtt, because of this situation, while you actually would like to upgrade?

amotl · 2024-03-28T00:57:40Z

@dlangille: Relaying the error you reported....

ValueError: Unsupported callback API version: version 2.0 added a callback_api_version, see migrations.md for details

... it probably relates to the topmost item from the v2.0.0 release notes:

Caution

BREAKING: Added callback_api_version. This breaks ALL users of paho-mqtt's Client class.

See migrations.rst for details on how to upgrade.
tl; dr; add CallbackAPIVersion.VERSION1 to first argument of Client().

Fixing this sounds easy.

amotl · 2024-03-28T01:01:58Z

Thoughts

There might be issues with 2.0.0 yet, for example AttributeError: ‘MQTTClient’ object has no attribute ‘_sock’, see below.
Not sure if we should better wait upgrading until a 2.0.3 version will be released? Maybe wait for 2.0.1 at least?
Sure enough, a patch can be prepared already, that can be used at everyones disposal.

References

Downstream

amotl · 2024-03-29T04:35:51Z

I've just submitted a patch to remedy the breaking change on the most prominent spot, based on learnings from mqtt-tools/pytest-mqtt#18.

Remedy breaking change with 2.x: Use CallbackAPIVersion.VERSION1 as a default eclipse/paho.mqtt.python#831

However, I don't think that patch will be accepted, as the authors seem to pretty strong about this detail?

paho-mqtt-2.0.0 client issue eclipse/paho.mqtt.python#814 (comment) ff.

amotl · 2024-04-07T18:57:40Z

Hi. It looks like my patch might get accepted, helping users in basic use case scenarios to upgrade to Paho MQTT client version 2 without a breaking change. Naturally, it will not work in more advanced cases, as outlined by @PierreF on the same discussion. Thanks!

Remedy breaking change with 2.x: Use CallbackAPIVersion.VERSION1 as a default eclipse/paho.mqtt.python#831

amotl · 2024-04-22T21:31:14Z

It looks like paho-mqtt v2 is causing serious havoc throughout the community. That is natural with breaking changes, and apparently, much of what is behind is about the proper transitioning to support MQTTv5, which makes things different under the hood.

However, I did not dig deeper, and, in this spirit, I am not able to evaluate the situation better than submitting that humble patch, and hoping for the best outcomes in the long run. That such an event feels annoying, is perfectly reasonable. However, sometimes such changes must happen, and all we can take away from that is that all users of packages should apply version pinning properly, in order not to run into such situations. In this case, paho-mqtt<2.

paho-mqtt-2.0.0 client issue eclipse/paho.mqtt.python#814

amotl · 2024-04-22T21:34:44Z

For mqttwarn, I think it is sensible to wait for any further changes happen on paho-mqtt. Because mqttwarn is correctly applying the version pinning, I guess there is not much else to do at this point in time? Please let us know if you think differently.

mqttwarn/setup.py

Line 21 in 9d61fc5

"paho-mqtt<2",

This was referenced Mar 29, 2024

Update paho-mqtt requirement from <2 to <3 mqtt-tools/pytest-mqtt#12

Closed

Add support for paho-mqtt 2.x, retaining compatibility for 1.x mqtt-tools/pytest-mqtt#18

Merged

amotl mentioned this issue Apr 14, 2024

Modularization, documentation, installation, configuration, and naming things hiveeyes/ds18b20-datalogger#1

Closed

amotl mentioned this issue Apr 22, 2024

Chore(deps): Update paho-mqtt requirement from <2 to <3 hiveeyes/ds18b20-datalogger#4

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mqttwarn does not like py39-paho-mqtt-2.0.0 #694

mqttwarn does not like py39-paho-mqtt-2.0.0 #694

dlangille commented Mar 2, 2024

jpmens commented Mar 2, 2024 via email

adequacy1 commented Mar 25, 2024 •

edited

jpmens commented Mar 25, 2024 via email

amotl commented Mar 27, 2024 •

edited

amotl commented Mar 28, 2024 •

edited

amotl commented Mar 28, 2024

amotl commented Mar 28, 2024 •

edited

amotl commented Mar 29, 2024

amotl commented Apr 7, 2024

amotl commented Apr 22, 2024 •

edited

amotl commented Apr 22, 2024 •

edited

mqttwarn does not like py39-paho-mqtt-2.0.0 #694

mqttwarn does not like py39-paho-mqtt-2.0.0 #694

Comments

dlangille commented Mar 2, 2024

jpmens commented Mar 2, 2024 via email

adequacy1 commented Mar 25, 2024 • edited

jpmens commented Mar 25, 2024 via email

amotl commented Mar 27, 2024 • edited

amotl commented Mar 28, 2024 • edited

amotl commented Mar 28, 2024

amotl commented Mar 28, 2024 • edited

Thoughts

References

Downstream

amotl commented Mar 29, 2024

amotl commented Apr 7, 2024

amotl commented Apr 22, 2024 • edited

amotl commented Apr 22, 2024 • edited

adequacy1 commented Mar 25, 2024 •

edited

amotl commented Mar 27, 2024 •

edited

amotl commented Mar 28, 2024 •

edited

amotl commented Mar 28, 2024 •

edited

amotl commented Apr 22, 2024 •

edited

amotl commented Apr 22, 2024 •

edited