Version 2.8.2 breaks listening on anything more than localhost

[tbclark3]

On Fedora 39, upgrading from nut 2.8.0 to 2.8.2 with LISTEN ::3493 causes all connections to be refused including localhost. Commenting out the LISTEN configuration results in upsd listening on localhost but makes remote monitoring impossible. Downgrading to 2.8.0 results in normal operation.

[jimklimov]

Did you try LISTEN * 3493? The spelling in your example does not make much sense, ::1 is a shortcut for (IPv6-only) localhost. Perhaps ::3493 is also interpreted as a weird IP address number.

Also, did you have any other LISTEN directives, before or after that one? There were some parsing mistakes addressed in recent releases e.g. due to order of ipv4 and ipv6 lines...

[tbclark3]

Sorry, there should have been a space between :: and 3493, ie ":: 3493". I am not using ::1. The :: interface binds to all ipv6 and ipv4 addresses. No, there are no other LISTEN directives.

[jimklimov]

Gotta check about :: then. Not sure this string's use was documented or implemented for NUT, but maybe I misremember. The * should fill that role.

[jimklimov]

Looked at standards while commuting now, and did not find any examples of double-colons as a standalone token in IPv6 - only that it can be used (at most once) in place of a consecutive block of zero bits. In that vein, an :: all-zero address makes sense as the wildcard, sinilar to 0.0.0.0 in IPv4, but I can't find examples or standards about that use of the string. Pointers to some RFC would be welcome :)

Wondering now, if it working this way in older NUT releases was by a coincidence (and/or part of another bug) rather than by design.

[jimklimov]

Ok, I stand corrected with my notes in issue #2012 - it had mentions of LISTEN :: and LISTEN ::0 as any-address (for IPv6 at least).

There was a lengthy community discussion about whether dual-support of IPv4 and IPv6 with one address spec is good, bad, portable, or outright insecure and best to be avoided. But at least for IPv6 it should have worked for you.

What did upsd logs say (perhaps with bumped debug_min setting)? Did netstat -an | grep 3493 (or equivalent in your OS/distro) show at least some listener?

[tbclark3]

Thank you for your very prompt response! The logs do not give any errors, and upsd does not show any listeners with version 2.8.2.

With version 2.8.0, and LISTENER :: 3493, upsd listens as follows:

tcp6 0 0 :::3493 :::* LISTEN 1447/upsd

and responds to ipv4 as well as ipv6.

I remember being surprised by the space in the LISTEN directives when I first configured nut several years ago, but it worked, and it has continued to work across upgrades until now.

If you're shooting for compliance with standards (as well as intuitiveness), you should probably accept
LISTEN :::3493 and LISTEN 0.0.0.0:3493 for dual stack listening and ipv4-only listening.

The asterisk in a listen address is unusual and I never would have thought of trying it, although it represents "any port" in Linux. In upsd.conf, it appears to be translated to 0.0.0.0 and so excludes ipv6.

[jimklimov]

We've had some quite outspoken community members from BSD and other secure backgrounds, who argued with examples against implicit enablement of IPv4 listeners when asked for IPv6 tokens. Many systems do not implement dual-stack at all, for a reason (e.g. overlooked firewall rules on the other stack).

Listening on asterisk as wildcard address is rather popular (since Apache httpd IIRC, maybe before it; many netstat or route implementations display it in this sense too). It also lacks connotations of IPv4-only vs. IPv6-only spelling of an address spec.

Documented at https://github.com/networkupstools/nut/pull/2013/files#diff-7ee66c4f1536ac84dc5bbff1b8312e2eef24b974b3e48a5c5c2bcfdf2eb8f3ce

[jimklimov]

Double-checked (with current master, but there were no changes since 2.8.2 release in this regard) -- seems to work as designed and documented:

:; touch /tmp/upsd.users /tmp/ups.conf
:; echo 'LISTEN :: 3493' > /tmp/upsd.conf
:; ALLOW_NO_DEVICE=yes NUT_STATEPATH=/tmp NUT_CONFPATH=/tmp ./server/upsd -DDDDDD &
Network UPS Tools upsd 2.8.2-137-g5b7ad929e
   0.000000     fopen /tmp/upsd.pid: No such file or directory
   0.000038     Could not find PID file '/tmp/upsd.pid' to see if previous upsd instance is already running!
   0.000256     /tmp/upsd.conf is world readable
   0.000338     [D3] listen_add: added :::3493
   0.000376     [D1] debug level is '6'
   0.000393     [D3] setuptcp: try to bind to :: port 3493
   0.000461     listening on :: port 3493
   0.000498     [D1] server_load: tried to set up 1 listening sockets, succeeded with 1
   0.000532     [D3] server_load: ...of those related to localhost: overall: 0 tried, 0 succeeded; by name: 0T/0S; by name(6): 0T/0S; by IPv4 addr: 0T/0S; by IPv6 addr: 0T/0S
   0.000570     [D1] Can not become_user(nobody): not root initially, remaining UID=1000 GID=1000
   0.000610     [D1] chdired into statepath /tmp for driver sockets
   0.000642     /tmp is world readable
   0.000687     Warning: no UPS definitions in ups.conf
   0.000721     Normally at least one UPS must be defined in ups.conf, currently there are none (please configure the file and reload the service)
   0.000769     /usr/local/ups/share/cmdvartab not found - disabling descriptions
   0.000803     /tmp/upsd.users is world readable
   0.000922     Running as foreground process, not saving a PID file
   0.000956     upsnotify: notify about state 2 with libsystemd: was requested, but not running as a service unit now, will not spam more about it
   0.001029     upsnotify: failed to notify about state 2: no notification tech defined, will not spam more about it
   0.001061     upsnotify: logged the systemd watchdog situation once, will not spam more about it
   0.001097     [D2] mainloop: polling 1 filedescriptors
   2.003312     [D2] mainloop: no data available
   2.003482     [D2] mainloop: polling 1 filedescriptors
...

:; ps -ef | grep upsd
jim      79208 79057  0 12:41 pts/5    00:00:00 ./server/upsd -DDDDDD

:; netstat -anp | grep 3493
tcp6       0      0 :::3493                 :::*                    LISTEN      79208 ./server/upsd

It indeed listens on only IPv6:

:; (echo "VER" ; echo "LOGOUT" ) | nc 127.0.0.1 3493 ; echo $?
1

:; (echo "VER" ; echo "LOGOUT" ) | nc ::1 3493 ; echo $?
 416.231547     [D2] Connect from ::1
 416.231608     [D2] mainloop: polling 2 filedescriptors
 416.231658     [D6] Entering check_command: VER
 416.231695     [D6] check_command: Calling command handler for VER
 416.231771     [D2] write: [destfd=4] [len=79] [Network UPS Tools upsd 2.8.2-137-g5b7ad929e - https://www.networkupstools.org/]
Network UPS Tools upsd 2.8.2-137-g5b7ad929e - https://www.networkupstools.org/
 416.231809     [D6] Entering check_command: LOGOUT
 416.231851     [D6] check_command: Calling command handler for LOGOUT
 416.231921     [D2] write: [destfd=4] [len=11] [OK Goodbye]
OK Goodbye
 416.231963     [D2] Disconnect from ::1
 416.232022     [D2] mainloop: polling 1 filedescriptors
0

[jimklimov]

Likewise, asterisk (still) works for two explicit listeners here:

:; echo 'LISTEN * 3493' > /tmp/upsd.conf
:; ALLOW_NO_DEVICE=yes NUT_STATEPATH=/tmp NUT_CONFPATH=/tmp ./server/upsd -DDDDDD &

Network UPS Tools upsd 2.8.2-137-g5b7ad929e
   0.000000     fopen /tmp/upsd.pid: No such file or directory
   0.000040     Could not find PID file '/tmp/upsd.pid' to see if previous upsd instance is already running!
   0.000246     /tmp/upsd.conf is world readable
   0.000366     [D3] listen_add: added *:3493
   0.000409     [D1] debug level is '6'
   0.000452     [D3] setuptcp: try to bind to * port 3493
   0.000467     [D1] setuptcp: handling 'LISTEN * 3493' with IPv4 any-address support
   0.000503     [D1] setuptcp: handling 'LISTEN * 3493' with IPv6 any-address support
   0.000557     [D3] setuptcp: try to bind to ::0 port 3493
   0.000660     listening on ::0 port 3493
   0.000716     [D3] setuptcp: try taking IPv4 'ANY' (if dual-stack IPv6 'ANY' did not grab it)
   0.000761     [D3] setuptcp: try to bind to 0.0.0.0 port 3493
   0.000836     listening on 0.0.0.0 port 3493
   0.000847     [D3] setuptcp: remembering IPv4 'ANY' instead of 'LISTEN *'
   0.000962     [D3] setuptcp: also remembering IPv6 'ANY' instead of 'LISTEN *'
   0.001057     [D6] setuptcp: SKIP bind to ::0 port 3493: entry already initialized
   0.001107     [D1] server_load: tried to set up 2 listening sockets, succeeded with 2
   0.001159     [D3] server_load: ...of those related to localhost: overall: 0 tried, 0 succeeded; by name: 0T/0S; by name(6): 0T/0S; by IPv4 addr: 0T/0S; by IPv6 addr: 0T/0S
   0.001195     [D1] Can not become_user(nobody): not root initially, remaining UID=1000 GID=1000
   0.001239     [D1] chdired into statepath /tmp for driver sockets
   0.001274     /tmp is world readable
   0.001324     Warning: no UPS definitions in ups.conf
   0.001362     Normally at least one UPS must be defined in ups.conf, currently there are none (please configure the file and reload the service)
   0.001416     /usr/local/ups/share/cmdvartab not found - disabling descriptions
   0.001452     /tmp/upsd.users is world readable
   0.001501     Running as foreground process, not saving a PID file
   0.001541     upsnotify: notify about state 2 with libsystemd: was requested, but not running as a service unit now, will not spam more about it
   0.001596     upsnotify: failed to notify about state 2: no notification tech defined, will not spam more about it
   0.001616     upsnotify: logged the systemd watchdog situation once, will not spam more about it
   0.001655     [D2] mainloop: polling 2 filedescriptors
   2.003771     [D2] mainloop: no data available
   2.003859     [D2] mainloop: polling 2 filedescriptors

:; netstat -anp | grep 3493
tcp        0      0 0.0.0.0:3493            0.0.0.0:*               LISTEN      79252 ./server/upsd
tcp6       0      0 :::3493                 :::*                    LISTEN      79252 ./server/upsd

And this time it accepts both types of connections:

:; (echo "VER" ; echo "LOGOUT" ) | nc 127.0.0.1 3493 ; echo $?
...
 168.195121     [D2] mainloop: no data available
 168.195300     [D2] mainloop: polling 2 filedescriptors

 168.778789     [D2] Connect from 127.0.0.1
 168.778845     [D2] mainloop: polling 3 filedescriptors
 168.778863     [D6] Entering check_command: VER
 168.778898     [D6] check_command: Calling command handler for VER
 168.779118     [D2] write: [destfd=5] [len=79] [Network UPS Tools upsd 2.8.2-137-g5b7ad929e - https://www.networkupstools.org/]
Network UPS Tools upsd 2.8.2-137-g5b7ad929e - https://www.networkupstools.org/
 168.779242     [D6] Entering check_command: LOGOUT
 168.779351     [D6] check_command: Calling command handler for LOGOUT
 168.779439     [D2] write: [destfd=5] [len=11] [OK Goodbye]
OK Goodbye
 168.779488     [D2] Disconnect from 127.0.0.1
 168.779603     [D2] mainloop: polling 2 filedescriptors
0

:; (echo "VER" ; echo "LOGOUT" ) | nc ::1 3493 ; echo $?
 180.793701     [D2] mainloop: no data available
 180.793882     [D2] mainloop: polling 2 filedescriptors

 180.843138     [D2] Connect from ::1
 180.843193     [D2] mainloop: polling 3 filedescriptors
 180.843212     [D6] Entering check_command: VER
 180.843245     [D6] check_command: Calling command handler for VER
 180.843320     [D2] write: [destfd=5] [len=79] [Network UPS Tools upsd 2.8.2-137-g5b7ad929e - https://www.networkupstools.org/]
Network UPS Tools upsd 2.8.2-137-g5b7ad929e - https://www.networkupstools.org/
 180.843371     [D6] Entering check_command: LOGOUT
 180.843407     [D6] check_command: Calling command handler for LOGOUT
 180.843482     [D2] write: [destfd=5] [len=11] [OK Goodbye]
OK Goodbye
 180.843521     [D2] Disconnect from ::1
 180.843588     [D2] mainloop: polling 2 filedescriptors
0

Also on non-localhost (per ip a):

:; ip a
...
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
...
    inet 172.28.79.185/20 brd 172.28.79.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.22.48.99/20 scope global eth0
       valid_lft forever preferred_lft forever
...

:; (echo "VER" ; echo "LOGOUT" ) | nc 172.28.79.185 3493 ; echo $?
 442.039740     [D2] Connect from 172.28.79.185
 442.039797     [D2] mainloop: polling 3 filedescriptors
 442.039843     [D6] Entering check_command: VER
 442.039879     [D6] check_command: Calling command handler for VER
 442.039949     [D2] write: [destfd=5] [len=79] [Network UPS Tools upsd 2.8.2-137-g5b7ad929e - https://www.networkupstools.org/]
Network UPS Tools upsd 2.8.2-137-g5b7ad929e - https://www.networkupstools.org/
 442.039991     [D6] Entering check_command: LOGOUT
 442.040028     [D6] check_command: Calling command handler for LOGOUT
 442.040092     [D2] write: [destfd=5] [len=11] [OK Goodbye]
OK Goodbye
 442.040131     [D2] Disconnect from 172.28.79.185
 442.040198     [D2] mainloop: polling 2 filedescriptors
0

[jimklimov]

Also, for the sake of completeness, an empty configuration file (no LISTEN directive) causes listening on one or both localhosts as available - as documented (e.g. in augeas lenses):

:; rm -rf /tmp/nnnn; mkdir /tmp/nnnn
:; touch /tmp/nnnn/ups.conf /tmp/nnnn/upsd.users /tmp/nnnn/upsd.conf

:; ALLOW_NO_DEVICE=yes NUT_STATEPATH=/tmp/nnnn NUT_CONFPATH=/tmp/nnnn ./server/upsd -DDDDDD &
Network UPS Tools upsd 2.8.2-137-g3c19d79de
   0.000000     fopen /tmp/nnnn/upsd.pid: No such file or directory
   0.000037     Could not find PID file '/tmp/nnnn/upsd.pid' to see if previous upsd instance is already running!
   0.000160     /tmp/nnnn/upsd.conf is world readable
   0.000229     [D1] debug level is '6'
   0.000268     [D1] server_load: No LISTEN configuration provided, will try IPv6 localhost
   0.000304     [D3] listen_add: added ::1:3493
   0.000414     [D1] server_load: No LISTEN configuration provided, will try IPv4 localhost
   0.000468     [D3] listen_add: added 127.0.0.1:3493
   0.000506     [D3] setuptcp: try to bind to ::1 port 3493
   0.000579     listening on ::1 port 3493
   0.000615     [D3] setuptcp: try to bind to 127.0.0.1 port 3493
   0.000662     listening on 127.0.0.1 port 3493
   0.000699     [D1] server_load: tried to set up 2 listening sockets, succeeded with 2
   0.000735     [D3] server_load: ...of those related to localhost: overall: 2 tried, 2 succeeded; by name: 0T/0S; by name(6): 0T/0S; by IPv4 addr: 1T/1S; by IPv6 addr: 1T/1S
   0.000773     [D1] Can not become_user(nobody): not root initially, remaining UID=1000 GID=1000
   0.000811     [D1] chdired into statepath /tmp/nnnn for driver sockets
   0.000847     /tmp/nnnn is world readable
   0.000931     Warning: no UPS definitions in ups.conf
   0.000980     Normally at least one UPS must be defined in ups.conf, currently there are none (please configure the file and reload the service)
   0.001043     /usr/local/ups/share/cmdvartab not found - disabling descriptions
   0.001083     /tmp/nnnn/upsd.users is world readable
   0.001127     Running as foreground process, not saving a PID file
   0.001164     upsnotify: notify about state 2 with libsystemd: was requested, but not running as a service unit now, will not spam more about it
   0.001206     upsnotify: failed to notify about state 2: no notification tech defined, will not spam more about it
   0.001240     upsnotify: logged the systemd watchdog situation once, will not spam more about it
   0.001280     [D2] mainloop: polling 2 filedescriptors

:; netstat -anp | grep 3493
tcp        0      0 127.0.0.1:3493          0.0.0.0:*               LISTEN      31156 ./server/upsd
tcp6       0      0 ::1:3493                :::*                    LISTEN      31156 ./server/upsd

[jimklimov]

@tbclark3 : While it would be interesting to reproduce and understand/root-cause the original issue you've had, I can not confirm it as related to LISTEN directive in particular. Maybe there was something else wrong in your configuration (e.g. absent devices in ups.conf are a problem by default - but since NUT v2.8.0 there are options to allow that situation), or permissions problems, or some missing config files altogether (at least ups.conf, upsd.conf and upsd.users must be there even if empty), or port occupied by someone else...

FWIW, I've crafted a case where upsd fails to LISTEN (on occupied and otherwise privileged port 22 for sshd) - and it reasonably refused to start:

:; echo 'LISTEN * 22' > /tmp/nnnn/upsd.conf
:; ALLOW_NO_DEVICE=yes NUT_STATEPATH=/tmp/nnnn NUT_CONFPATH=/tmp/nnnn ./server/upsd -DDDDDD ; echo $?
Network UPS Tools upsd 2.8.2-137-g3c19d79de
   0.000000     fopen /tmp/nnnn/upsd.pid: No such file or directory
   0.000035     Could not find PID file '/tmp/nnnn/upsd.pid' to see if previous upsd instance is already running!
   0.000137     /tmp/nnnn/upsd.conf is world readable
   0.000225     [D3] listen_add: added *:22
   0.000285     [D1] debug level is '6'
   0.000300     [D3] setuptcp: try to bind to * port 22
   0.000312     [D1] setuptcp: handling 'LISTEN * 22' with IPv4 any-address support
   0.000323     [D1] setuptcp: handling 'LISTEN * 22' with IPv6 any-address support
   0.000332     [D3] setuptcp: try to bind to ::0 port 22
   0.000371     [D3] setuptcp: bind: Permission denied
   0.000428     not listening on ::0 port 22
   0.000476     [D3] setuptcp: Could not bind to ::0:22 trying to handle a 'LISTEN *' directive
   0.000539     [D3] setuptcp: try taking IPv4 'ANY'
   0.000583     [D3] setuptcp: try to bind to 0.0.0.0 port 22
   0.000636     [D3] setuptcp: bind: Permission denied
   0.000649     not listening on 0.0.0.0 port 22
   0.000682     [D3] setuptcp: Could not bind to IPv4 0.0.0.0:22
   0.000691     Handling of 'LISTEN * 22' directive failed to bind to 'ANY' address
1

:; netstat -anp | grep 3493
### empty

(could be a bit more verbose about aborting though).

NOTE: This failure should have been seen also without debug:

:; ALLOW_NO_DEVICE=yes NUT_STATEPATH=/tmp/nnnn NUT_CONFPATH=/tmp/nnnn ./server/upsd ; echo $?
Network UPS Tools upsd 2.8.2-137-g3c19d79de
Could not find PID file '/tmp/nnnn/upsd.pid' to see if previous upsd instance is already running!
/tmp/nnnn/upsd.conf is world readable
not listening on ::0 port 22
not listening on 0.0.0.0 port 22
Handling of 'LISTEN * 22' directive failed to bind to 'ANY' address
1

[jimklimov]

Just in case, checked what it would do if ::3493 were one token (as typo'ed in the original post). Also tested the new messages from PR #2418.

Little surprise here: the string is seen as an IP address, happens to not be available on the current system (port 3493 is separate from that). Lack of a listener is fatal:

:; echo 'LISTEN ::3493' > /tmp/nnnn/upsd.conf
:; ALLOW_NO_DEVICE=yes NUT_STATEPATH=/tmp/nnnn NUT_CONFPATH=/tmp/nnnn ./server/upsd -DDDDDD ; echo $?
Network UPS Tools upsd 2.8.2
   0.000000     fopen /tmp/nnnn/upsd.pid: No such file or directory
   0.000040     Could not find PID file '/tmp/nnnn/upsd.pid' to see if previous upsd instance is already running!
   0.000165     WARNING: /tmp/nnnn/upsd.conf is world readable (hope you don't have passwords there)
   0.000234     [D3] listen_add: added ::3493:3493
   0.000284     [D1] debug level is '6'
   0.000335     [D3] setuptcp: try to bind to ::3493 port 3493
   0.000390     [D3] setuptcp: bind: Cannot assign requested address
   0.000411     not listening on ::3493 port 3493
   0.000451     [D1] server_load: tried to set up 1 listening sockets, succeeded with 0
   0.000493     [D3] server_load: ...of those related to localhost: overall: 0 tried, 0 succeeded; by name: 0T/0S; by name(6): 0T/0S; by IPv4 addr: 0T/0S; by IPv6 addr: 0T/0S
   0.000533     no listening interface available
   0.000576     [D1] upsd_cleanup: starting the end-game
   0.000615     [D1] upsd_cleanup: finished
1

[jimklimov]

tcp6 0 0 :::3493 :::* LISTEN 1447/upsd
I remember being surprised by the space in the LISTEN directives ...

Thinking of it... might make sense to parse a single token on a LISTEN <token1>:<token2> (with at least one colon character), as an addr-colon-port so spelling of LISTEN :::3493 would get split into :: and 3493, and some LISTEN 127.0.0.1:3493 would become 127.0.0.1 and 3493.

That said, in ~25 years of NUT's existence I think this is the first time such issue was raised (to the best of my personal knowledge) - so I guess it is a rare surprise after all the RTFM :)

[desertwitch]

I've also tested around with this on current master and all works as documented.

[jimklimov]

@tbclark3 : I am afraid we could not reproduce your issue, as well as explain it theoretically. Seems like some setup misconfiguration on your side (e.g. your expectations for LISTEN differed from the config syntax used and documented by NUT for decades).

Closing the issue, but if you find some more specific clues about the current implementation not working as documented - please do let us know and feel free to reopen :)