Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixops closes SSH connection when running nixops check #1571

Open
datafoo opened this issue Nov 14, 2023 · 1 comment
Open

nixops closes SSH connection when running nixops check #1571

datafoo opened this issue Nov 14, 2023 · 1 comment

Comments

@datafoo
Copy link

datafoo commented Nov 14, 2023

When running nixops check, I sometime observe the Connection closed by authenticating user root log on the target machines. When this happens nixops check displays Up = No for the corresponding machines. But the real problem is that, if I retry multiple times, fail2ban bans my IP address because of Connection closed by authenticating user root:

Nov 14 09:15:12 mymachine sshd[1320]: Connection from 192.168.1.102 port 54986 on 192.168.1.50 port 22 rdomain ""
Nov 14 09:15:13 mymachine sshd[1320]: Accepted key ED25519 SHA256:******************************************* found at /etc/ssh/authorized_keys.d/root:1
Nov 14 09:15:13 mymachine sshd[1320]: Postponed publickey for root from 192.168.1.102 port 54986 ssh2 [preauth]
Nov 14 09:15:13 mymachine sshd[1320]: Accepted key ED25519 SHA256:******************************************* found at /etc/ssh/authorized_keys.d/root:1
Nov 14 09:15:13 mymachine sshd[1320]: Accepted publickey for root from 192.168.1.102 port 54986 ssh2: ED25519 SHA256:*******************************************
Nov 14 09:15:13 mymachine sshd[1320]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Nov 14 09:15:13 mymachine sshd[1327]: Connection from 192.168.1.102 port 54992 on 192.168.1.50 port 22 rdomain ""
Nov 14 09:15:14 mymachine sshd[1327]: Connection closed by authenticating user root 192.168.1.102 port 54992 [preauth]
Nov 14 09:15:14 mymachine fail2ban.filter[989]: INFO [sshd] Found 192.168.1.102 - 2023-11-14 09:15:14

This was done running NixOps 2.0.0-pre-fc9b55c

Why is NixOps closing the SSH connection?

I have not been able to consistently reproduce the problem but I can say that it happens very often when the target machine is a Hetzner Cloud CX11 virtual machine.

@datafoo
Copy link
Author

datafoo commented Nov 14, 2023

It is probably worth specifying the output of nixops check which shows some special:

[me@laptop:~/dev/mydeployments/deployment1.example.com]$ nixops check -d deployment1.example.com --include mymachine
Machines state:
+-----------+--------+----+-----------+----------+-----------+-------+-------+
| Name      | Exists | Up | Reachable | Disks OK | Load avg. | Units | Notes |
+-----------+--------+----+-----------+----------+-----------+-------+-------+
| mymachine | Yes    | No | N/A       | N/A      |           |       |       |
+-----------+--------+----+-----------+----------+-----------+-------+-------+
Non machines resources state:
+------+--------+
| Name | Exists |
+------+--------+
+------+--------+
mux_client_request_session: read from master failed: Broken pipe

[me@laptop:~/dev/mydeployments/deployment1.example.com]$ no such identity: /tmp/nix-shell.hgG5u9/nixops-tmp8cuxc17o/id_nixops-mymachine: No such file or directory
[email protected]: Permission denied (publickey).

The no such identity: /tmp/nix-shell.hgG5u9/nixops-tmp8cuxc17o/id_nixops-mymachine: No such file or directory [email protected]: Permission denied (publickey). part is not something I type myself, it just pops up in the console after 1 second or 2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant