failed: listen tcp 0.0.0.0:22000: listen: address already in use

[bpopovich4]

We are using okteto on a kubernetes cluster. We have many users that are running jobs and we keep getting frozen pods. The initial sync seems fine, but alot of the time when the job completes on the kubernetes pod and is syncing data back it freezes. There are no errors on the containers or the server node that launched the job. I looked in the syncthing log and found below errors. I’m guess this means the address/port is in use because of another user. If this seems the case based on the logs, how do I change the default port and will I need to have each user use a different port? I don’t see a /config/syncthing anywhere and I’m thinking it’s because it come packaged with okteto. [ABKAV] 09:32:20 INFO: Listen (BEP/tcp): listen tcp 0.0.0.0:22000: listen: address already in use [ABKAV] 09:32:20 INFO: listenerSupervisor@tcp://0.0.0.0:22000: service tcp://0.0.0.0:22000 failed: listen tcp 0.0.0.0:22000: listen: address already in use [ABKAV] 09:32:20 INFO: Listen (BEP/tcp): listen tcp 0.0.0.0:22000: listen: address already in use [ABKAV] 09:32:20 INFO: listenerSupervisor@tcp://0.0.0.0:22000: service tcp://0.0.0.0:22000 failed: listen tcp 0.0.0.0:22000: listen: address already in use [ABKAV] 09:32:20 INFO: Listen (BEP/tcp): listen tcp 0.0.0.0:22000: listen: address already in use [ABKAV] 09:32:20 INFO: listenerSupervisor@tcp://0.0.0.0:22000: service tcp://0.0.0.0:22000 failed: listen tcp 0.0.0.0:22000: listen: address already in use [ABKAV] 09:32:33 INFO: Exiting

[jLopezbarb]

Hi @bpopovich4, I have several question to narrow down the error:

We are using okteto on a kubernetes cluster.

Does it have okteto chart installed? If so, which version it's installed.
Which CLI version are you using?

We have many users that are running jobs and we keep getting frozen pods

What do you mean by getting frozen pods? Do you receive any kind of error?
Do you mean kubernetes jobs? Are the job pods the ones that freeze?
If you are running okteto up, they could be a k8s deployment or sfs so I'll continue to ask questions assuming that.

I looked in the syncthing log and found below errors. I’m guess this means the address/port is in use because of another user.

Those errors seems fine, can you check the okteto logs generated by okteto doctor? There should be more information about a hidden error there.

[bpopovich4]

Thanks for responding. By frozen, the devs are on a server node, run okteto up and it launches a pod on the kubernetes cluster. The pods finishes the job and through the sync back to the node for the data, it just sits there and eventually dies. Some it is somewhere while it is writing data back to the original node. There are no errors in the pods or on the server in the logs. The pods don't freeze. They run the job to completion. Here is a copy with some things redacted of the okteto.yaml

name: xxx-$USER
namespace: xx
autocreate: true
image: repo.xxx.com/image
persistentVolume:
enabled: false
command: bash
workdir: /build
sync:

• .:/build
  #- .docker:/build/.docker
  #- platform:/build/platform
  #- src:/build/src
  #- bridge_apps:/build/bridge_apps
  #- name:/builder/name
  #- integration-tests:/build/integration-tests
  #securityContext:
  #runAsUser: 30000
  #runAsGroup: 30000
  #volumes:

- /opt/dir/apps/project/name_7_xxxxx

initContainer:
image: repo.xxx.com:9900/okteto/bin:1.3.5
resources:
requests:
cpu: "0"
memory: "12Gi"
ephemeral-storage: "12Gi"
limits:

[bpopovich4]

I have not heard of okteto doctor. I will look into that

[bpopovich4]

Is there a way to change the default port 22000 for syncthing? If I look at the syncthing documentation there is a syncthing.yaml. I don't see one of those though with the okteto install

[bpopovich4]

time="2023-04-13T09:31:43-06:00" level=info msg="retrying syncthing call[rest/events] local=false: failed to call syncthing [rest/events]: Get "http://localhost:41713/rest/events?events=FolderSummary&limit=1&since=0&timeout=0\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:31:48-06:00" level=info msg="retrying syncthing call[rest/events] local=false: failed to call syncthing [rest/events]: Get "http://localhost:41713/rest/events?events=FolderSummary&limit=1&since=0&timeout=0\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:31:53-06:00" level=info msg="error getting events: failed to call syncthing [rest/events]: Get "http://localhost:41713/rest/events?events=FolderSummary&limit=1&since=0&timeout=0\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=info msg="syncthing pull error local=true: syncing: no connected device has the required version of this file\n" action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=info msg="syncthing error local=true retry 4: docker/WRSDK_ls1043ardb.tar.bz2: syncing: no connected device has the required version of this file\n" action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=info msg="syncthing monitor error, sending disconnect signal: docker/WRSDK_ls1043ardb.tar.bz2: syncing: no connected device has the required version of this file\n" action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=info msg="starting shutdown sequence" action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=info msg="sent cancellation signal" action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=info msg="stopping syncthing" action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=info msg="ssh forward localhost:44441->0.0.0.0:22000 -> done" action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=debug msg="call to up.applyToApp cancelled: {}" action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=info msg="ssh forward localhost:41713->0.0.0.0:8384 -> done" action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=info msg="command failed: wait: remote command exited without exit status or exit signal" action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=info msg="ssh client for exec closed" action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=info msg="terminal restored" action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=info msg="syncthing ping error 0" action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=info msg="terminating syncthing 112179 without wait" action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=info msg="terminated syncthing 112179 without wait" action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=info msg="stopping forwarders" action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=info msg="stopped k8s forwarder" action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=info msg="stopped SSH forward manager" action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=info msg="completed shutdown sequence" action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=info msg="ssh forward localhost:41713->0.0.0.0:8384 -> failed to dial remote connection: ssh: unexpected packet in response to channel open: " action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=info msg="ssh forward localhost:41713->0.0.0.0:8384 -> failed to dial remote connection: ssh: unexpected packet in response to channel open: " action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=info msg="ssh forward localhost:41713->0.0.0.0:8384 -> failed to dial remote connection: ssh: unexpected packet in response to channel open: " action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=info msg="ssh forward localhost:41713->0.0.0.0:8384 -> failed to dial remote connection: ssh: unexpected packet in response to channel open: " action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=info msg="ssh forward localhost:41713->0.0.0.0:8384 -> failed to dial remote connection: ssh: unexpected packet in response to channel open: " action=a7352dd9-6735-49a1-8e42-f87ddadb5274 version=1.15.8
time="2023-04-13T09:32:33-06:00" level=info msg="ssh forward localhost:41713->0.0.0.0:8384 -> failed to dial remote connection: ssh: unexpected packet in response to channel open: " action=a7352dd9-6735-49a

[bpopovich4]

Is there a way to change the default port for syncthing? I was told I can with the '- forward' option, but that doesn't work. It just forwards a port. I am having issues with multiple users competing for the tcp/quic 22000 port causing failures.

Thank you

[pchico83]

@bpopovich4 did you try the remote field?
https://www.okteto.com/docs/reference/manifest/#remote-integer-optional

[bpopovich4]

Yes, I did after this post. It will open a port on 22001 that I specified, but it still opens syncthing on port 22000 for tcp and udp. I also tried forward and reverse options but it still puts syncthing on port 22000

[pchico83]

Makes sense, I was reading it wrongly. What you need is to be able to modify the syncthing port in the remote container.
We could support it with the following syntax:

sync:
  folders:
    - .:/code
  remotePort: 22000

Would that work?

[bpopovich4]

I tried your suggestion with the following code:
sync:
folders:
- .:/build
remotePort: 22001

And received the below error. Seems not many fields work in the sync:
i Using common-code @ local as context
x Unmarshal error: 'invalid manifest:
line 28: field remotePort not found in type model.syncRaw'

[bpopovich4]

I indentation is correct in the yaml, but did not format here

[pchico83]

@bpopovich4 sync.remotePort is not supported. we would need to implement it

[bpopovich4]

Thank you for your help, BTW. I appreciate it. I thought that is what you had in your screenshot, but I might have misinterpreted it.
My issue is, We have a server that multiple developers use. For example, we have a server node named ntest. There will be three developers on the server running their processes. The first user will run okteto up and get a good connection to tcp/udp 22000 as noted in the syncthing.log.
[ABKAV] 10:21:03 INFO: QUIC listener ([::]:22000) starting
[ABKAV] 10:21:03 INFO: TCP listener ([::]:22000) starting

When the second developer runs okteto up, they get errors in the syncthing log because port 22000 is already in use by the first user. Is there a way to change the syncthing default port? I know there is a way if syncthing is a standalone install with the syncthing.conf. How can I change this port using okteto?
[ABKAV] 11:47:51 INFO: Listen (BEP/tcp): listen tcp 0.0.0.0:22000: listen: address already in use
[ABKAV] 11:47:51 INFO: Listen (BEP/quic): listen udp 0.0.0.0:22000: bind: address already in use
[ABKAV] 11:47:51 INFO: listenerSupervisor@tcp://0.0.0.0:22000: service tcp://0.0.0.0:22000 failed: listen tcp 0.0.0.0:22000: listen: address already in use
[ABKAV] 11:47:51 INFO: listenerSupervisor@quic://0.0.0.0:22000: service quic://0.0.0.0:22000 failed: listen udp 0.0.0.0:22000: bind: address already in use

[pchico83]

Hi @bpopovich4. Okteto is not designed to cover that use case. Each developer should run okteto up on a different container/server.
The port collision is just the first issue you are going to encounter, but you will face more issues.
If you want you can schedule a discovery call from okteto.com to understand your scenario and see if okteto is a good fit for you

[bpopovich4]

Ok, thank you for your help. Much appreciated. I've been troubleshooting this for awhile.

[bpopovich4]

Do you know how well okteto runs on Windows machines? I've read the sync has issues. Or would you recommend linux?
Thank you

[pchico83]

The Okteto CLI works well on Windows and WSL. We aren0t aware of performance issues with windows

[bpopovich4]

Ok thank you. I will test on Windows

[bpopovich4]

okteto is working good on Windows minus a few things like vim doesn't seem to work well in the container. example when I press escape to exit vim, it doesn't recognize it. But we can work around that by editing on the Windows machine. The sync works great. In the yaml, we pulled the environment variable for $USER on linux. Do you know of a way to do that on Windows in the yaml? I know the environment variable is %UserName% but the yaml doesn't recognize it.

Thank you