Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Caprover restart after each app creation #1893

Open
alexomc opened this issue Sep 21, 2023 · 23 comments
Open

Caprover restart after each app creation #1893

alexomc opened this issue Sep 21, 2023 · 23 comments

Comments

@alexomc
Copy link

alexomc commented Sep 21, 2023

IMPORTANT: This is a bug report. If you are having problem with deploying a particular app use the deployment issue type.

If you are having problems with deploying a particular app using a customized dockerfile - please change your questions to Dockerfile and post it where appropriate such as StackOverflow. This is a Docker question, not a CapRover specific issue.


What is the problem?

Since the last update, each time an app is created using the CLI/API (not tested using the caprover dashboard), Caprover restart. This revokes the API token and makes Caprover completely unusable.

If applicable, content of captain-definition file:

Steps to reproduce the problem:

Create a new app from a docker image from the CLI or the API.

Output of the following command on your server:

uname -a && lsb_release -a && free -h

nux ams-sm5038mr-h8trf-4-3 5.15.0-84-generic #93-Ubuntu SMP Tue Sep 5 17:16:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy
total used free shared buff/cache available
Mem: 125Gi 2.4Gi 119Gi 36Mi 3.5Gi 122Gi
Swap: 3.9Gi 0B 3.9Gi

@githubsaturn
Copy link
Collaborator

Can you get the logs and include them?

docker service logs captain-captain

@alexomc
Copy link
Author

alexomc commented Sep 21, 2023

sure:

captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | Captain Starting ...
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | Overriding skipVerifyingDomains from /captain/data/config-override.json
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:12.351 pm    
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | Error: Salt is not set! Cannot create authenticator
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | Error: Salt is not set! Cannot create authenticator
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at Authenticator.getAuthenticator (/usr/src/app/built/user/Authenticator.js:179:23)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at new UserManager (/usr/src/app/built/user/UserManager.js:18:95)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at UserManagerProvider.get (/usr/src/app/built/user/UserManagerProvider.js:18:32)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at /usr/src/app/built/injection/Injector.js:31:84
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at Layer.handle [as handle_request] (/usr/src/app/node_modules/express/lib/router/layer.js:95:5)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at trim_prefix (/usr/src/app/node_modules/express/lib/router/index.js:328:13)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at /usr/src/app/node_modules/express/lib/router/index.js:286:9
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at Function.process_params (/usr/src/app/node_modules/express/lib/router/index.js:346:12)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at next (/usr/src/app/node_modules/express/lib/router/index.js:280:10)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at cookieParser (/usr/src/app/node_modules/cookie-parser/index.js:71:5)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | GET /api/v2/user/system/loadbalancerinfo 500 104.639 ms - 21
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:12.376 pm    
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | Error: Salt is not set! Cannot create authenticator
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | Error: Salt is not set! Cannot create authenticator
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at Authenticator.getAuthenticator (/usr/src/app/built/user/Authenticator.js:179:23)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at new UserManager (/usr/src/app/built/user/UserManager.js:18:95)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at UserManagerProvider.get (/usr/src/app/built/user/UserManagerProvider.js:18:32)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at /usr/src/app/built/injection/Injector.js:31:84
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at Layer.handle [as handle_request] (/usr/src/app/node_modules/express/lib/router/layer.js:95:5)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at trim_prefix (/usr/src/app/node_modules/express/lib/router/index.js:328:13)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at /usr/src/app/node_modules/express/lib/router/index.js:286:9
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at Function.process_params (/usr/src/app/node_modules/express/lib/router/index.js:346:12)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at next (/usr/src/app/node_modules/express/lib/router/index.js:280:10)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at cookieParser (/usr/src/app/node_modules/cookie-parser/index.js:71:5)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | GET /api/v2/user/system/netdata 500 1.692 ms - 21
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:13.282 pm    Emptying generated and temp folders.
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:13.292 pm    Ensuring directories are available on host. Started.
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:13.302 pm    Ensuring directories are available on host. Finished.
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:13.311 pm    Network captain-overlay-network is already attached to service: captain-captain
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:13.322 pm    captain-captain (service) has already been connected to secret: captain-salt
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:13.325 pm    Migration not needed, skipping.
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:13.336 pm    Copying fake certificates...
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:13.345 pm    Updating Load Balancer - Setting up NGINX conf file...
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:13.346 pm    Locking NGINX configuration reloading...
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:13.395 pm    SUCCESS: UNLocking NGINX configuration reloading...
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:13.399 pm    Captain Nginx is already running.. 
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:13.403 pm    Updating NGINX service...
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:13.414 pm    Waiting for 5 seconds for nginx reload to take into effect
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | GET /api/v2/user/apps/appDefinitions 200 6.481 ms - 69
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:18.420 pm    Pruning containers...
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:18.422 pm    NGINX is fully set up and working...
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:18.428 pm    Captain Certbot is already running.. 
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:18.433 pm    Updating Certbot service...
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:21.451 pm    Retrying to get containerId for captain-certbot retry count:0
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:23.446 pm    Pruning containers...
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:24.459 pm    Retrying to get containerId for captain-certbot retry count:1
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:27.466 pm    Retrying to get containerId for captain-certbot retry count:2
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | GET / 304 3.690 ms - -
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | GET /manifest.json 304 3.834 ms - -
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | GET /dark-theme.css 304 3.553 ms - -
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | GET /static/js/main.163848a9.js 304 3.498 ms - -
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | GET /icon-512x512.png 304 0.968 ms - -
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | GET /api/v2/user/apps/appDefinitions 304 1.871 ms - -
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:30.470 pm    Retrying to get containerId for captain-certbot retry count:3
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:33.477 pm    Retrying to get containerId for captain-certbot retry count:4
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | GET /api/v2/user/apps/appDefinitions 200 1.994 ms - 69
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | GET /api/v2/user/apps/appDefinitions 200 2.466 ms - 69
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:36.483 pm    Retrying to get containerId for captain-certbot retry count:5
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:36.488 pm    executeCommand Container: captain-certbot certbot certificates --non-interactive 
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | POST /api/v2/login 200 22.303 ms - 69
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | GET /api/v2/user/apps/appDefinitions 200 2.374 ms - 69
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:40.238 pm    **** Captain is initialized and ready to serve you! ****
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | POST /api/v2/login 200 345.867 ms - 307
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | GET /api/v2/user/system/info 304 16.721 ms - -
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | GET /api/v2/user/apps/appDefinitions 200 11.102 ms - 56121
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | GET /api/v2/user/system/versioninfo 304 485.732 ms - -
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:50.234 pm    executeCommand Container: captain-certbot certbot certificates --non-interactive 
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | GET /api/v2/user/apps/appDefinitions 200 2.935 ms - 101
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:51.758 pm    
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | Error: Invalid credentials
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | Error: Invalid credentials
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at ApiStatusCodes.createError (/usr/src/app/built/api/ApiStatusCodes.js:8:16)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at /usr/src/app/built/user/Authenticator.js:69:48
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | POST /api/v2/login 200 167.663 ms - 61
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:52.488 pm    executeCommand Container: captain-certbot certbot renew --non-interactive 
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:57.169 pm    Updating Load Balancer - renewAllCerts
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:57.169 pm    Locking NGINX configuration reloading...
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:57.233 pm    SUCCESS: UNLocking NGINX configuration reloading...
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:57.234 pm    sendReloadSignal...
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:34:57.238 pm    Kill HUP Container: 6a4d082276b1ba496f5939471a39fb9a6a4fd968e812237d82b97730d5715833
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | GET /api/v2/user/apps/appDefinitions 200 2.395 ms - 101
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:36:08.695 pm    
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | Error: Invalid credentials
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | Error: Invalid credentials
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at ApiStatusCodes.createError (/usr/src/app/built/api/ApiStatusCodes.js:8:16)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at /usr/src/app/built/user/Authenticator.js:69:48
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | POST /api/v2/login 200 168.160 ms - 61
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | GET /api/v2/user/apps/appDefinitions 200 1.449 ms - 101
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:36:14.877 pm    
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | Error: Invalid credentials
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | Error: Invalid credentials
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at ApiStatusCodes.createError (/usr/src/app/built/api/ApiStatusCodes.js:8:16)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at /usr/src/app/built/user/Authenticator.js:69:48
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | POST /api/v2/login 200 168.288 ms - 61
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | GET /api/v2/user/apps/appDefinitions 200 2.147 ms - 101
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 6:41:03.440 pm    
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | Error: Invalid credentials
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | Error: Invalid credentials
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at ApiStatusCodes.createError (/usr/src/app/built/api/ApiStatusCodes.js:8:16)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at /usr/src/app/built/user/Authenticator.js:69:48
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | POST /api/v2/login 200 168.395 ms - 61
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | POST /api/v2/login 200 232.008 ms - 307

@githubsaturn
Copy link
Collaborator

I don't see any restarts happening here. Looks like it was a starter successfully, then there were a few endpoint calls, some of them resulted in some errors which is totally fine.

@githubsaturn
Copy link
Collaborator

Captain is initialized

Is the log that shows a successful start up. I only see one of this.

@alexomc
Copy link
Author

alexomc commented Sep 21, 2023

I think the log are returning only the logs since the last restart... and since Caprover restarts each time a new app is created you won't see anything here.

@githubsaturn
Copy link
Collaborator

If you use docker service logs, it should return the whole logs (before restart)

Alternatively, you can just add --follow to the end of the command, and create an app. This will show you the live logs.

@alexomc
Copy link
Author

alexomc commented Sep 21, 2023

got it, here's what happens when an app is created and Caprover restarts:

captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:05.026 pm    Registering app started: jferacwunx
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:05.044 pm    AppName is saved: jferacwunx
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | POST /api/v2/user/apps/appDefinitions/register?detached=1 200 20.968 ms - 61
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:05.088 pm    Building docker image. This might take a few minutes...
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:05.113 pm    Docker Auth is found. Pushing the image...
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:05.117 pm    
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | Error: Wrapped: PUSH FAILED
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | Error: Wrapped: PUSH FAILED
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at errorize (/usr/src/app/built/utils/Logger.js:7:16)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at Logger.e (/usr/src/app/built/utils/Logger.js:27:21)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at /usr/src/app/built/user/DockerRegistryHelper.js:59:42
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at new Promise (<anonymous>)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at /usr/src/app/built/user/DockerRegistryHelper.js:58:28
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:05.117 pm    
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | Error: (HTTP code 400) bad parameter - invalid reference format 
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | Error: (HTTP code 400) bad parameter - invalid reference format 
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at /usr/src/app/node_modules/docker-modem/lib/modem.js:343:17
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at getCause (/usr/src/app/node_modules/docker-modem/lib/modem.js:373:7)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at Modem.buildPayload (/usr/src/app/node_modules/docker-modem/lib/modem.js:342:5)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at IncomingMessage.<anonymous> (/usr/src/app/node_modules/docker-modem/lib/modem.js:310:16)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at IncomingMessage.emit (node:events:526:35)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at endReadableNT (node:internal/streams/readable:1359:12)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | /usr/src/app/built/api/ApiStatusCodes.js:8
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |         return new CaptainError_1.CaptainError(code, message || 'NONE');
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |                ^
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | 
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | CaptainError: Push failed: Error: (HTTP code 400) bad parameter - invalid reference format 
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at ApiStatusCodes.createError (/usr/src/app/built/api/ApiStatusCodes.js:8:16)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at /usr/src/app/built/user/DockerRegistryHelper.js:61:57
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at new Promise (<anonymous>)
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at /usr/src/app/built/user/DockerRegistryHelper.js:58:28
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |   captainErrorType: 1000,
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    |   apiMessage: 'Push failed: Error: (HTTP code 400) bad parameter - invalid reference format '
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | }
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | 
captain-captain.1.sasqli1w0jip@ams-sm5038mr-h8trf-4-3    | Node.js v18.17.1
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | Captain Starting ...
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | Overriding skipVerifyingDomains from /captain/data/config-override.json
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:13.318 pm    Emptying generated and temp folders.
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:13.327 pm    Ensuring directories are available on host. Started.
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:13.334 pm    Ensuring directories are available on host. Finished.
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:13.341 pm    Network captain-overlay-network is already attached to service: captain-captain
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:13.348 pm    captain-captain (service) has already been connected to secret: captain-salt
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:13.351 pm    Migration not needed, skipping.
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:13.357 pm    Copying fake certificates...
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:13.364 pm    Updating Load Balancer - Setting up NGINX conf file...
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:13.364 pm    Locking NGINX configuration reloading...
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:13.397 pm    SUCCESS: UNLocking NGINX configuration reloading...
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:13.401 pm    Captain Nginx is already running.. 
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:13.404 pm    Updating NGINX service...
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:13.414 pm    Waiting for 5 seconds for nginx reload to take into effect
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:18.419 pm    Pruning containers...
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:18.421 pm    NGINX is fully set up and working...
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:18.425 pm    Captain Certbot is already running.. 
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:18.431 pm    Updating Certbot service...
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:21.448 pm    Retrying to get containerId for captain-certbot retry count:0
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:23.445 pm    Pruning containers...
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:24.456 pm    Retrying to get containerId for captain-certbot retry count:1
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:03:27.464 pm    Retrying to get containerId for captain-certbot retry count:2

@alexomc
Copy link
Author

alexomc commented Sep 21, 2023

note that the exact same API command works fine with older version of Caprover

@githubsaturn
Copy link
Collaborator

Super helpful!

  • what version works?
  • does using the UI also result in a restart?

@alexomc
Copy link
Author

alexomc commented Sep 21, 2023

Yes, that's weird, just creating a blank app (basically just entering the name of a new app inside the UI) triggers a restart:

captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:29:27.189 pm    Registering app started: test
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:29:27.207 pm    AppName is saved: test
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | POST /api/v2/user/apps/appDefinitions/register?detached=1 200 22.365 ms - 61
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:29:27.253 pm    Building docker image. This might take a few minutes...
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:29:27.280 pm    Docker Auth is found. Pushing the image...
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:29:27.283 pm    
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | Error: Wrapped: PUSH FAILED
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | Error: Wrapped: PUSH FAILED
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    |     at errorize (/usr/src/app/built/utils/Logger.js:7:16)
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    |     at Logger.e (/usr/src/app/built/utils/Logger.js:27:21)
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    |     at /usr/src/app/built/user/DockerRegistryHelper.js:59:42
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    |     at new Promise (<anonymous>)
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    |     at /usr/src/app/built/user/DockerRegistryHelper.js:58:28
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:29:27.283 pm    
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | Error: (HTTP code 400) bad parameter - invalid reference format 
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | Error: (HTTP code 400) bad parameter - invalid reference format 
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    |     at /usr/src/app/node_modules/docker-modem/lib/modem.js:343:17
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    |     at getCause (/usr/src/app/node_modules/docker-modem/lib/modem.js:373:7)
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    |     at Modem.buildPayload (/usr/src/app/node_modules/docker-modem/lib/modem.js:342:5)
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    |     at IncomingMessage.<anonymous> (/usr/src/app/node_modules/docker-modem/lib/modem.js:310:16)
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    |     at IncomingMessage.emit (node:events:526:35)
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    |     at endReadableNT (node:internal/streams/readable:1359:12)
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    |     at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | /usr/src/app/built/api/ApiStatusCodes.js:8
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    |         return new CaptainError_1.CaptainError(code, message || 'NONE');
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    |                ^
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | 
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | CaptainError: Push failed: Error: (HTTP code 400) bad parameter - invalid reference format 
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    |     at ApiStatusCodes.createError (/usr/src/app/built/api/ApiStatusCodes.js:8:16)
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    |     at /usr/src/app/built/user/DockerRegistryHelper.js:61:57
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    |     at new Promise (<anonymous>)
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    |     at /usr/src/app/built/user/DockerRegistryHelper.js:58:28
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    |   captainErrorType: 1000,
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    |   apiMessage: 'Push failed: Error: (HTTP code 400) bad parameter - invalid reference format '
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | }
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | 
captain-captain.1.tz8lukhrgdxv@ams-sm5038mr-h8trf-4-3    | Node.js v18.17.1
captain-captain.1.if2w4n2u1rsi@ams-sm5038mr-h8trf-4-3    | Captain Starting ...
captain-captain.1.if2w4n2u1rsi@ams-sm5038mr-h8trf-4-3    | Overriding skipVerifyingDomains from /captain/data/config-override.json
captain-captain.1.if2w4n2u1rsi@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:29:35.613 pm    Emptying generated and temp folders.
captain-captain.1.if2w4n2u1rsi@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:29:35.621 pm    Ensuring directories are available on host. Started.
captain-captain.1.if2w4n2u1rsi@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:29:35.628 pm    Ensuring directories are available on host. Finished.
captain-captain.1.if2w4n2u1rsi@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:29:35.635 pm    Network captain-overlay-network is already attached to service: captain-captain
captain-captain.1.if2w4n2u1rsi@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:29:35.642 pm    captain-captain (service) has already been connected to secret: captain-salt
captain-captain.1.if2w4n2u1rsi@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:29:35.644 pm    Migration not needed, skipping.
captain-captain.1.if2w4n2u1rsi@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:29:35.650 pm    Copying fake certificates...
captain-captain.1.if2w4n2u1rsi@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:29:35.656 pm    Updating Load Balancer - Setting up NGINX conf file...
captain-captain.1.if2w4n2u1rsi@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:29:35.657 pm    Locking NGINX configuration reloading...
captain-captain.1.if2w4n2u1rsi@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:29:35.689 pm    SUCCESS: UNLocking NGINX configuration reloading...
captain-captain.1.if2w4n2u1rsi@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:29:35.693 pm    Captain Nginx is already running.. 
captain-captain.1.if2w4n2u1rsi@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:29:35.696 pm    Updating NGINX service...
captain-captain.1.if2w4n2u1rsi@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:29:35.708 pm    Waiting for 5 seconds for nginx reload to take into effect
captain-captain.1.if2w4n2u1rsi@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:29:40.713 pm    Pruning containers...
captain-captain.1.if2w4n2u1rsi@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:29:40.715 pm    NGINX is fully set up and working...
captain-captain.1.if2w4n2u1rsi@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:29:40.720 pm    Captain Certbot is already running.. 
captain-captain.1.if2w4n2u1rsi@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:29:40.723 pm    Updating Certbot service...
captain-captain.1.if2w4n2u1rsi@ams-sm5038mr-h8trf-4-3    | September 21st 2023, 7:29:43.737 pm    Retrying to get containerId for captain-certbot retry count:0

@alexomc
Copy link
Author

alexomc commented Sep 21, 2023

doesn't seem to be a common issue or Github would be flooded, do you think reinstalling Caprover might fix it?

@alexomc
Copy link
Author

alexomc commented Sep 21, 2023

1.10.1 version works perfectly btw

@githubsaturn
Copy link
Collaborator

Definitely not a common issue. It is still quite confusing. Can you try disabling your default push on your registries in the cluster tab? Does that fix the issue?

@alexomc
Copy link
Author

alexomc commented Sep 22, 2023

I disabled the default push and that fixed the issue, thanks a lot!
I'm available if you need to do more tests, install it to another server to see if it does the same issue or else.

Not related, I see on the changelog: Improved: Added support for overriding mesh network settings and using an existing Docker Swarm 56e739c
But I don't see anywhere we can manage the mesh network settings, do you know where it is? I can create another ticket if it's simpler btw.

Thanks!

Alex.

@githubsaturn
Copy link
Collaborator

RE: swarm

1- You can just add useExistingSwarm to your override config BEFORE your install CapRover.
2- Run docker swarm init with any options that you want
3- Install CapRover as you'd do normally

RE: restart on fail to push
It's still a valid bug, I'll keep the issue open to investigate further.

@jbalatero
Copy link

Have this same issue after creating an app, is says 1001: Captain is not ready yet.

image

here's the logs after create of app.

captain-captain.1.iq9red6rx1ty@aerp-srv01    | May 13th 2024, 12:31:38.578 pm    Registering app started: tst
captain-captain.1.iq9red6rx1ty@aerp-srv01    | May 13th 2024, 12:31:38.583 pm    AppName is saved: tst
captain-captain.1.iq9red6rx1ty@aerp-srv01    | POST /api/v2/user/apps/appDefinitions/register?detached=1 200 6.222 ms - 61
captain-captain.1.iq9red6rx1ty@aerp-srv01    | May 13th 2024, 12:31:38.591 pm    Building docker image. This might take a few minutes...
captain-captain.1.iq9red6rx1ty@aerp-srv01    | May 13th 2024, 12:31:38.600 pm    Docker Auth is found. Pushing the image...
captain-captain.1.iq9red6rx1ty@aerp-srv01    | May 13th 2024, 12:31:38.601 pm    
captain-captain.1.iq9red6rx1ty@aerp-srv01    | Error: Wrapped: PUSH FAILED
captain-captain.1.iq9red6rx1ty@aerp-srv01    | Error: Wrapped: PUSH FAILED
captain-captain.1.iq9red6rx1ty@aerp-srv01    |     at errorize (/usr/src/app/built/utils/Logger.js:7:16)
captain-captain.1.iq9red6rx1ty@aerp-srv01    |     at Logger.e (/usr/src/app/built/utils/Logger.js:27:21)
captain-captain.1.iq9red6rx1ty@aerp-srv01    |     at /usr/src/app/built/user/DockerRegistryHelper.js:59:42
captain-captain.1.iq9red6rx1ty@aerp-srv01    |     at new Promise (<anonymous>)
captain-captain.1.iq9red6rx1ty@aerp-srv01    |     at /usr/src/app/built/user/DockerRegistryHelper.js:58:28
captain-captain.1.iq9red6rx1ty@aerp-srv01    |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
captain-captain.1.iq9red6rx1ty@aerp-srv01    | May 13th 2024, 12:31:38.601 pm    
captain-captain.1.iq9red6rx1ty@aerp-srv01    | Error: (HTTP code 400) bad parameter - invalid reference format 
captain-captain.1.iq9red6rx1ty@aerp-srv01    | Error: (HTTP code 400) bad parameter - invalid reference format 
captain-captain.1.iq9red6rx1ty@aerp-srv01    |     at /usr/src/app/node_modules/docker-modem/lib/modem.js:343:17
captain-captain.1.iq9red6rx1ty@aerp-srv01    |     at getCause (/usr/src/app/node_modules/docker-modem/lib/modem.js:373:7)
captain-captain.1.iq9red6rx1ty@aerp-srv01    |     at Modem.buildPayload (/usr/src/app/node_modules/docker-modem/lib/modem.js:342:5)
captain-captain.1.iq9red6rx1ty@aerp-srv01    |     at IncomingMessage.<anonymous> (/usr/src/app/node_modules/docker-modem/lib/modem.js:310:16)
captain-captain.1.iq9red6rx1ty@aerp-srv01    |     at IncomingMessage.emit (node:events:526:35)
captain-captain.1.iq9red6rx1ty@aerp-srv01    |     at endReadableNT (node:internal/streams/readable:1359:12)
captain-captain.1.iq9red6rx1ty@aerp-srv01    |     at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
captain-captain.1.iq9red6rx1ty@aerp-srv01    | /usr/src/app/built/api/ApiStatusCodes.js:8
captain-captain.1.iq9red6rx1ty@aerp-srv01    |         return new CaptainError_1.CaptainError(code, message || 'NONE');
captain-captain.1.iq9red6rx1ty@aerp-srv01    |                ^
captain-captain.1.iq9red6rx1ty@aerp-srv01    | 
captain-captain.1.iq9red6rx1ty@aerp-srv01    | CaptainError: Push failed: Error: (HTTP code 400) bad parameter - invalid reference format 
captain-captain.1.iq9red6rx1ty@aerp-srv01    |     at ApiStatusCodes.createError (/usr/src/app/built/api/ApiStatusCodes.js:8:16)
captain-captain.1.iq9red6rx1ty@aerp-srv01    |     at /usr/src/app/built/user/DockerRegistryHelper.js:61:57
captain-captain.1.iq9red6rx1ty@aerp-srv01    |     at new Promise (<anonymous>)
captain-captain.1.iq9red6rx1ty@aerp-srv01    |     at /usr/src/app/built/user/DockerRegistryHelper.js:58:28
captain-captain.1.iq9red6rx1ty@aerp-srv01    |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
captain-captain.1.iq9red6rx1ty@aerp-srv01    |   captainErrorType: 1000,
captain-captain.1.iq9red6rx1ty@aerp-srv01    |   apiMessage: 'Push failed: Error: (HTTP code 400) bad parameter - invalid reference format '
captain-captain.1.iq9red6rx1ty@aerp-srv01    | }
captain-captain.1.iq9red6rx1ty@aerp-srv01    | 
captain-captain.1.iq9red6rx1ty@aerp-srv01    | Node.js v18.17.1
captain-captain.1.l9lr9hbi6h02@aerp-srv01    | Captain Starting ...
captain-captain.1.l9lr9hbi6h02@aerp-srv01    | May 13th 2024, 12:31:46.856 pm    Emptying generated and temp folders.
captain-captain.1.l9lr9hbi6h02@aerp-srv01    | May 13th 2024, 12:31:46.868 pm    Ensuring directories are available on host. Started.
captain-captain.1.l9lr9hbi6h02@aerp-srv01    | May 13th 2024, 12:31:46.874 pm    Ensuring directories are available on host. Finished.
captain-captain.1.l9lr9hbi6h02@aerp-srv01    | May 13th 2024, 12:31:46.879 pm    Network captain-overlay-network is already attached to service: captain-captain
captain-captain.1.l9lr9hbi6h02@aerp-srv01    | May 13th 2024, 12:31:46.886 pm    captain-captain (service) has already been connected to secret: captain-salt
captain-captain.1.l9lr9hbi6h02@aerp-srv01    | May 13th 2024, 12:31:46.888 pm    Migration not needed, skipping.
captain-captain.1.l9lr9hbi6h02@aerp-srv01    | May 13th 2024, 12:31:46.893 pm    Copying fake certificates...
captain-captain.1.l9lr9hbi6h02@aerp-srv01    | May 13th 2024, 12:31:46.898 pm    Updating Load Balancer - Setting up NGINX conf file...
captain-captain.1.l9lr9hbi6h02@aerp-srv01    | May 13th 2024, 12:31:46.898 pm    Locking NGINX configuration reloading...
captain-captain.1.l9lr9hbi6h02@aerp-srv01    | May 13th 2024, 12:31:46.910 pm    SUCCESS: UNLocking NGINX configuration reloading...
captain-captain.1.l9lr9hbi6h02@aerp-srv01    | May 13th 2024, 12:31:46.913 pm    Captain Nginx is already running.. 
captain-captain.1.l9lr9hbi6h02@aerp-srv01    | May 13th 2024, 12:31:46.916 pm    Updating NGINX service...
captain-captain.1.l9lr9hbi6h02@aerp-srv01    | May 13th 2024, 12:31:46.926 pm    Waiting for 5 seconds for nginx reload to take into effect
captain-captain.1.l9lr9hbi6h02@aerp-srv01    | May 13th 2024, 12:31:51.931 pm    Pruning containers...
captain-captain.1.l9lr9hbi6h02@aerp-srv01    | May 13th 2024, 12:31:51.932 pm    NGINX is fully set up and working...
captain-captain.1.l9lr9hbi6h02@aerp-srv01    | May 13th 2024, 12:31:51.936 pm    Captain Certbot is already running.. 
captain-captain.1.l9lr9hbi6h02@aerp-srv01    | May 13th 2024, 12:31:51.939 pm    Updating Certbot service...
captain-captain.1.l9lr9hbi6h02@aerp-srv01    | May 13th 2024, 12:31:54.957 pm    Retrying to get containerId for captain-certbot retry count:0
captain-captain.1.l9lr9hbi6h02@aerp-srv01    | May 13th 2024, 12:31:56.954 pm    Pruning containers...
captain-captain.1.l9lr9hbi6h02@aerp-srv01    | May 13th 2024, 12:31:57.962 pm    Retrying to get containerId for captain-certbot retry count:1
captain-captain.1.l9lr9hbi6h02@aerp-srv01    | May 13th 2024, 12:32:00.966 pm    Retrying to get containerId for captain-certbot retry count:2
captain-captain.1.l9lr9hbi6h02@aerp-srv01    | May 13th 2024, 12:32:03.970 pm    Retrying to get containerId for captain-certbot retry count:3

Note: I have docker registry setup.

@jbalatero
Copy link

After disabling Default Push Registry, the creation of app now works fine.
But I cannot keep on disabling it since I will be clustering it

@jbalatero
Copy link

I believe this is related on this issue: when creating a one-click app, this error occurs Failed: Error: Push failed: Error: (HTTP code 400) bad parameter - invalid reference format , but this time, it did not cause Caprover to crash

image

@githubsaturn
Copy link
Collaborator

@jbalatero - a few questions for you:
1- what version of CapRover are you on?
2- Does it crash everytime you create a new one click app or just sometimes?
3- Are you using a self-hosted registry through CapRover or you're using a 3rd party registry like aws registry / github registry etc?

@jbalatero
Copy link

jbalatero commented May 14, 2024

@githubsaturn

1- what version of CapRover are you on?
1.11.1

2- Does it crash everytime you create a new one click app or just sometimes?

It crashes everytime. Even after fresh install. After 30 seconds or so, it comes up again and works fine.

3- Are you using a self-hosted registry through CapRover or you're using a 3rd party registry like aws registry / github registry etc?
I'm using digitalocean container registry added in Caprover

@jbalatero
Copy link

caprover/caprover:1.11.1
caprover/certbot-sleeping:v1.6.0
nginx:1.24

Docker version 26.1.2, build 211e74b

@githubsaturn
Copy link
Collaborator

@jbalatero - can you try enabling the self hosted registry and set that as a default registry to see the issue is reproducible?

@jbalatero
Copy link

jbalatero commented May 15, 2024

@githubsaturn I can confirm it is reproducible with that config.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants