Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

502 Bad Gateway on Centos 8 #1529

Open
KarlBaumann opened this issue Nov 9, 2020 · 15 comments
Open

502 Bad Gateway on Centos 8 #1529

KarlBaumann opened this issue Nov 9, 2020 · 15 comments
Labels
kind/bug Issue reporting a bug

Comments

@KarlBaumann
Copy link

Is there something special we have to do to get it running on Centos 8?

I tried to run the basic docker-compose.yaml example from README

version: '2'

services:
  nginx-proxy:
    image: jwilder/nginx-proxy
    ports:
      - "80:80"
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro

  whoami:
    image: jwilder/whoami
    environment:
      - VIRTUAL_HOST=whoami.local

And I got:

<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.19.3</center>
</body>
</html>
@mamyn0va
Copy link

mamyn0va commented Nov 9, 2020

Same on Debian 10 here.

@KarlBaumann
Copy link
Author

Is this project dead? I see many issues mentioning 502 error.

@nickv2002
Copy link

nickv2002 commented Nov 15, 2020 •
edited

Same issue here on an Unbutu 20.04 host:

I have some older containers that I configured a month or so ago and continue to work fine with this proxy. But everything I've tried to set up with this proxy over the last few weeks has this same 502 error. Very weird and not too helpful I know, but the change roughly correlates to the creation date on the newest nginx-proxy image (Created 2020-10-30 09:46:00 sha256:5e6f392e99e5454851caa8a3dbb021560a32375edff772b64f862c4d7831d038) so maybe that is related?

In any case here's the tail of the log from nginx-proxy for while I try to access the proxy:

today at 10:38 AM Custom dhparam.pem file found, generation skipped
today at 10:38 AM forego     | starting dockergen.1 on port 5000
today at 10:38 AM forego     | starting nginx.1 on port 5100
today at 10:38 AM dockergen.1 | 2020/11/03 18:38:28 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'
today at 10:38 AM dockergen.1 | 2020/11/03 18:38:28 Watching docker events
today at 10:38 AM dockergen.1 | 2020/11/03 18:38:28 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'
today at 10:39 AM dockergen.1 | 2020/11/03 18:39:26 Received event die for container ee889cabfcd8
today at 10:39 AM dockergen.1 | 2020/11/03 18:39:26 Received event stop for container ee889cabfcd8
today at 10:39 AM dockergen.1 | 2020/11/03 18:39:26 Generated '/etc/nginx/conf.d/default.conf' from 11 containers
today at 10:39 AM dockergen.1 | 2020/11/03 18:39:26 Running 'nginx -s reload'
today at 10:39 AM dockergen.1 | 2020/11/03 18:39:26 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'
today at 10:39 AM dockergen.1 | 2020/11/03 18:39:27 Received event die for container 2dadf9c97119
today at 10:39 AM dockergen.1 | 2020/11/03 18:39:27 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'
today at 10:39 AM dockergen.1 | 2020/11/03 18:39:27 Received event start for container 2dadf9c97119
today at 10:39 AM dockergen.1 | 2020/11/03 18:39:27 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'
today at 10:40 AM dockergen.1 | 2020/11/03 18:40:29 Received event start for container 5fbff26b8d87
today at 10:40 AM dockergen.1 | 2020/11/03 18:40:29 Generated '/etc/nginx/conf.d/default.conf' from 12 containers
today at 10:40 AM dockergen.1 | 2020/11/03 18:40:30 Received event die for container 2dadf9c97119
today at 10:40 AM dockergen.1 | 2020/11/03 18:40:30 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'
today at 10:40 AM dockergen.1 | 2020/11/03 18:40:31 Received event start for container 2dadf9c97119
today at 10:40 AM dockergen.1 | 2020/11/03 18:40:31 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'
today at 10:40 AM nginx.1    | 2020/11/03 18:40:41 [error] 42#42: *1 no live upstreams while connecting to upstream, client: 172.18.0.1, server: twldev.local, request: "GET / HTTP/1.1", upstream: "http://twldev.local/", host: "twldev.local"
today at 10:40 AM nginx.1    | twldev.local 172.18.0.1 - - [03/Nov/2020:18:40:41 +0000] "GET / HTTP/1.1" 502 157 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15"
today at 10:40 AM nginx.1    | 2020/11/03 18:40:41 [error] 42#42: *2 no live upstreams while connecting to upstream, client: 172.18.0.1, server: twldev.local, request: "GET /favicon.ico HTTP/1.1", upstream: "http://twldev.local/favicon.ico", host: "twldev.local", referrer: "http://twldev.local/"
today at 10:40 AM nginx.1    | twldev.local 172.18.0.1 - - [03/Nov/2020:18:40:41 +0000] "GET /favicon.ico HTTP/1.1" 502 157 "http://twldev.local/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15"

Looks like the mDNS cname has been setup but nginx can't connect to the container.

Any idea what is going on and how I could fix things?
Thanks again!

@mamyn0va
Copy link

Hi, here is what I got

❯ docker-compose logs nginx-proxy | more                                                                                                                                                    22:05:15
Attaching to nginx-proxy
nginx-proxy                | WARNING: /etc/nginx/dhparam/dhparam.pem was not found. A pre-generated dhparam.pem will be used for now while a new one
nginx-proxy                | is being generated in the background.  Once the new dhparam.pem is in place, nginx will be reloaded.
nginx-proxy                | forego     | starting dockergen.1 on port 5000
nginx-proxy                | forego     | starting nginx.1 on port 5100
nginx-proxy                | dockergen.1 | 2020/11/17 22:04:37 Generated '/etc/nginx/conf.d/default.conf' from 4 containers
nginx-proxy                | dockergen.1 | 2020/11/17 22:04:37 Running 'nginx -s reload'
nginx-proxy                | dockergen.1 | 2020/11/17 22:04:37 Watching docker events
nginx-proxy                | dockergen.1 | 2020/11/17 22:04:37 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'
nginx-proxy                | nginx.1    | 2020/11/17 22:04:45 [error] 80#80: *1 no live upstreams while connecting to upstream, client: 90.46.166.58, server: plop.fr, request: "GET / HTTP/2.0", upstream: "http://plop.fr/", host: "plop.fr"
nginx-proxy                | nginx.1    | plop.fr 90.46.166.58 - - [17/Nov/2020:22:04:45 +0000] "GET / HTTP/2.0" 502 157 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"
nginx-proxy                | nginx.1    | 2020/11/17 22:04:50 [error] 80#80: *1 no live upstreams while connecting to upstream, client: 90.46.166.58, server: plop.fr, request: "GET / HTTP/2.0", upstream: "http://plop.fr/", host: "plop.fr"
nginx-proxy                | nginx.1    | plop.fr 90.46.166.58 - - [17/Nov/2020:22:04:50 +0000] "GET / HTTP/2.0" 502 157 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:78
.0) Gecko/20100101 Firefox/78.0"

@dahoba
Copy link

dahoba commented Nov 29, 2020

Did you try to enabling masquerading in CentOS 8?

@KarlBaumann
Copy link
Author

Did you try to enabling masquerading in CentOS 8?

I did not. I just switched to Traefik.

@Hamsterman
Copy link

Did you try to enabling masquerading in CentOS 8?

I did not. I just switched to Traefik.

That looks like a good replacement - what is your experience with it so far ?

@KarlBaumann
Copy link
Author

KarlBaumann commented Apr 7, 2021 •
edited

That looks like a good replacement - what is your experience with it so far ?

It is very powerful, but also a bit tricky to configure.

@buchdag
Copy link
Member

buchdag commented Apr 26, 2021

This might have been fixed by #1601

@mamyn0va, @nickv2002, care to test ?

@buchdag buchdag added the kind/bug Issue reporting a bug label Apr 26, 2021
@nickv2002
Copy link

I pulled the latest image with sha256:7e3a002124fc0221b7952f761ec34d6e68fb5c626b4a047c395e5fac1b4fcc93 but still get the 502 errors I was before - no change in behavior:

today at 11:25:44 AM  nginx.1    | 2021/04/26 18:25:44 [error] 40#40: *15 no live upstreams while connecting to upstream, client: 10.0.1.32, server: mylar.local, request: "GET / HTTP/1.1", upstream: "http://mylar.local/", host: "mylar.local"
today at 11:25:44 AM  nginx.1    | mylar.local 10.0.1.32 - - [26/Apr/2021:18:25:44 +0000] "GET / HTTP/1.1" 502 157 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15"

@buchdag
Copy link
Member

buchdag commented Apr 26, 2021

Could you post the output of those commands ?

docker inspect yournginxproxycontainer
docker exec yournginxproxycontainer nginx -T
docker exec yournginxproxycontainer cat /proc/1/cpuset
docker exec yournginxproxycontainer cat /proc/self/cgroup
docker exec yournginxproxycontainer cat /proc/self/mountinfo

@nickv2002
Copy link

nickv2002 commented Apr 28, 2021 •
edited

Could you post the output of those commands ?

docker inspect yournginxproxycontainer
docker exec yournginxproxycontainer nginx -T
docker exec yournginxproxycontainer cat /proc/1/cpuset
docker exec yournginxproxycontainer cat /proc/self/cgroup
docker exec yournginxproxycontainer cat /proc/self/mountinfo

A long dump of text from running those commands on my system
output.txt

@buchdag
Copy link
Member

buchdag commented Apr 28, 2021

Are you certain the mylar.local container is sharing a network with the nginx-proxy (or docker-gen if you use a two container setup) container ?

docker inspect --format='{{.NetworkSettings.Networks}}' yournginxproxycontainer
docker inspect --format='{{.NetworkSettings.Networks}}' mylar.localcontainer

@nickv2002
Copy link

nickv2002 commented Apr 28, 2021 •
edited

Are you certain the mylar.local container is sharing a network with the nginx-proxy (or docker-gen if you use a two container setup) container ?

docker inspect --format='{{.NetworkSettings.Networks}}' yournginxproxycontainer
docker inspect --format='{{.NetworkSettings.Networks}}' mylar.localcontainer

Ahh so they're on different networks... hmm. Must be related to having different docker-compose stacks for them in my case.

nick@uzfs:~$ docker inspect --format='{{.NetworkSettings.Networks}}' 6f745d600817
map[bridge:0xc000535b00]
nick@uzfs:~$ docker inspect --format='{{.NetworkSettings.Networks}}' 4f3c9e8cd81e
map[compose_default:0xc000576000]

Thanks for you help debugging that. I'll work on figuring out how to bridge that networking unless you have any tips offhand.

@buchdag
Copy link
Member

buchdag commented Apr 28, 2021 •
edited

I usually create a separate Docker network outside of Docker compose then use it as an external network in the compose file.

docker network create nginx-proxy

then

version: '3'

services:
  nginx:
    [...]
    networks:
      - nginx-proxy

# Do not forget to 'docker network create nginx-proxy' before launch
# and to connect the proxied containers to the nginx-proxy network.

networks:
  nginx-proxy:
    external: true

Repeated in every compose file for services that will be proxied.

Also take a look at https://github.com/buchdag/letsencrypt-nginx-proxy-companion-compose

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Issue reporting a bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@nickv2002 @dahoba @KarlBaumann @mamyn0va @buchdag @Hamsterman