24.4.2 - OCI runtime create failed: runc create failed

[futureweb]

Self-Hosted Version

24.4.2

CPU Architecture

x86_64

Docker Version

Docker version 26.1.1, build 4cf5afa

Docker Compose Version

Docker Compose version v2.27.0

Steps to Reproduce

Update or fresh Re-Install Version 24.4.2 (coming from Sentry 23.7.2 adbbcd8)
Fire it up (docker compose up -d)
Won't start - Error:
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't get final child's PID from pipe: EOF: unknown

Expected Result

Sentry should start without Problems

Actual Result

composer.log
install.log

Event ID

No response

[azaslavsky]

Two previous issues, both of which were related to permissions and unexpected machine state:

#1658
#1794

Can you try the approaches recommended here and report back?

[futureweb]

@azaslavsky - one approach there is to run "./reset.sh" which is no option as we need the captured Data.

The other one is to chmod 777 everything - doesn't sound like an approach one wants to do on a productive machine?

chmod 777 /opt/sentry
chmod 777 /opt/sentry/sentry/postgres/init_hba.sh
chmod 777 /opt/sentry/sentry/postgres/postgres-entrypoint.sh
chmod 777 /opt/sentry/sentry/sentry/entrypoint.sh

Also on both of those 2 linked issues it's failing on postgres which in this case isn't the problem if I don't overlook anything?
In my case it's not even saying where it's failing?!?

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't get final child's PID from pipe: EOF: unknown

Any other ideas on how to further debug this issue?

[azaslavsky]

Some googling revealed this suggestion, but tat this point I am just prowling the internet for random answers. This error is very likely to do with your system configuration - in the past, this has often been with regards to permissioning, hence why chmod solution was suggested, but it could be a number of other settings as well. Ultimately, I don't have very much information about your specific system's settings, and docker is not great at providing detailed error information for investigation.

[futureweb]

@azaslavsky unfortunately the max_user_namespaces also seems to be already high enough here:

# sysctl user.max_user_namespaces
user.max_user_namespaces = 2147483647

I'm running Sentry on clean install of AlmaLinux 8:

# cat /etc/redhat-release 
AlmaLinux release 8.9 (Midnight Oncilla)

running out of ideas on what else to try to get sentry updated here :-(

[azaslavsky]

I'm not sure what we can do remotely either. We test on ubuntu and alpine linux, and it works in both environments. Ultimately, there are any number of reasons why this could pop up, and the solution is likely not self-hosted specific.

[aldy505]

I'm running Sentry on clean install of AlmaLinux 8:

# cat /etc/redhat-release 
AlmaLinux release 8.9 (Midnight Oncilla)

running out of ideas on what else to try to get sentry updated here :-(

Hey @futureweb can you try to install it on higher version of AlmaLinux? A wild guess from me is that it failed to build on Linux kernel 4.x, but AlmaLinux 9.x and Ubuntu 20.x uses Linux kernel 5.x. If you're okay with moving to some other distro other than RHEL derivatives, I'd highly recommend to install sentry self-hosted on Debian 12. If installing on AlmaLinux 9.x or Debian 11/12 works with no problem, then I can assume that.. Docker does not plays well with old distro running old kernel version.

[futureweb]

@aldy505 - as we operate all our Servers on RHEL based Servers we would like to stay within this universe, if possible.
Just tried to install same Sentry Version (23.7.2) on AlmaLinux 9 as we got on our AlmaLinux 8 to migrate current installation to new server. But it won't even install in first place! :-/

VERSION="23.7.2"
git clone https://github.com/getsentry/self-hosted.git
cd self-hosted
git checkout ${VERSION}
SENTRY_IMAGE=getsentry/sentry:${VERSION} ./install.sh --report-self-hosted-issues
...
...
...
  File "/usr/local/lib/python3.8/site-packages/django/core/cache/backends/memcached.py", line 234, in __init__
    import pymemcache.serde
ModuleNotFoundError: No module named 'pymemcache'
Error in install/set-up-and-migrate-database.sh:12.
'$dcr web upgrade' exited with status 1
-> ./install.sh:main:34
--> install/set-up-and-migrate-database.sh:source:12

install/error-handling.sh: line 82: /bin/docker: Argument list too long

# uname -a
Linux hostname 5.14.0 #1 SMP Tue Jan 9 19:45:01 MSK 2024 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/redhat-release 
AlmaLinux release 9.4 (Seafoam Ocelot)
# docker info --format '{{.Architecture}}'
x86_64
# docker --version
Docker version 26.1.2, build 211e74b
# docker compose version
Docker Compose version v2.27.0

seems like no luck with rhel based servers here?!? :-(

[futureweb]

update: after running /var/www/sentry/self-hosted/scripts/reset.sh and another fresh install the installation went through completely ... but then when I tried to fire it up:

# docker compose up -d
[+] Running 36/39
 ✔ Container sentry-self-hosted-redis-1                                     Healthy                                                                                                                                                                                            50.2s 
 ✔ Container sentry-self-hosted-symbolicator-cleanup-1                      Started                                                                                                                                                                                             4.9s 
 ✔ Container sentry-self-hosted-vroom-cleanup-1                             Started                                                                                                                                                                                             4.4s 
 ✔ Container sentry-self-hosted-smtp-1                                      Started                                                                                                                                                                                             5.0s 
 ✔ Container sentry-self-hosted-zookeeper-1                                 Healthy                                                                                                                                                                                            35.5s 
 ✔ Container sentry-self-hosted-symbolicator-1                              Started                                                                                                                                                                                             4.5s 
 ✔ Container sentry-self-hosted-clickhouse-1                                Healthy                                                                                                                                                                                            36.6s 
 ✔ Container sentry-self-hosted-geoipupdate-1                               Started                                                                                                                                                                                             5.0s 
 ✔ Container sentry-self-hosted-postgres-1                                  Healthy                                                                                                                                                                                            50.2s 
 ✔ Container sentry-self-hosted-memcached-1                                 Started                                                                                                                                                                                             4.3s 
 ✔ Container sentry-self-hosted-kafka-1                                     Healthy                                                                                                                                                                                            50.2s 
 ✔ Container sentry-self-hosted-snuba-profiling-profiles-consumer-1         Started                                                                                                                                                                                            49.6s 
 ✔ Container sentry-self-hosted-snuba-replays-consumer-1                    Started                                                                                                                                                                                            49.6s 
 ✔ Container sentry-self-hosted-snuba-outcomes-consumer-1                   Started                                                                                                                                                                                            49.0s 
 ✔ Container sentry-self-hosted-snuba-subscription-consumer-events-1        Started                                                                                                                                                                                            49.6s 
 ✔ Container sentry-self-hosted-snuba-consumer-1                            Started                                                                                                                                                                                            49.7s 
 ✔ Container sentry-self-hosted-vroom-1                                     Started                                                                                                                                                                                            49.6s 
 ✔ Container sentry-self-hosted-snuba-replacer-1                            Started                                                                                                                                                                                            49.3s 
 ✔ Container sentry-self-hosted-snuba-transactions-consumer-1               Started                                                                                                                                                                                            48.7s 
 ✔ Container sentry-self-hosted-snuba-subscription-consumer-sessions-1      Started                                                                                                                                                                                            49.6s 
 ✔ Container sentry-self-hosted-snuba-profiling-functions-consumer-1        Started                                                                                                                                                                                            49.6s 
 ✔ Container sentry-self-hosted-snuba-subscription-consumer-transactions-1  Started                                                                                                                                                                                            49.7s 
 ✔ Container sentry-self-hosted-snuba-api-1                                 Started                                                                                                                                                                                            49.2s 
 ✔ Container sentry-self-hosted-snuba-sessions-consumer-1                   Started                                                                                                                                                                                            49.0s 
 ✔ Container sentry-self-hosted-worker-1                                    Started                                                                                                                                                                                            54.6s 
 ✔ Container sentry-self-hosted-post-process-forwarder-errors-1             Started                                                                                                                                                                                            54.4s 
 ✔ Container sentry-self-hosted-ingest-replay-recordings-1                  Started                                                                                                                                                                                            54.6s 
 ✔ Container sentry-self-hosted-web-1                                       Started                                                                                                                                                                                            54.4s 
 ✔ Container sentry-self-hosted-post-process-forwarder-transactions-1       Started                                                                                                                                                                                            54.6s 
 ✔ Container sentry-self-hosted-cron-1                                      Started                                                                                                                                                                                            54.3s 
 ⠇ Container sentry-self-hosted-sentry-cleanup-1                            Starting                                                                                                                                                                                           54.6s 
 ✔ Container sentry-self-hosted-events-consumer-1                           Started                                                                                                                                                                                            54.2s 
 ⠇ Container sentry-self-hosted-subscription-consumer-events-1              Starting                                                                                                                                                                                           54.6s 
 ✔ Container sentry-self-hosted-attachments-consumer-1                      Started                                                                                                                                                                                            54.6s 
 ✔ Container sentry-self-hosted-transactions-consumer-1                     Started                                                                                                                                                                                            54.4s 
 ⠇ Container sentry-self-hosted-subscription-consumer-transactions-1        Starting                                                                                                                                                                                           54.6s 
 ✔ Container sentry-self-hosted-ingest-profiles-1                           Started                                                                                                                                                                                            54.0s 
 ✔ Container sentry-self-hosted-relay-1                                     Created                                                                                                                                                                                             0.1s 
 ✔ Container sentry-self-hosted-nginx-1                                     Created                                                                                                                                                                                             0.1s 
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't get final child's PID from pipe: EOF: unknown

damnit ... this is frustrating :-(

[futureweb]

further update:

chmod 777 self-hosted
cd self-hosted
# chmod 777 /var/www/sentry/self-hosted/postgres/postgres-entrypoint.sh
# chmod 777 /var/www/sentry/self-hosted/postgres/init_hba.sh 
# chmod 777 /var/www/sentry/self-hosted/sentry/entrypoint.sh

Then the
'Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't get final child's PID from pipe: EOF: unknown'
Error was gone.

Will try to see if I can migrate old Data to new Server + then Update to latest Sentry

[aldy505]

I wonder if you could actually try Debian or Ubuntu just for the sake of it, and see if it suddenly works.....

[futureweb]

@aldy505 never worked with Debian so far - but yea - could create a Debian Machine and try. I guess it should work there when it's tested on Debian ^^
Just tried to avoid it as it would be the only Debian Machine in our Infrastructure with noone really used to work with Debian and also blowing up Virtuozzo Template Cache, Ploop Cache and some other things on the Root Servers ... :-/