-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Python SPIN App pod fails to start in SpinKube #202
Comments
Could you paste in the whole kubectl describe output? (Alongside kubectl events?) thanks! |
Sure thing @endocrimes. I just re-built and re-pushed to k3d to get a fresh set. Here is the describe pod: Name: py-spin-86c88d45b-t7xjg
Namespace: default
Priority: 0
Runtime Class Name: wasmtime-spin-v2
Service Account: default
Node: k3d-wasm-cluster-server-0/172.18.0.3
Start Time: Sat, 13 Apr 2024 09:33:17 -0500
Labels: core.spinoperator.dev/app-name=py-spin
core.spinoperator.dev/app.py-spin.status=ready
pod-template-hash=86c88d45b
Annotations: <none>
Status: Running
IP: 10.42.2.18
IPs:
IP: 10.42.2.18
Controlled By: ReplicaSet/py-spin-86c88d45b
Containers:
py-spin:
Container ID: containerd://5cc0054fc98e2a7a75d46eadcf2a6bada0360db878f00506866634c95484d941
Image: bolbeck/py-spin
Image ID: docker.io/bolbeck/py-spin@sha256:2c8d16c414fe050a430852317546af4fe0b485706e638ce553d8e70b4683e5e8
Port: 80/TCP
Host Port: 0/TCP
Command:
/
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 137
Started: Sat, 13 Apr 2024 09:33:42 -0500
Finished: Sat, 13 Apr 2024 09:33:43 -0500
Ready: False
Restart Count: 2
Environment: <none>
Mounts:
/runtime-config.toml from spin-runtime-config (ro,path="runtime-config.toml")
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vndh9 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
spin-runtime-config:
Type: Secret (a volume populated by a Secret)
SecretName: py-spin-760b133c
Optional: true
kube-api-access-vndh9:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 53s default-scheduler Successfully assigned default/py-spin-86c88d45b-t7xjg to k3d-wasm-cluster-server-0
Normal Pulled 51s kubelet Successfully pulled image "bolbeck/py-spin" in 2.467593251s (2.467613084s including waiting)
Normal Pulled 43s kubelet Successfully pulled image "bolbeck/py-spin" in 430.641583ms (430.664042ms including waiting)
Normal Pulling 29s (x3 over 54s) kubelet Pulling image "bolbeck/py-spin"
Normal Pulled 29s kubelet Successfully pulled image "bolbeck/py-spin" in 451.691292ms (451.716584ms including waiting)
Normal Created 29s (x3 over 51s) kubelet Created container py-spin
Normal Started 29s (x3 over 45s) kubelet Started container py-spin
Warning BackOff 15s (x4 over 43s) kubelet Back-off restarting failed container py-spin in pod py-spin-86c88d45b-t7xjg_default(33e98adb-aca5-448c-b331-b122d0f8b53c) And here is the Kube events: LAST SEEN TYPE REASON OBJECT MESSAGE
5h28m (x356 over 17h) Warning BackOff Pod/py-spin-86c88d45b-hbmwr Back-off restarting failed container py-spin in pod py-spin-86c88d45b-hbmwr_default(62754499-2370-4dea-8e16-9214f05d1cfd)
54m (x8 over 15h) Warning FailedToUpdateEndpoint Endpoints/py-spin Failed to update endpoint default/py-spin: Unauthorized
54m (x8 over 15h) Warning FailedToUpdateEndpointSlices Service/py-spin Error updating Endpoint Slices for Service default/py-spin: failed to update py-spin-x7s7k EndpointSlice for Service default/py-spin: Unauthorized
28m (x49 over 17h) Normal Started Pod/py-spin-86c88d45b-hbmwr Started container py-spin
28m (x388 over 17h) Warning BackOff Pod/py-spin-86c88d45b-r4nzd Back-off restarting failed container py-spin in pod py-spin-86c88d45b-r4nzd_default(305165a0-19a6-4026-98b2-e19c6c8ef850)
9m18s Normal SuccessfulCreate ReplicaSet/py-spin-86c88d45b Created pod: py-spin-86c88d45b-t7xjg
9m18s Normal ScalingReplicaSet Deployment/py-spin Scaled up replica set py-spin-86c88d45b to 2
9m18s (x2 over 21h) Warning FailedToCreateEndpoint Endpoints/py-spin Failed to create endpoint for service default/py-spin: endpoints "py-spin" already exists
9m18s Normal SuccessfulCreate ReplicaSet/py-spin-86c88d45b Created pod: py-spin-86c88d45b-sts2w
9m17s Normal Scheduled Pod/py-spin-86c88d45b-t7xjg Successfully assigned default/py-spin-86c88d45b-t7xjg to k3d-wasm-cluster-server-0
9m17s Normal Scheduled Pod/py-spin-86c88d45b-sts2w Successfully assigned default/py-spin-86c88d45b-sts2w to k3d-wasm-cluster-agent-1
9m15s Normal Pulled Pod/py-spin-86c88d45b-t7xjg Successfully pulled image "bolbeck/py-spin" in 2.467593251s (2.467613084s including waiting)
9m14s Normal Pulled Pod/py-spin-86c88d45b-sts2w Successfully pulled image "bolbeck/py-spin" in 3.228073377s (3.22807821s including waiting)
9m8s Warning FailedToUpdateEndpoint Endpoints/py-spin Failed to update endpoint default/py-spin: Operation cannot be fulfilled on endpoints "py-spin": the object has been modified; please apply your changes to the latest version and try again
9m7s Normal Pulled Pod/py-spin-86c88d45b-sts2w Successfully pulled image "bolbeck/py-spin" in 434.327167ms (434.361834ms including waiting)
9m7s Normal Pulled Pod/py-spin-86c88d45b-t7xjg Successfully pulled image "bolbeck/py-spin" in 430.641583ms (430.664042ms including waiting)
8m53s Normal Pulled Pod/py-spin-86c88d45b-t7xjg Successfully pulled image "bolbeck/py-spin" in 451.691292ms (451.716584ms including waiting)
8m53s Normal Pulled Pod/py-spin-86c88d45b-sts2w Successfully pulled image "bolbeck/py-spin" in 399.4945ms (399.508959ms including waiting)
8m25s Normal Pulled Pod/py-spin-86c88d45b-sts2w Successfully pulled image "bolbeck/py-spin" in 425.151917ms (425.177959ms including waiting)
8m25s (x4 over 9m14s) Normal Created Pod/py-spin-86c88d45b-sts2w Created container py-spin
8m24s (x4 over 9m8s) Normal Started Pod/py-spin-86c88d45b-sts2w Started container py-spin
8m24s Normal Pulled Pod/py-spin-86c88d45b-t7xjg Successfully pulled image "bolbeck/py-spin" in 349.710334ms (349.715875ms including waiting)
8m24s (x4 over 9m15s) Normal Created Pod/py-spin-86c88d45b-t7xjg Created container py-spin
8m24s (x4 over 9m9s) Normal Started Pod/py-spin-86c88d45b-t7xjg Started container py-spin
7m43s (x5 over 9m18s) Normal Pulling Pod/py-spin-86c88d45b-t7xjg Pulling image "bolbeck/py-spin"
7m43s Normal Pulled Pod/py-spin-86c88d45b-t7xjg Successfully pulled image "bolbeck/py-spin" in 414.483125ms (414.515375ms including waiting)
7m32s (x5 over 9m18s) Normal Pulling Pod/py-spin-86c88d45b-sts2w Pulling image "bolbeck/py-spin"
4m13s (x25 over 9m7s) Warning BackOff Pod/py-spin-86c88d45b-sts2w Back-off restarting failed container py-spin in pod py-spin-86c88d45b-sts2w_default(2024ad6b-c5d8-40f6-ad7f-fb4ef70d1c82)
4m6s (x25 over 9m7s) Warning BackOff Pod/py-spin-86c88d45b-t7xjg Back-off restarting failed container py-spin in pod py-spin-86c88d45b-t7xjg_default(33e98adb-aca5-448c-b331-b122d0f8b53c) Here is the describe for the replicaset, which interestingly enough does not show an error anymore: Name: py-spin-86c88d45b
Namespace: default
Selector: core.spinoperator.dev/app-name=py-spin,core.spinoperator.dev/app.py-spin.status=ready,pod-template-hash=86c88d45b
Labels: core.spinoperator.dev/app-name=py-spin
core.spinoperator.dev/app.py-spin.status=ready
pod-template-hash=86c88d45b
Annotations: deployment.kubernetes.io/desired-replicas: 2
deployment.kubernetes.io/max-replicas: 3
deployment.kubernetes.io/revision: 1
Controlled By: Deployment/py-spin
Replicas: 2 current / 2 desired
Pods Status: 2 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: core.spinoperator.dev/app-name=py-spin
core.spinoperator.dev/app.py-spin.status=ready
pod-template-hash=86c88d45b
Containers:
py-spin:
Image: bolbeck/py-spin
Port: 80/TCP
Host Port: 0/TCP
Command:
/
Environment: <none>
Mounts:
/runtime-config.toml from spin-runtime-config (ro,path="runtime-config.toml")
Volumes:
spin-runtime-config:
Type: Secret (a volume populated by a Secret)
SecretName: py-spin-760b133c
Optional: true
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 10m replicaset-controller Created pod: py-spin-86c88d45b-t7xjg
Normal SuccessfulCreate 10m replicaset-controller Created pod: py-spin-86c88d45b-sts2w Also, the image can be downloaded from docker hub as it is publicly available (bolbeck/py-spin) Thanks for your help |
I was able to repro this with a simple Python http application. This is an issue with the containerd shim, so i filed an issue there as well to track spinkube/containerd-shim-spin#84 Thank you @camba1 for raising this |
Thanks @kate-goldenring ! |
@camba1 there appears to be a compatibility issue with the 3.0+ python sdk. Can you install the 2.0 one? That should solve the issue |
@kate-goldenring Sorry of the delayed response, I was traveling last week. I tried your suggestion and I still get the same error, so I probably misunderstand what you meant. Here is what I did in my existing project:
spin template upgrade --repo https://github.com/fermyon/spin-python-sdk --branch v2.0
spin up --build
spin registry push --build bolbeck/py-spin
spin kube deploy --from bolbeck/py-spin At that point the pod is once again in CrashLoopBackOff. How can I tell if the application is using the new version of the template? None of the options in spin template command seem to show this info. describe pod: Name: py-spin-86c88d45b-ksglx
Namespace: default
Priority: 0
Runtime Class Name: wasmtime-spin-v2
Service Account: default
Node: k3d-wasm-cluster-agent-0/172.18.0.4
Start Time: Mon, 22 Apr 2024 08:46:38 -0500
Labels: core.spinoperator.dev/app-name=py-spin
core.spinoperator.dev/app.py-spin.status=ready
pod-template-hash=86c88d45b
Annotations: <none>
Status: Running
IP: 10.42.0.23
IPs:
IP: 10.42.0.23
Controlled By: ReplicaSet/py-spin-86c88d45b
Containers:
py-spin:
Container ID: containerd://86076db541770614941def75b43dbf688c920b9f0926450b5b3765775d4d7f1c
Image: bolbeck/py-spin
Image ID: docker.io/bolbeck/py-spin@sha256:e20be15ae4c7b93fe6c53fac3135d9c3b32174a41bf8e9ebbf35484dce4d3307
Port: 80/TCP
Host Port: 0/TCP
Command:
/
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 22 Apr 2024 10:01:35 -0500
Finished: Mon, 22 Apr 2024 10:01:35 -0500
Ready: False
Restart Count: 15
Environment: <none>
Mounts:
/runtime-config.toml from spin-runtime-config (ro,path="runtime-config.toml")
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-b6lvr (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
spin-runtime-config:
Type: Secret (a volume populated by a Secret)
SecretName: py-spin-760b133c
Optional: true
kube-api-access-b6lvr:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 78m default-scheduler Successfully assigned default/py-spin-86c88d45b-ksglx to k3d-wasm-cluster-agent-0
Normal Pulled 78m kubelet Successfully pulled image "bolbeck/py-spin" in 3.219803127s (3.219816793s including waiting)
Normal Pulled 77m kubelet Successfully pulled image "bolbeck/py-spin" in 376.047916ms (376.063167ms including waiting)
Normal Pulled 77m kubelet Successfully pulled image "bolbeck/py-spin" in 393.366001ms (393.39775ms including waiting)
Normal Pulled 77m kubelet Successfully pulled image "bolbeck/py-spin" in 425.157292ms (425.187458ms including waiting)
Normal Created 77m (x4 over 78m) kubelet Created container py-spin
Normal Started 77m (x4 over 77m) kubelet Started container py-spin
Normal Pulling 76m (x5 over 78m) kubelet Pulling image "bolbeck/py-spin"
Normal Pulled 76m kubelet Successfully pulled image "bolbeck/py-spin" in 451.551542ms (451.582583ms including waiting)
Warning BackOff 64s (x187 over 77m) kubelet Back-off restarting failed container py-spin in pod py-spin-86c88d45b-ksglx_default(4942c8d2-d71a-4105-adf3-76005689474e) Kube events: LAST SEEN TYPE REASON OBJECT MESSAGE
91m Normal ScalingReplicaSet Deployment/py-spin Scaled up replica set py-spin-86c88d45b to 2
91m Normal SuccessfulCreate ReplicaSet/py-spin-86c88d45b Created pod: py-spin-86c88d45b-8n47f
91m Normal SuccessfulCreate ReplicaSet/py-spin-86c88d45b Created pod: py-spin-86c88d45b-mr644
91m Normal Scheduled Pod/py-spin-86c88d45b-8n47f Successfully assigned default/py-spin-86c88d45b-8n47f to k3d-wasm-cluster-agent-0
91m Normal Scheduled Pod/py-spin-86c88d45b-mr644 Successfully assigned default/py-spin-86c88d45b-mr644 to k3d-wasm-cluster-server-0
91m Normal Pulled Pod/py-spin-86c88d45b-8n47f Successfully pulled image "bolbeck/py-spin" in 473.698333ms (473.72025ms including waiting)
91m Normal Pulled Pod/py-spin-86c88d45b-mr644 Successfully pulled image "bolbeck/py-spin" in 376.161417ms (376.167708ms including waiting)
91m Warning FailedToUpdateEndpoint Endpoints/py-spin Failed to update endpoint default/py-spin: Operation cannot be fulfilled on endpoints "py-spin": the object has been modified; please apply your changes to the latest version and try again
91m Normal Pulled Pod/py-spin-86c88d45b-mr644 Successfully pulled image "bolbeck/py-spin" in 384.564875ms (384.574917ms including waiting)
91m Normal Pulled Pod/py-spin-86c88d45b-8n47f Successfully pulled image "bolbeck/py-spin" in 403.642375ms (403.663625ms including waiting)
91m Normal Pulled Pod/py-spin-86c88d45b-8n47f Successfully pulled image "bolbeck/py-spin" in 396.631376ms (396.662958ms including waiting)
91m Normal Pulled Pod/py-spin-86c88d45b-mr644 Successfully pulled image "bolbeck/py-spin" in 406.33475ms (406.350042ms including waiting)
90m Normal Pulled Pod/py-spin-86c88d45b-8n47f Successfully pulled image "bolbeck/py-spin" in 617.745459ms (617.777917ms including waiting)
90m (x4 over 91m) Normal Started Pod/py-spin-86c88d45b-8n47f Started container py-spin
90m Normal Pulled Pod/py-spin-86c88d45b-mr644 Successfully pulled image "bolbeck/py-spin" in 366.54975ms (366.571251ms including waiting)
90m (x4 over 91m) Normal Created Pod/py-spin-86c88d45b-mr644 Created container py-spin
90m (x4 over 91m) Normal Started Pod/py-spin-86c88d45b-mr644 Started container py-spin
90m (x6 over 91m) Warning BackOff Pod/py-spin-86c88d45b-8n47f Back-off restarting failed container py-spin in pod py-spin-86c88d45b-8n47f_default(ca663fe6-0b34-4413-afdb-a4242379b577)
90m (x7 over 91m) Warning BackOff Pod/py-spin-86c88d45b-mr644 Back-off restarting failed container py-spin in pod py-spin-86c88d45b-mr644_default(913510d2-e33f-4b77-a1ef-4962587b252a)
90m (x5 over 91m) Normal Pulling Pod/py-spin-86c88d45b-8n47f Pulling image "bolbeck/py-spin"
90m Normal Pulled Pod/py-spin-86c88d45b-8n47f Successfully pulled image "bolbeck/py-spin" in 449.55775ms (449.583625ms including waiting)
90m (x5 over 91m) Normal Created Pod/py-spin-86c88d45b-8n47f Created container py-spin
90m (x5 over 91m) Normal Pulling Pod/py-spin-86c88d45b-mr644 Pulling image "bolbeck/py-spin"
90m Normal Pulled Pod/py-spin-86c88d45b-mr644 Successfully pulled image "bolbeck/py-spin" in 391.837042ms (391.849959ms including waiting)
79m Normal ScalingReplicaSet Deployment/py-spin Scaled up replica set py-spin-86c88d45b to 2
79m Normal SuccessfulCreate ReplicaSet/py-spin-86c88d45b Created pod: py-spin-86c88d45b-ksglx
79m Normal SuccessfulCreate ReplicaSet/py-spin-86c88d45b Created pod: py-spin-86c88d45b-nh47s
79m (x3 over 13h) Warning FailedToCreateEndpoint Endpoints/py-spin Failed to create endpoint for service default/py-spin: endpoints "py-spin" already exists
79m Normal Scheduled Pod/py-spin-86c88d45b-ksglx Successfully assigned default/py-spin-86c88d45b-ksglx to k3d-wasm-cluster-agent-0
79m Normal Scheduled Pod/py-spin-86c88d45b-nh47s Successfully assigned default/py-spin-86c88d45b-nh47s to k3d-wasm-cluster-server-0
79m Normal Pulled Pod/py-spin-86c88d45b-nh47s Successfully pulled image "bolbeck/py-spin" in 2.868770209s (2.868788418s including waiting)
79m Normal Pulled Pod/py-spin-86c88d45b-ksglx Successfully pulled image "bolbeck/py-spin" in 3.219803127s (3.219816793s including waiting)
79m Warning FailedToUpdateEndpoint Endpoints/py-spin Failed to update endpoint default/py-spin: Operation cannot be fulfilled on endpoints "py-spin": the object has been modified; please apply your changes to the latest version and try again
79m Normal Pulled Pod/py-spin-86c88d45b-ksglx Successfully pulled image "bolbeck/py-spin" in 376.047916ms (376.063167ms including waiting)
79m Normal Pulled Pod/py-spin-86c88d45b-nh47s Successfully pulled image "bolbeck/py-spin" in 381.368583ms (381.38075ms including waiting)
79m Normal Pulled Pod/py-spin-86c88d45b-ksglx Successfully pulled image "bolbeck/py-spin" in 393.366001ms (393.39775ms including waiting)
79m Normal Pulled Pod/py-spin-86c88d45b-nh47s Successfully pulled image "bolbeck/py-spin" in 415.871167ms (415.896709ms including waiting)
78m Normal Pulled Pod/py-spin-86c88d45b-ksglx Successfully pulled image "bolbeck/py-spin" in 425.157292ms (425.187458ms including waiting)
78m (x4 over 79m) Normal Created Pod/py-spin-86c88d45b-ksglx Created container py-spin
78m (x4 over 79m) Normal Started Pod/py-spin-86c88d45b-ksglx Started container py-spin
78m Normal Pulled Pod/py-spin-86c88d45b-nh47s Successfully pulled image "bolbeck/py-spin" in 399.768084ms (399.800167ms including waiting)
78m (x4 over 79m) Normal Created Pod/py-spin-86c88d45b-nh47s Created container py-spin
78m (x4 over 79m) Normal Started Pod/py-spin-86c88d45b-nh47s Started container py-spin
78m (x5 over 79m) Normal Pulling Pod/py-spin-86c88d45b-ksglx Pulling image "bolbeck/py-spin"
78m Normal Pulled Pod/py-spin-86c88d45b-ksglx Successfully pulled image "bolbeck/py-spin" in 451.551542ms (451.582583ms including waiting)
77m (x5 over 79m) Normal Pulling Pod/py-spin-86c88d45b-nh47s Pulling image "bolbeck/py-spin"
77m Normal Pulled Pod/py-spin-86c88d45b-nh47s Successfully pulled image "bolbeck/py-spin" in 413.693083ms (413.716084ms including waiting)
29m Warning FailedToUpdateEndpoint Endpoints/py-spin Failed to update endpoint default/py-spin: Unauthorized
29m Warning FailedToUpdateEndpointSlices Service/py-spin Error updating Endpoint Slices for Service default/py-spin: failed to update py-spin-x5n9j EndpointSlice for Service default/py-spin: Unauthorized
2m30s (x187 over 79m) Warning BackOff Pod/py-spin-86c88d45b-ksglx Back-off restarting failed container py-spin in pod py-spin-86c88d45b-ksglx_default(4942c8d2-d71a-4105-adf3-76005689474e)
2m23s (x185 over 79m) Warning BackOff Pod/py-spin-86c88d45b-nh47s Back-off restarting failed container py-spin in pod py-spin-86c88d45b-nh47s_default(726da669-4880-4810-b8a4-9e74f118f9ee) |
@camba1 thank you for giving it a try. After downloading the older template, you will need to scaffold a new application with the template and move your business logic over to that: So after |
@camba1 alternatively, you should be able to just update spin-sdk == 2.0.0
componentize-py == 0.11.2 |
Ok, cool. That did it . Thanks @kate-goldenring ! |
What Were you doing?
Trying to deploy minimal sample Python SPIN container in SpinKube
What did you expect?
Container to deploy in SpinKube and start normally
What happened?
Pod fails to start and reports CrashLoopBackOff status
Additional info:
Tried the same thing with a Go container and that works without issues. So SpinKube deployment was done properly into k3d
Steps to reproduce:
At this point the application works normally, now we deploy to k3d
Checking the deployment of the pod:
Pods in CrashLoopBackOff status
In the pod description:
Created container py-spin
Started container py-spin
Back-off restarting failed container py-spin in pod py-spin-86c88d45b-r4nzd_default(305165a0-19a6-4026-98b2-e19c6c8ef850)
Note that pod logs show do not show anything
Error in the replicate set description: Error creating: Unauthorized
Environment
k3d version v5.6.0
k3s version v1.27.5-k3s1 (default)
spin-operator 0.1.0
The text was updated successfully, but these errors were encountered: