[Question]: "An empty namespace may not be set during creation" error #1107

electrocucaracha · 2024-05-10T01:16:08Z

Checklist

I've searched for similar issues and couldn't find anything matching
I've included steps to reproduce the behavior

Affected Components

K8sGPT (CLI)
K8sGPT Operator

K8sGPT Version

v0.3.30

Kubernetes Version

v1.28.6

Host OS and its Version

Ubuntu 18.04.6 LTS

Steps to reproduce

Deploy LocalAI services
Deploy the K8sGPT Operator v1.4
Wait for Operator readiness

$ kubectl logs -n k8sgpt-operator-system k8sgpt-operator-controller-manager-695d646b5d-bffth -f
2024-05-10T00:54:35Z    INFO    controller-runtime.metrics      Metrics server is starting to listen    {"addr": "127.0.0.1:8080"}
2024-05-10T00:54:35Z    INFO    setup   starting manager                                                                                                     
2024-05-10T00:54:35Z    INFO    Starting server {"kind": "health probe", "addr": "[::]:8081"}       
2024-05-10T00:54:35Z    INFO    starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"}
I0510 00:54:35.546330       1 leaderelection.go:250] attempting to acquire leader lease k8sgpt-operator-system/ea9c19f7.k8sgpt.ai...                         
I0510 00:54:35.595158       1 leaderelection.go:260] successfully acquired lease k8sgpt-operator-system/ea9c19f7.k8sgpt.ai
2024-05-10T00:54:35Z    INFO    Starting EventSource    {"controller": "k8sgpt", "controllerGroup": "core.k8sgpt.ai", "controllerKind": "K8sGPT", "source": "kind source: *v1alpha1.K8sGPT"}                                                                                                                               
2024-05-10T00:54:35Z    DEBUG   events  k8sgpt-operator-controller-manager-695d646b5d-bffth_a11702aa-4a27-4657-9cdd-b33168857e8b became leader  {"type": "Normal", "object": {"kind":"Lease","namespace":"k8sgpt-operator-system","name":"ea9c19f7.k8sgpt.ai","uid":"811341dd-5fe9-48fd-b744-5a4fe6245690","apiVersion":"co
ordination.k8s.io/v1","resourceVersion":"26610"}, "reason": "LeaderElection"}                                                                                                                                                                                                                                              
2024-05-10T00:54:35Z    INFO    Starting Controller     {"controller": "k8sgpt", "controllerGroup": "core.k8sgpt.ai", "controllerKind": "K8sGPT"}                                                                                                                                                                          
2024-05-10T00:54:35Z    INFO    Starting workers        {"controller": "k8sgpt", "controllerGroup": "core.k8sgpt.ai", "controllerKind": "K8sGPT", "worker count": 1}

Deploy the following K8sGPT CR:

apiVersion: core.k8sgpt.ai/v1alpha1
kind: K8sGPT
metadata:
  name: k8sgpt-local
  namespace: k8sgpt-operator-system
spec:
  ai:
    model: gpt-4
    backend: localai
    baseUrl: http://local-ai.local-ai-system.svc.cluster.local:8080/v1
    enabled: true
  version: v0.3.30
  noCache: false

Wait for CR readiness checking operator's logs:

Finished Reconciling k8sGPT                                                                                                                                  
Finished Reconciling k8sGPT                                                                                                                                  
Creating new client for 10.233.1.108:8080                                                                                                                    
Connection established between 10.233.1.108:8080 and localhost with time out of 1 seconds.          
Remote Address : 10.233.1.108:8080                                                                                                                           
K8sGPT address: 10.233.1.108:8080

Deploy a broken pod:

apiVersion: v1
kind: Pod
metadata:
  name: broken-pod
spec:
  containers:
    - name: broken-pod
      image: nginx:1.a.b.c
      livenessProbe:
        httpGet:
          path: /
          port: 90
        initialDelaySeconds: 3
        periodSeconds: 3

Getting error in Operator's logs:

Finished Reconciling k8sGPT with error: an empty namespace may not be set during creation                                                                                                                                                                                                                                  
2024-05-10T00:59:16Z    ERROR   Reconciler error        {"controller": "k8sgpt", "controllerGroup": "core.k8sgpt.ai", "controllerKind": "K8sGPT", "K8sGPT": {"name":"k8sgpt-local","namespace":"k8sgpt-operator-system"}, "namespace": "k8sgpt-operator-system", "name": "k8sgpt-local", "reconcileID": "4f9ace39-1e48-4211
-8a56-e7f78a37bebb", "error": "an empty namespace may not be set during creation"}                                                                                                                                                                                                                                         
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler                                                                        
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem            
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2                  
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226
Creating new client for 10.233.1.108:8080                                                                                                                    
Connection established between 10.233.1.108:8080 and localhost with time out of 1 seconds.                                                                   
Remote Address : 10.233.1.108:8080                                                                                                                           
K8sGPT address: 10.233.1.108:8080                                                                                                                            
Finished Reconciling k8sGPT with error: an empty namespace may not be set during creation

Expected behaviour

I expect some results resources

Actual behaviour

$ kubectl get results.core.k8sgpt.ai -A
No resources found

Additional Information

No response

calebyyy · 2024-05-10T08:03:20Z

Hi I am getting this issue too did u manage to resolve it?

JuHyung-Son · 2024-05-13T14:47:37Z

would you re install k8sgpt and operator?

electrocucaracha · 2024-05-13T15:23:25Z

@JuHyung-Son I don't mind giving it a try but could you explain what was changed on the release version that I tested?

electrocucaracha · 2024-05-15T01:54:38Z

@JuHyung-Son I suspect that this error could be caused by PodAnalyzer sending an empty namespace value to the Analyzer

Nitishmane · 2024-05-15T18:43:03Z

I am seeing a similar issue. However, providing a 'targetNamespace' generates results for only that particular namespace.

electrocucaracha · 2024-05-15T21:02:56Z

I am seeing a similar issue. However, providing a 'targetNamespace' generates results for only that particular namespace.

It seems like that is the missing part, thanks @Nitishmane for the solution.

This is the Operator CR:

apiVersion: core.k8sgpt.ai/v1alpha1
kind: K8sGPT
metadata:
  name: k8sgpt-local
  namespace: k8sgpt-operator-system
spec:
  ai:
    model: gpt-4
    backend: localai
    baseUrl: http://local-ai.local-ai-system.svc.cluster.local:8080/v1
    enabled: true
  version: v0.3.30
  noCache: false
  targetNamespace: default

and this is the pod broken example

apiVersion: v1
kind: Pod
metadata:
  name: broken-pod
  namespace: default
spec:
  containers:
    - name: broken-pod
      image: nginx:1.a.b.c
      livenessProbe:
        httpGet:
          path: /
          port: 90
        initialDelaySeconds: 3
        periodSeconds: 3

Which generates the following result:

$ kubectl describe results.core.k8sgpt.ai 
Name:         defaultbrokenpod
Namespace:    default
Labels:       k8sgpts.k8sgpt.ai/backend=localai
              k8sgpts.k8sgpt.ai/name=k8sgpt-local
              k8sgpts.k8sgpt.ai/namespace=k8sgpt-operator-system
Annotations:  <none>
API Version:  core.k8sgpt.ai/v1alpha1
Kind:         Result
Metadata:
  Creation Timestamp:  2024-05-15T20:58:20Z
  Generation:          1
  Resource Version:    3550
  UID:                 d34c1a0b-ee29-477e-b721-0f29bef9d7e7
Spec:
  Backend:  localai
  Details:  Error: Back-off pulling image "nginx:1.a.b.c" indicates that Kubernetes is unable to pull the specified Docker image due to network issues or registry unavailability.
Solution: 
1. Check the internet connection: Ensure that the Kubernetes cluster has stable network access to the Docker registry hosting the image.
2. Verify image URL: Double-check the image name and tag "nginx:1.a.b.c" for any typos or missing information.
3. Pull image manually: Run the command "docker pull nginx:1.a.b.c" outside of Kubernetes to ensure that the image can be pulled successfully.
4. Check image availability: Confirm that the specified image version "1.a.b.c" is available on the registry.
5. Update image pull policy: If the image is available, consider updating the image pull policy in the pod's specification to "IfNotPresent" or "Always" to force Kubernetes to pull the image.
  Error:
    Text:         Back-off pulling image "nginx:1.a.b.c"
  Kind:           Pod
  Name:           default/broken-pod
  Parent Object:  
Events:           <none>

g3david405 · 2024-05-17T07:16:44Z

electrocucaracha
so k8sgpt operator can analyze for only one namespace now?

electrocucaracha · 2024-05-17T17:39:48Z

electrocucaracha so k8sgpt operator can analyze for only one namespace now?

I'm not sure if the operator was designed to work in that way, but it looks like that's the intention.

MartyB-007 · 2024-05-21T12:37:15Z

I'm also seeing the same issue on v0.3.32, it doesn't feel like the intent is to force a target namespace, otherwise we'd have to run one K8sGPT per namespace on the cluster to get a clear view in the results crd.. seems like a maintenance nightmare. Unless I'm missing it the documentation has never alluded to this need

JuHyung-Son · 2024-05-21T14:09:10Z

i guess this is bug, i will take a look

g3david405 · 2024-05-21T14:31:10Z

@JuHyung-Son thank you!

JuHyung-Son · 2024-05-21T16:13:48Z

this will be fixed soon.
k8sgpt-ai/k8sgpt-operator#449

JuHyung-Son · 2024-05-22T15:03:26Z

it's done. Test with a new version
k8sgpt-ai/k8sgpt-operator#449

MartyB-007 · 2024-05-23T13:35:26Z

Hey @JuHyung-Son thanks for the fast reply and fast fix! I just tested and it is working now. Thank you :)

electrocucaracha closed this as completed May 15, 2024

This was referenced May 17, 2024

an empty namespace may not be set during creation for k8sgpt operator controller manager #1116

Closed

an empty namespace may not be set during creation for k8sgpt operator controller manager k8sgpt-ai/k8sgpt-operator#447

Closed

JuHyung-Son mentioned this issue May 21, 2024

fix: rollback namespace bug k8sgpt-ai/k8sgpt-operator#449

Merged

4 tasks

electrocucaracha reopened this May 21, 2024

JuHyung-Son closed this as completed May 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Question]: "An empty namespace may not be set during creation" error #1107

[Question]: "An empty namespace may not be set during creation" error #1107

electrocucaracha commented May 10, 2024

calebyyy commented May 10, 2024

JuHyung-Son commented May 13, 2024

electrocucaracha commented May 13, 2024

electrocucaracha commented May 15, 2024

Nitishmane commented May 15, 2024

electrocucaracha commented May 15, 2024 •

edited

g3david405 commented May 17, 2024 •

edited

electrocucaracha commented May 17, 2024

MartyB-007 commented May 21, 2024 •

edited

JuHyung-Son commented May 21, 2024

g3david405 commented May 21, 2024

JuHyung-Son commented May 21, 2024

JuHyung-Son commented May 22, 2024

MartyB-007 commented May 23, 2024

[Question]: "An empty namespace may not be set during creation" error #1107

[Question]: "An empty namespace may not be set during creation" error #1107

Comments

electrocucaracha commented May 10, 2024

Checklist

Affected Components

K8sGPT Version

Kubernetes Version

Host OS and its Version

Steps to reproduce

Expected behaviour

Actual behaviour

Additional Information

calebyyy commented May 10, 2024

JuHyung-Son commented May 13, 2024

electrocucaracha commented May 13, 2024

electrocucaracha commented May 15, 2024

Nitishmane commented May 15, 2024

electrocucaracha commented May 15, 2024 • edited

g3david405 commented May 17, 2024 • edited

electrocucaracha commented May 17, 2024

MartyB-007 commented May 21, 2024 • edited

JuHyung-Son commented May 21, 2024

g3david405 commented May 21, 2024

JuHyung-Son commented May 21, 2024

JuHyung-Son commented May 22, 2024

MartyB-007 commented May 23, 2024

electrocucaracha commented May 15, 2024 •

edited

g3david405 commented May 17, 2024 •

edited

MartyB-007 commented May 21, 2024 •

edited