Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Installation issues: How to add other sub-clusters to the parent cluster #785

Open
googs1025 opened this issue Jan 25, 2024 · 6 comments
Open

Installation issues: How to add other sub-clusters to the parent cluster #785

googs1025 opened this issue Jan 25, 2024 · 6 comments
Labels
kind/question Further information is requested

Comments

@googs1025
Copy link

避免来回翻译，以下直接使用中文。

您好，目前正在研究多集群，对 clusternet 项目很有兴趣。
自己使用环境部署了一下，但是发现 clusternet-agent 加入父集群失败。

使用环境：

腾讯云 ECS 4C8G
k8s version: v1.20.15
多集群工具： kind

问题：
目前参考 安装文档 使用 helm 部署。目前使用 kind 创建 mater-cluster cluster2 模拟父集群与子集群

[root@VM-0-13-centos ~]# kubectl get node
NAME                           STATUS   ROLES                  AGE    VERSION
master-cluster-control-plane   Ready    control-plane,master   4d8h   v1.20.15
[root@VM-0-13-centos ~]# kubectl get node
NAME                           STATUS   ROLES                  AGE    VERSION
master-cluster-control-plane   Ready    control-plane,master   4d9h   v1.20.15
[root@VM-0-13-centos ~]# kubectl --context kind-cluster2 get node
NAME                     STATUS   ROLES                  AGE     VERSION
cluster2-control-plane   Ready    control-plane,master   3d23h   v1.20.15
cluster2-worker          Ready    <none>                 3d23h   v1.20.15
  • 父集群 cluster-hub 部署完成
[root@VM-0-13-centos ~]# kubectl get pods -nclusternet-system
NAME                                            READY   STATUS    RESTARTS   AGE
clusternet-controller-manager-594549889-7h5vk   1/1     Running   0          3d16h
clusternet-controller-manager-594549889-cwv7x   1/1     Running   0          3d16h
clusternet-controller-manager-594549889-nv4jb   1/1     Running   0          3d16h
clusternet-hub-5d9f4bd454-5n8n4                 1/1     Running   0          3d17h
clusternet-hub-5d9f4bd454-tg78b                 1/1     Running   0          3d17h
clusternet-hub-5d9f4bd454-xd75g                 1/1     Running   0          3d17h
clusternet-scheduler-bb5b68d5-km9vm             1/1     Running   1          3d16h
clusternet-scheduler-bb5b68d5-pjw7z             1/1     Running   1          3d16h
clusternet-scheduler-bb5b68d5-qn9rk             1/1     Running   1          3d16h
[root@VM-0-13-centos ~]# kubectl get secret -n clusternet-system -o=jsonpath='{.items[?(@.metadata.annotations.kubernetes\.io/service-account\.name=="cluster-bootstrap-use")].data.token}' | base64 -i --decode; echo
eyJhbGciOiJSUzI1NiIsImtpZCI6ImNlVVVZRlNfRm5qZHgxcnpsQjBwR0NfX0dyeDltTlBRbWFhMjN4ODVBam8ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjbHVzdGVybmV0LXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbHVzdGVyLWJvb3RzdHJhcC11c2UiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiY2x1c3Rlci1ib290c3RyYXAtdXNlIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNGExMjk0YmEtZjdmMi00ZDE5LWJjMGQtMmEzYWI3ZmZkZTJmIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmNsdXN0ZXJuZXQtc3lzdGVtOmNsdXN0ZXItYm9vdHN0cmFwLXVzZSJ9.IVGcC23mkbQdvmMlfBOfXJJtlAYAfi6Y-xjmmI_ORyQJVsZhd571kIn28O2XdaBVOP3V-8-98M_6w6h2Im6-qzaCIkz_0YgB-DIa7ur5iP43wSH5TwOvW0zduuDCop-WcbQNJ-g07MvF_cpIZO_-yUA_yhAjKp_t_mcKja28WMYX4zcr_Z0Sla-8XMyc1NPoMflybN2MATLUxzwB9yNc4-uiNm1hvt5-84uvLbXfrPtnAQwSj19cR_vs89ebJRGFGo--FA97KxdeAFHeM1Knxn9k8VS5qcRfCwFKG_qXaFuMfHiRJe9__Y9W_a1UIh1s7oWvKeWViAX8phn0746e0weyJhbGciOiJSUzI1NiIsImtpZCI6ImNlVVVZRlNfRm5qZHgxcnpsQjBwR0NfX0dyeDltTlBRbWFhMjN4ODVBam8ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjbHVzdGVybmV0LXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbHVzdGVyLWJvb3RzdHJhcC11c2UtdG9rZW4tOTRoY2siLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiY2x1c3Rlci1ib290c3RyYXAtdXNlIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNGExMjk0YmEtZjdmMi00ZDE5LWJjMGQtMmEzYWI3ZmZkZTJmIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmNsdXN0ZXJuZXQtc3lzdGVtOmNsdXN0ZXItYm9vdHN0cmFwLXVzZSJ9.MRUGHHWstT4uCAytXnHgQ36aLrt9wtelQCjdQn3UGmu-WFO54mIjCry-RRj9maulBCp925au5OoyvuUyPwJNudL5RgCG7vL9OQU3kLLFSAD3QFZe6_lTxfCPmE3J0TNI2N_A5yEOPaIyH8Rqxnvtk-zrTwEDsOG3z-Dm4u0WrPLezvQ8KwNGBV55O1LRlzoj926mcAtGWCl-VwB7eDMxI5orPHVbuJ-k5-oeg8-KMXyWOBruebRTSNxqNEgiNcFm9xeahiUEv02AKV0EYbiLMvYqKqykY5APWcotr2G-Qeowou4CyukNpCcMCfZKWT0doK3FoKJhz62CoscK8xz8aQ
[root@VM-0-13-centos ~]#
  • clusternet-agent
    切换 context 后才部署的 在 cluster2 下部署
[root@VM-0-13-centos clusternet-agent]# helm install clusternet-agent -n clusternet-system --create-namespace   --set parentURL=https://172.18.0.2:6443   --set registrationToken=eyJhbGciOiJSUzI1NiIsImtpZCI6ImNlVVVZRlNfRm5qZHgxcnpsQjBwR0NfX0dyeDltTlBRbWFhMjN4ODVBam8ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjbHVzdGVybmV0LXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbHVzdGVyLWJvb3RzdHJhcC11c2UiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiY2x1c3Rlci1ib290c3RyYXAtdXNlIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNGExMjk0YmEtZjdmMi00ZDE5LWJjMGQtMmEzYWI3ZmZkZTJmIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmNsdXN0ZXJuZXQtc3lzdGVtOmNsdXN0ZXItYm9vdHN0cmFwLXVzZSJ9.IVGcC23mkbQdvmMlfBOfXJJtlAYAfi6Y-xjmmI_ORyQJVsZhd571kIn28O2XdaBVOP3V-8-98M_6w6h2Im6-qzaCIkz_0YgB-DIa7ur5iP43wSH5TwOvW0zduuDCop-WcbQNJ-g07MvF_cpIZO_-yUA_yhAjKp_t_mcKja28WMYX4zcr_Z0Sla-8XMyc1NPoMflybN2MATLUxzwB9yNc4-uiNm1hvt5-84uvLbXfrPtnAQwSj19cR_vs89ebJRGFGo--FA97KxdeAFHeM1Knxn9k8VS5qcRfCwFKG_qXaFuMfHiRJe9__Y9W_a1UIh1s7oWvKeWViAX8phn0746e0weyJhbGciOiJSUzI1NiIsImtpZCI6ImNlVVVZRlNfRm5qZHgxcnpsQjBwR0NfX0dyeDltTlBRbWFhMjN4ODVBam8ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjbHVzdGVybmV0LXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbHVzdGVyLWJvb3RzdHJhcC11c2UtdG9rZW4tOTRoY2siLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiY2x1c3Rlci1ib290c3RyYXAtdXNlIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNGExMjk0YmEtZjdmMi00ZDE5LWJjMGQtMmEzYWI3ZmZkZTJmIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmNsdXN0ZXJuZXQtc3lzdGVtOmNsdXN0ZXItYm9vdHN0cmFwLXVzZSJ9.MRUGHHWstT4uCAytXnHgQ36aLrt9wtelQCjdQn3UGmu-WFO54mIjCry-RRj9maulBCp925au5OoyvuUyPwJNudL5RgCG7vL9OQU3kLLFSAD3QFZe6_lTxfCPmE3J0TNI2N_A5yEOPaIyH8Rqxnvtk-zrTwEDsOG3z-Dm4u0WrPLezvQ8KwNGBV55O1LRlzoj926mcAtGWCl-VwB7eDMxI5orPHVbuJ-k5-oeg8-KMXyWOBruebRTSNxqNEgiNcFm9xeahiUEv02AKV0EYbiLMvYqKqykY5APWcotr2G-Qeowou4CyukNpCcMCfZKWT0doK3FoKJhz62CoscK8xz8aQ \ .
NAME: clusternet-agent
LAST DEPLOYED: Thu Jan 25 09:02:40 2024
NAMESPACE: clusternet-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
[root@VM-0-13-centos clusternet-agent]# kubectl get pods -nclusternet-system
NAME                               READY   STATUS    RESTARTS   AGE
clusternet-agent-9d5b74997-2kww8   1/1     Running   0          13s
clusternet-agent-9d5b74997-7m9rr   1/1     Running   0          13s
clusternet-agent-9d5b74997-k58w4   1/1     Running   0          13s
[root@VM-0-13-centos clusternet-agent]# kubectl logs clusternet-agent-9d5b74997-2kww8 -nclusternet-system
I0125 01:02:42.384817       1 serving.go:348] Generated self-signed cert in-memory
I0125 01:02:42.386130       1 secure_serving.go:210] Serving securely on [::]:10650
I0125 01:02:42.386152       1 agent.go:226] starting agent controller ...
I0125 01:02:42.386195       1 leaderelection.go:248] attempting to acquire leader lease clusternet-system/clusternet-agent...
I0125 01:02:42.386211       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0125 01:02:42.390876       1 agent.go:263] new leader elected: clusternet-agent-9d5b74997-rrp5s_146b6822-e2b7-497e-98f6-6b4eefc035e8
I0125 01:02:59.385754       1 agent.go:263] new leader elected: clusternet-agent-9d5b74997-k58w4_cb03fe7c-333e-48c9-b824-793e6b2e6b22
[root@VM-0-13-centos clusternet-agent]# kubectl logs clusternet-agent-9d5b74997-k58w4 -nclusternet-system
I0125 01:02:41.887464       1 serving.go:348] Generated self-signed cert in-memory
I0125 01:02:41.889755       1 secure_serving.go:210] Serving securely on [::]:10650
I0125 01:02:41.889779       1 agent.go:226] starting agent controller ...
I0125 01:02:41.889818       1 leaderelection.go:248] attempting to acquire leader lease clusternet-system/clusternet-agent...
I0125 01:02:41.890164       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0125 01:02:41.894768       1 agent.go:263] new leader elected: clusternet-agent-9d5b74997-rrp5s_146b6822-e2b7-497e-98f6-6b4eefc035e8
I0125 01:02:57.222801       1 leaderelection.go:258] successfully acquired lease clusternet-system/clusternet-agent
I0125 01:02:57.223025       1 agent.go:326] start registering current cluster as a child cluster...
I0125 01:02:57.223046       1 agent.go:336] retrieving cluster id
I0125 01:02:57.226877       1 agent.go:341] current cluster id is "9add3aec-9faa-42ab-ac76-22228e41f707"
I0125 01:02:57.228406       1 agent.go:443] try to bootstrap cluster registration if needed
E0125 01:02:57.232793       1 agent.go:378] failed to create ClusterRegistrationRequest: Unauthorized
W0125 01:02:57.232811       1 agent.go:379] something went wrong when using existing parent cluster credentials, switch to use bootstrap token instead
I0125 01:03:03.014886       1 agent.go:443] try to bootstrap cluster registration if needed
E0125 01:03:03.018432       1 agent.go:378] failed to create ClusterRegistrationRequest: Unauthorized
W0125 01:03:03.018452       1 agent.go:379] something went wrong when using existing parent cluster credentials, switch to use bootstrap token instead
I0125 01:03:08.433278       1 agent.go:443] try to bootstrap cluster registration if needed
E0125 01:03:08.434590       1 agent.go:378] failed to create ClusterRegistrationRequest: Unauthorized

问题：目前发现 子集群中报 failed to create ClusterRegistrationRequest 请问是哪步骤出问题呢？
感谢抽空回答～～
@dixudx
Copy link
Member

dixudx commented Jan 25, 2024 •
edited

E0125 01:03:08.434590 1 agent.go:378] failed to create ClusterRegistrationRequest: Unauthorized

@googs1025 Please apply below step in parent cluster.

# this will create a serviceaccount token
kubectl apply -f https://raw.githubusercontent.com/clusternet/clusternet/main/manifests/samples/cluster_serviceaccount_token.yaml

@dixudx
Copy link
Member

dixudx commented Jan 25, 2024

@googs1025 Also you can follow this Quick Start Guide, which will setup Clusternet locally with 1 parent cluster and 3 child clusters by using kind.

@googs1025
Copy link
Author

googs1025 commented Jan 25, 2024 •
edited

@dixudx
查了一下 kind 确实有这个字段 --enable-bootstrap-token-auth=true

[root@VM-0-13-centos clusternet-agent]# docker ps
CONTAINER ID   IMAGE                   COMMAND                   CREATED      STATUS      PORTS                                                                                                 NAMES
a41a3ba28b2c   kindest/node:v1.20.15   "/usr/local/bin/entr…"   4 days ago   Up 4 days   0.0.0.0:40080->80/tcp, 0.0.0.0:40443->443/tcp                                                         cluster2-worker
ef621726e649   kindest/node:v1.20.15   "/usr/local/bin/entr…"   4 days ago   Up 4 days   0.0.0.0:6444->6443/tcp, 0.0.0.0:46443->6444/tcp                                                       cluster2-control-plane
e68974e6f705   kindest/node:v1.20.15   "/usr/local/bin/entr…"   4 days ago   Up 4 days   0.0.0.0:6443->6443/tcp, 0.0.0.0:30009->30009/tcp, 0.0.0.0:31443->31443/tcp, 0.0.0.0:36443->6443/tcp   master-cluster-control-plane
[root@VM-0-13-centos clusternet-agent]# docker exec -it ef621726e649 cat /etc/kubernetes/manifests/kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 172.18.0.4:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=172.18.0.4
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt

之前没找到是因为 一直以为 bootstrap-token-07401b 是在 clusternet-system 命名空间中，原来是在 kube-system 。。。
重新部署一次 应该是没问题了！

[root@VM-0-13-centos clusternet-agent]# helm install clusternet-agent -n clusternet-system --create-namespace   --set parentURL=https://172.18.0.2:6443   --set registrationToken=07401b.f395accd246ae52d \ .
NAME: clusternet-agent
LAST DEPLOYED: Thu Jan 25 11:14:08 2024
NAMESPACE: clusternet-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
  • clusternet-agent
[root@VM-0-13-centos clusternet-agent]# kubectl --context kind-cluster2 get pods -nclusternet-system
NAME                               READY   STATUS    RESTARTS   AGE
clusternet-agent-9d5b74997-c8ptw   1/1     Running   0          6m23s
clusternet-agent-9d5b74997-k2qfx   1/1     Running   0          6m23s
clusternet-agent-9d5b74997-slkpt   1/1     Running   0          6m23s
[root@VM-0-13-centos clusternet-agent]# kubectl --context kind-cluster2 logs clusternet-agent-9d5b74997-slkpt -nclusternet-system
I0125 03:14:10.860952       1 serving.go:348] Generated self-signed cert in-memory
I0125 03:14:10.862457       1 secure_serving.go:210] Serving securely on [::]:10650
I0125 03:14:10.862478       1 agent.go:226] starting agent controller ...
I0125 03:14:10.862524       1 leaderelection.go:248] attempting to acquire leader lease clusternet-system/clusternet-agent...
I0125 03:14:10.862985       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0125 03:14:10.870183       1 agent.go:263] new leader elected: clusternet-agent-9d5b74997-k58w4_cb03fe7c-333e-48c9-b824-793e6b2e6b22
I0125 03:14:26.482229       1 leaderelection.go:258] successfully acquired lease clusternet-system/clusternet-agent
I0125 03:14:26.482420       1 agent.go:326] start registering current cluster as a child cluster...
I0125 03:14:26.482438       1 agent.go:336] retrieving cluster id
I0125 03:14:26.485716       1 agent.go:341] current cluster id is "9add3aec-9faa-42ab-ac76-22228e41f707"
I0125 03:14:26.487410       1 agent.go:443] try to bootstrap cluster registration if needed
I0125 03:14:26.504397       1 agent.go:468] successfully create ClusterRegistrationRequest "clusternet-9add3aec-9faa-42ab-ac76-22228e41f707"
  • clusternet-hub
[root@VM-0-13-centos clusternet-agent]# kubectl --context kind-master-cluster get clsrr
NAME                                              CLUSTER ID                             STATUS   AGE
clusternet-9add3aec-9faa-42ab-ac76-22228e41f707   9add3aec-9faa-42ab-ac76-22228e41f707            21s

@dixudx dixudx added the kind/question Further information is requested label Jan 25, 2024
@googs1025 googs1025 reopened this Jan 25, 2024
@googs1025
Copy link
Author

不好意思～
再请教个问题，这个 "ClusterRegistrationRequest 如何被批准"呢？ 是自动审批？
我看文档 没有说明如何审批。
而且 clsrr STATUS 字段貌似也没显示，想问是否是还有哪个步骤没执行？

[root@VM-0-13-centos .kube]# kubectl get clsrr --context kind-master-cluster
NAME                                              CLUSTER ID                             STATUS   AGE
clusternet-9add3aec-9faa-42ab-ac76-22228e41f707   9add3aec-9faa-42ab-ac76-22228e41f707            21m
[root@VM-0-13-centos .kube]# kubectl get clsrr clusternet-9add3aec-9faa-42ab-ac76-22228e41f707 -o yaml --context kind-master-cluster
apiVersion: clusters.clusternet.io/v1beta1
kind: ClusterRegistrationRequest
metadata:
  creationTimestamp: "2024-01-25T03:14:26Z"
  generation: 1
  labels:
    clusters.clusternet.io/cluster-id: 9add3aec-9faa-42ab-ac76-22228e41f707
    clusters.clusternet.io/cluster-name: clusternet-cluster-nkxd4
    clusters.clusternet.io/registered-by: clusternet-agent
  name: clusternet-9add3aec-9faa-42ab-ac76-22228e41f707
  resourceVersion: "1960889"
  uid: 598053b9-5d93-4df9-870e-a91f408b0aa8
spec:
  clusterId: 9add3aec-9faa-42ab-ac76-22228e41f707
  clusterName: clusternet-cluster-nkxd4
  clusterType: StandardCluster
  syncMode: Dual

@dixudx
Copy link
Member

dixudx commented Jan 25, 2024

Currently ClusterRegistrationRequest will be auto approved by clusternet-controller-manager running in parent cluster.

Please check whether your clusternet-controller-manager is running properly.

@googs1025
Copy link
Author 

[root@VM-0-13-centos clusternet]# kubectl get pods -nclusternet-system
NAME                                            READY   STATUS    RESTARTS   AGE
clusternet-controller-manager-594549889-7h5vk   1/1     Running   0          3d21h
clusternet-controller-manager-594549889-cwv7x   1/1     Running   0          3d21h
clusternet-controller-manager-594549889-nv4jb   1/1     Running   0          3d21h
clusternet-hub-5d9f4bd454-5n8n4                 1/1     Running   0          3d22h
clusternet-hub-5d9f4bd454-tg78b                 1/1     Running   0          3d22h
clusternet-hub-5d9f4bd454-xd75g                 1/1     Running   0          3d22h
clusternet-scheduler-bb5b68d5-km9vm             1/1     Running   1          3d21h
clusternet-scheduler-bb5b68d5-pjw7z             1/1     Running   1          3d21h
clusternet-scheduler-bb5b68d5-qn9rk             1/1     Running   1          3d21h
  • clusternet-controller-manager
    三副本分别
[root@VM-0-13-centos clusternet]# kubectl logs clusternet-controller-manager-594549889-7h5vk -nclusternet-system | tail -20
I0124 04:37:57.508731       1 shared_informer.go:273] Waiting for caches to sync for localization-controller
I0124 04:37:57.508745       1 shared_informer.go:280] Caches are synced for localization-controller
I0124 04:38:00.576304       1 localization.go:139] shutting down localization controller
E0124 04:38:00.583304       1 shared_informer.go:276] unable to sync caches for generic-deployer
I0124 04:38:00.583337       1 generic.go:120] shutting generic deployer
E0124 04:38:00.583363       1 shared_informer.go:276] unable to sync caches for helm-deployer
I0124 04:38:00.583377       1 aggregatestatus.go:126] shutting down aggregatestatus controller
I0124 04:38:00.583399       1 globalization.go:139] shutting down Globalization controller
I0124 04:38:00.583417       1 base.go:129] shutting down base controller
I0124 04:38:00.583436       1 manifest.go:138] shutting down manifest controller
I0124 04:38:00.583454       1 subscription.go:137] shutting down subscription controller
I0124 04:38:00.583468       1 helmchart.go:134] shutting down helmchart controller
I0124 04:38:00.583484       1 feedinventory.go:155] shutting down feedInventory controller
I0124 04:38:00.583601       1 clusterregistrationrequest_controller.go:113] shutting down cluster-registration-requests controller
E0124 04:38:00.583668       1 controllermanager.go:138] leader election got lost
I0124 04:38:00.583711       1 yacht.go:253] shutting down controller cluster-lifecycle
I0124 04:38:00.583731       1 clusterlifecycle_controller.go:129] Shutting down cluster lifecycle controller
I0124 04:38:00.715366       1 helm.go:164] shutting helm deployer
I0124 04:38:00.715515       1 trace.go:219] Trace[2080334431]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169 (24-Jan-2024 04:37:45.822) (total time: 14892ms):
Trace[2080334431]: [14.892781943s] [14.892781943s] END


[root@VM-0-13-centos clusternet]# kubectl logs clusternet-controller-manager-594549889-cwv7x  -nclusternet-system | tail -20
I0121 19:09:15.328134       1 leaderelection.go:283] failed to renew lease clusternet-system/clusternet-controller-manager: timed out waiting for the condition
E0121 19:09:22.470954       1 leaderelection.go:306] Failed to release lock: Operation cannot be fulfilled on leases.coordination.k8s.io "clusternet-controller-manager": the object has been modified; please apply your changes to the latest version and try again
E0121 19:09:22.471051       1 controllermanager.go:138] leader election got lost
I0121 19:09:23.076783       1 localization.go:139] shutting down localization controller
I0121 19:09:23.134786       1 secret.go:117] shutting down Secret controller
I0121 19:09:23.248062       1 description.go:140] shutting down description controller
I0121 19:09:23.501757       1 yacht.go:253] shutting down controller cluster-lifecycle
I0121 19:09:23.552565       1 clusterlifecycle_controller.go:129] Shutting down cluster lifecycle controller
I0121 19:09:23.552694       1 clusterregistrationrequest_controller.go:113] shutting down cluster-registration-requests controller
I0121 19:09:23.552805       1 helm.go:173] shutting helm deployer
I0121 19:09:23.552824       1 description.go:140] shutting down description controller
I0121 19:09:23.560357       1 helmrelease.go:117] shutting down helmRelease controller
I0121 19:09:23.560402       1 manifest.go:138] shutting down manifest controller
I0121 19:09:23.560421       1 generic.go:126] shutting generic deployer
I0121 19:09:23.560439       1 base.go:129] shutting down base controller
I0121 19:09:23.560473       1 helmchart.go:134] shutting down helmchart controller
I0121 19:09:23.841026       1 aggregatestatus.go:126] shutting down aggregatestatus controller
I0121 19:09:23.841079       1 subscription.go:137] shutting down subscription controller
I0121 19:09:23.841109       1 feedinventory.go:155] shutting down feedInventory controller
I0121 19:09:23.841257       1 globalization.go:139] shutting down Globalization controller


[root@VM-0-13-centos clusternet]# kubectl logs clusternet-controller-manager-594549889-nv4jb  -nclusternet-system | tail -20
Trace[1619064350]: [12.599357237s] [12.599357237s] END
I0124 04:37:44.752633       1 trace.go:219] Trace[1633887607]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169 (24-Jan-2024 04:37:31.579) (total time: 13173ms):
Trace[1633887607]: [13.17356457s] [13.17356457s] END
I0124 04:37:44.752655       1 trace.go:219] Trace[571396356]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169 (24-Jan-2024 04:37:30.978) (total time: 13773ms):
Trace[571396356]: [13.773769936s] [13.773769936s] END
I0124 04:37:44.752681       1 trace.go:219] Trace[676846949]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169 (24-Jan-2024 04:37:30.928) (total time: 13824ms):
Trace[676846949]: [13.824117756s] [13.824117756s] END
I0124 04:37:44.752711       1 description.go:140] shutting down description controller
I0124 04:37:44.822004       1 globalization.go:139] shutting down Globalization controller
I0124 04:37:44.822050       1 secret.go:117] shutting down Secret controller
I0124 04:37:44.822072       1 helm.go:173] shutting helm deployer
I0124 04:37:44.822087       1 description.go:140] shutting down description controller
I0124 04:37:44.822103       1 generic.go:126] shutting generic deployer
I0124 04:37:44.822115       1 localization.go:139] shutting down localization controller
I0124 04:37:44.822142       1 aggregatestatus.go:126] shutting down aggregatestatus controller
I0124 04:37:44.822290       1 subscription.go:137] shutting down subscription controller
I0124 04:37:44.822316       1 base.go:129] shutting down base controller
I0124 04:37:44.822335       1 manifest.go:138] shutting down manifest controller
I0124 04:37:44.822437       1 feedinventory.go:155] shutting down feedInventory controller
I0124 04:37:44.858774       1 helmchart.go:134] shutting down helmchart controller

scheduler

[root@VM-0-13-centos clusternet]# kubectl get pods -nclusternet-system
NAME                                            READY   STATUS    RESTARTS   AGE
clusternet-controller-manager-594549889-7h5vk   1/1     Running   0          3d21h
clusternet-controller-manager-594549889-cwv7x   1/1     Running   0          3d21h
clusternet-controller-manager-594549889-nv4jb   1/1     Running   0          3d21h
clusternet-hub-5d9f4bd454-5n8n4                 1/1     Running   0          3d22h
clusternet-hub-5d9f4bd454-tg78b                 1/1     Running   0          3d22h
clusternet-hub-5d9f4bd454-xd75g                 1/1     Running   0          3d22h
clusternet-scheduler-bb5b68d5-km9vm             1/1     Running   1          3d21h
clusternet-scheduler-bb5b68d5-pjw7z             1/1     Running   1          3d21h
clusternet-scheduler-bb5b68d5-qn9rk             1/1     Running   1          3d21h
[root@VM-0-13-centos clusternet]# kubectl logs clusternet-scheduler-bb5b68d5-km9vm -nclusternet-system
I0124 04:39:13.411265       1 serving.go:348] Generated self-signed cert in-memory
I0124 04:39:13.612662       1 secure_serving.go:210] Serving securely on [::]:10659
I0124 04:39:13.620972       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0124 04:39:14.029658       1 leaderelection.go:248] attempting to acquire leader lease clusternet-system/clusternet-scheduler...
I0124 04:39:14.092573       1 scheduler.go:269] new leader elected: clusternet-scheduler-bb5b68d5-pjw7z_0362f003-232a-49c2-a86b-1f0533011be5
E0124 06:04:16.679193       1 leaderelection.go:330] error retrieving resource lock clusternet-system/clusternet-scheduler: Get "https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/clusternet-system/leases/clusternet-scheduler": stream error: stream ID 3155; INTERNAL_ERROR; received from peer
I0124 06:04:21.385885       1 leaderelection.go:258] successfully acquired lease clusternet-system/clusternet-scheduler

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dixudx @googs1025