tags: worker, kubelet
kublet 运行在每个 worker 节点上,接收 kube-apiserver 发送的请求,管理 Pod 容器,执行交互式命令,如 exec、run、logs 等。
kublet 启动时自动向 kube-apiserver 注册节点信息,内置的 cadvisor 统计和监控节点的资源使用情况。
为确保安全,本文档只开启接收 https 请求的安全端口,对请求进行认证和授权,拒绝未授权的访问(如 apiserver、heapster)。
[admin@k8s-admin ~]$ source environment.sh
[admin@k8s-admin ~]$ for node_name in ${NODE_NAMES[@]}
do
echo ">>> ${node_name}"
# 创建 token
BOOTSTRAP_TOKEN=$(kubeadm token create \
--description kubelet-bootstrap-token \
--groups system:bootstrappers:${node_name} \
--kubeconfig ~/.kube/config)
# 设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=cert/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=conf/kubelet-bootstrap-${node_name}.kubeconfig
# 设置客户端认证参数
kubectl config set-credentials kubelet-bootstrap \
--token=${BOOTSTRAP_TOKEN} \
--kubeconfig=conf/kubelet-bootstrap-${node_name}.kubeconfig
# 设置上下文参数
kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=conf/kubelet-bootstrap-${node_name}.kubeconfig
# 设置默认上下文
kubectl config use-context default \
--kubeconfig=conf/kubelet-bootstrap-${node_name}.kubeconfig
done
- 证书中写入 Token 而非证书,证书后续由 kube-controller-manager 创建。
查看 kubeadm 为各节点创建的 token:
[admin@k8s-admin ~]$ kubeadm token list --kubeconfig ~/.kube/config
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
1cieie.adxbcdengb3bk38c 23h 2019-03-07T19:49:59+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:k8s-node02
4mx8qs.pnn0qb2sgjdioap1 23h 2019-03-07T19:50:00+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:k8s-node04
9hecth.nxnh0rfrs8tyg6dz 23h 2019-03-07T19:50:00+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:k8s-node03
acs4mc.hlioi15awk92vp76 23h 2019-03-07T19:49:59+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:k8s-node01
tl2cow.6rp7bju7529vncrn 23h 2019-03-07T19:50:00+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:k8s-node05
- 创建的 token 有效期为 1 天,超期后将不能再被使用,且会被 kube-controller-manager 的 tokencleaner 清理(如果启用该 controller 的话);
- kube-apiserver 接收 kubelet 的 bootstrap token 后,将请求的 user 设置为 system:bootstrap:,group 设置为 system:bootstrappers。
查看各 token 关联的 Secret:
[admin@k8s-admin ~]$ kubectl get secrets -n kube-system|grep bootstrap-token
bootstrap-token-1cieie bootstrap.kubernetes.io/token 7 3m59s
bootstrap-token-4mx8qs bootstrap.kubernetes.io/token 7 3m58s
bootstrap-token-9hecth bootstrap.kubernetes.io/token 7 3m58s
bootstrap-token-acs4mc bootstrap.kubernetes.io/token 7 3m59s
bootstrap-token-tl2cow bootstrap.kubernetes.io/token 7 3m58s
在 Kubernetes 1.8 版本上,除了可以通过命令行参数外,还可以通过保存在硬盘的配置文件设置 Kubelet 的配置子集。 将来,大部分现存的命令行参数都将被废弃,取而代之以配置文件的方式提供参数,以简化节点部署过程。该结构体在 [这里](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/config/types.go) 可以找到。配置文件必须是这个结构体中参数的 JSON 或 YAML 表现形式。
从 Kubernetes 1.10 开始,kubelet 部分参数 需在配置文件中配置,kubelet --help
会提示:
`DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag`
创建 kubelet 参数配置模板文件:
[admin@k8s-admin ~]$ cat >conf/kubelet.config.yaml.j2 <<"EOF"
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
anonymous:
enabled: false
webhook:
enabled: true
cacheTTL: "2m0s"
x509:
clientCAFile: "/etc/kubernetes/cert/ca.pem"
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: "5m0s"
cacheUnauthorizedTTL: "30s"
clusterDomain: "{{ DNS_DOMAIN }}"
clusterDNS:
- "{{ DNS_SERVER_IP }}"
maxPods: 220
serializeImagePulls: false
hairpinMode: promiscuous-bridge
cgroupDriver: cgroupfs
runtimeRequestTimeout: "15m"
rotateCertificates: true
serverTLSBootstrap: true
readOnlyPort: 0
port: 10250
address: "{{ ansible_eth0.ipv4.address }}"
KubeAPIQPS: 1000
KubeAPIBurst: 2000
RegistryPullQPS: 0
EventRecordQPS: 0
EOF
- address:API 监听地址,不能为 127.0.0.1,否则 kube-apiserver、heapster 等不能调用 kubelet 的 API;
- readOnlyPort=0:关闭只读端口(默认 10255),等效为未指定;
- authentication.anonymous.enabled:设置为 false,不允许匿名�访问 10250 端口;
- authentication.x509.clientCAFile:指定签名客户端证书的 CA 证书,开启 HTTP 证书认证;
- authentication.webhook.enabled=true:开启 HTTPs bearer token 认证;
- 对于未通过 x509 证书和 webhook 认证的请求(kube-apiserver 或其他客户端),将被拒绝,提示 Unauthorized;
- authroization.mode=Webhook:kubelet 使用 SubjectAccessReview API 查询 kube-apiserver 某 user、group 是否具有操作资源的权限(RBAC);
- featureGates.RotateKubeletClientCertificate、featureGates.RotateKubeletServerCertificate:自动 rotate 证书,证书的有效期取决于 kube-controller-manager 的 --experimental-cluster-signing-duration 参数;
- 需要 root 账户运行。
[admin@k8s-admin ~]$ cat >conf/kubelet.service.j2 <<"EOF"
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory={{ K8S_DIR }}/kubelet
EnvironmentFile=/etc/kubernetes/kubelet.conf
ExecStart=/usr/local/bin/kubelet $KUBELET_ARGS
Restart=always
RestartSec=5
StartLimitInterval=0
[Install]
WantedBy=multi-user.target
EOF
[admin@k8s-admin ~]$ cat >conf/kubelet.conf.j2 <<EOF
KUBELET_ARGS=" \
--bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \
--root-dir={{ K8S_DIR }}/kubelet \
--cert-dir=/etc/kubernetes/cert \
--kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
--config=/etc/kubernetes/kubelet.config.yaml \
--hostname-override={{ ansible_host }} \
--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest \
--allow-privileged=true \
--image-pull-progress-deadline=30m \
--logtostderr=true \
--v=2"
EOF
- 如果设置了
--hostname-override
选项,则kube-proxy
也需要设置该选项,否则会出现找不到 Node 的情况; --bootstrap-kubeconfig
:指向 bootstrap kubeconfig 文件,kubelet 使用该文件中的用户名和 token 向 kube-apiserver 发送 TLS Bootstrapping 请求;- K8S approve kubelet 的 csr 请求后,在
--cert-dir
目录创建证书和私钥文件,然后写入--kubeconfig
文件; --pod-infra-container-image
不使用 redhat 的pod-infrastructure:latest
镜像,它不能回收容器的僵尸。
kublet 启动时查找配置的 --kubeletconfig 文件是否存在,如果不存在则使用 --bootstrap-kubeconfig 向 kube-apiserver 发送证书签名请求 (CSR)。
kube-apiserver 收到 CSR 请求后,对其中的 Token 进行认证(事先使用 kubeadm 创建的 token),认证通过后将请求的 user 设置为 system:bootstrap:,group 设置为 system:bootstrappers,这一过程称为 Bootstrap Token Auth。
默认情况下,这个 user 和 group 没有创建 CSR 的权限,kubelet 启动失败,错误日志如下:
[root@k8s-node01 ~]# journalctl -u kubelet.service |grep -A 2 'certificatesigningrequests'
3月 06 20:57:02 k8s-node01 kubelet[20699]: F0306 20:57:02.843256 20699 server.go:261] failed to run Kubelet: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:bootstrap:acs4mc" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope
3月 06 20:57:02 k8s-node01 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
3月 06 20:57:02 k8s-node01 systemd[1]: Unit kubelet.service entered failed state.
解决办法是:创建一个 clusterrolebinding,将 group system:bootstrappers
和 clusterrole system:node-bootstrapper
绑定:
[admin@k8s-admin ~]$ kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
[admin@k8s-admin ~]$ cat >conf/kubelet.service.yaml <<EOF
- hosts: k8s-workers
remote_user: root
tasks:
- name: copy kubelet.config.yaml
template:
src: kubelet.config.yaml.j2
dest: /etc/kubernetes/kubelet.config.yaml
- name: copy kubelet-bootstrap.kubeconfig
copy:
src: kubelet-bootstrap-{{ ansible_host }}.kubeconfig
dest: /etc/kubernetes/kubelet-bootstrap.kubeconfig
- name: copy kubelet.service
template:
src: kubelet.service.j2
dest: /usr/lib/systemd/system/kubelet.service
- name: copy kubelet.conf
template:
src: kubelet.conf.j2
dest: /etc/kubernetes/kubelet.conf
- name: mkdir the datadir for kubelet
file:
path: "{{ K8S_DIR }}/kubelet"
state: directory
- name: enable and start kubelet.service
systemd:
name: kubelet
state: restarted
enabled: yes
daemon_reload: yes
EOF
[admin@k8s-admin ~]$ ansible-playbook conf/kubelet.service.yaml
[admin@k8s-admin ~]$ ansible k8s-workers -m shell -a "systemctl status kubelet.service|grep -e Loaded -e Active"
- 必须创建工作目录;
- 关闭 swap 分区,否则 kubelet 会启动失败;
- 确保状态为“active (running)”并且“enabled”。
[root@k8s-node01 ~]# journalctl -u kubelet -n 12
-- Logs begin at 五 2019-03-01 17:13:41 CST, end at 三 2019-03-06 22:01:01 CST. --
3月 06 21:59:06 k8s-node01 kubelet[4181]: I0306 21:59:06.638559 4181 feature_gate.go:206] feature gates: &{map[]}
3月 06 21:59:06 k8s-node01 kubelet[4181]: I0306 21:59:06.638647 4181 feature_gate.go:206] feature gates: &{map[]}
3月 06 21:59:06 k8s-node01 kubelet[4181]: I0306 21:59:06.651477 4181 mount_linux.go:180] Detected OS with systemd
3月 06 21:59:06 k8s-node01 kubelet[4181]: I0306 21:59:06.651582 4181 server.go:407] Version: v1.13.4
3月 06 21:59:06 k8s-node01 kubelet[4181]: I0306 21:59:06.651638 4181 feature_gate.go:206] feature gates: &{map[]}
3月 06 21:59:06 k8s-node01 kubelet[4181]: I0306 21:59:06.651698 4181 feature_gate.go:206] feature gates: &{map[]}
3月 06 21:59:06 k8s-node01 kubelet[4181]: I0306 21:59:06.651803 4181 plugins.go:103] No cloud provider specified.
3月 06 21:59:06 k8s-node01 kubelet[4181]: I0306 21:59:06.651821 4181 server.go:523] No cloud provider specified: "" from the config file: ""
3月 06 21:59:06 k8s-node01 kubelet[4181]: I0306 21:59:06.651855 4181 bootstrap.go:65] Using bootstrap kubeconfig to generate TLS client cert, key
3月 06 21:59:06 k8s-node01 kubelet[4181]: I0306 21:59:06.660441 4181 bootstrap.go:96] No valid private key and/or certificate found, reusing exist
3月 06 21:59:06 k8s-node01 kubelet[4181]: I0306 21:59:06.722171 4181 csr.go:68] csr for this node already exists, reusing
3月 06 21:59:06 k8s-node01 kubelet[4181]: I0306 21:59:06.727206 4181 csr.go:76] csr for this node is still valid
kubelet 启动后使用 --bootstrap-kubeconfig 向 kube-apiserver 发送 CSR 请求,当这个 CSR 被 approve 后,kube-controller-manager 为 kubelet 创建 TLS 客户端证书、私钥和 --kubeletconfig 文件。
注意:kube-controller-manager 需要配置 --cluster-signing-cert-file
和 --cluster-signing-key-file
参数,才会为 TLS Bootstrap 创建证书和私钥。
[admin@k8s-admin ~]$ kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-8sGRRYc4C4A2NIyFdQ1f_7KY3MmDc7oYkBN8EW1D_qA 14m system:bootstrap:9hecth Pending
node-csr-YWLcuZW6nRTCydJe_wi7_OVE3tOlpIMH80p0D-ztBgw 14m system:bootstrap:1cieie Pending
node-csr-v_aD9gjQZWLwmDOCoaBLwWh0atqfnESaeRKQ1G09miw 14m system:bootstrap:acs4mc Pending
node-csr-bKCTmXdewYFkEoBCPtjo1BF22fU87ZUSIS4E7v2sEv0 14m system:bootstrap:4mx8qs Pending
node-csr-qRePTjnLbxlBj-P9bVr_LhInhSELUiNg-1MBCt1CM6U 14m system:bootstrap:tl2cow Pending
[admin@k8s-admin ~]$ kubectl get nodes
No resources found.
- 三个 work 节点的 csr 均处于 pending 状态。
创建三个 ClusterRoleBinding,分别用于自动 approve client、renew client、renew server 证书:
[admin@k8s-admin ~]$ cat > conf/csr-crb.yaml <<EOF
# Approve all CSRs for the group "system:bootstrappers"
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: auto-approve-csrs-for-group
subjects:
- kind: Group
name: system:bootstrappers
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
apiGroup: rbac.authorization.k8s.io
---
# To let a node of the group "system:nodes" renew its own credentials
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: node-client-cert-renewal
subjects:
- kind: Group
name: system:nodes
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
apiGroup: rbac.authorization.k8s.io
---
# A ClusterRole which instructs the CSR approver to approve a node requesting a
# serving cert matching its client cert.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: approve-node-server-renewal-csr
rules:
- apiGroups: ["certificates.k8s.io"]
resources: ["certificatesigningrequests/selfnodeserver"]
verbs: ["create"]
---
# To let a node of the group "system:nodes" renew its own server credentials
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: node-server-cert-renewal
subjects:
- kind: Group
name: system:nodes
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: approve-node-server-renewal-csr
apiGroup: rbac.authorization.k8s.io
EOF
auto-approve-csrs-for-group
:自动 approve node 的第一次 CSR; 注意第一次 CSR 时,请求的 Group 为system:bootstrappers
;node-client-cert-renewal
:自动 approve node 后续过期的 client 证书,自动生成的证书 Group 为system:nodes
;node-server-cert-renewal
:自动 approve node 后续过期的 server 证书,自动生成的证书 Group 为system:nodes
。
使配置生效:
[admin@k8s-admin ~]$ kubectl apply -f conf/csr-crb.yaml
clusterrolebinding.rbac.authorization.k8s.io/auto-approve-csrs-for-group created
clusterrolebinding.rbac.authorization.k8s.io/node-client-cert-renewal created
clusterrole.rbac.authorization.k8s.io/approve-node-server-renewal-csr created
clusterrolebinding.rbac.authorization.k8s.io/node-server-cert-renewal created
等待一段时间(1-10 分钟),三个节点的 CSR 都被自动 approved:
[admin@k8s-admin ~]$ kubectl get csr
NAME AGE REQUESTOR CONDITION
csr-lk6kv 5m33s system:node:k8s-node01 Pending
csr-nrhtt 5m33s system:node:k8s-node03 Pending
csr-sfn5g 5m33s system:node:k8s-node02 Pending
csr-67bk4 5m33s system:node:k8s-node05 Pending
csr-789cc 5m33s system:node:k8s-node04 Pending
node-csr-8sGRRYc4C4A2NIyFdQ1f_7KY3MmDc7oYkBN8EW1D_qA 32m system:bootstrap:9hecth Approved,Issued
node-csr-YWLcuZW6nRTCydJe_wi7_OVE3tOlpIMH80p0D-ztBgw 32m system:bootstrap:1cieie Approved,Issued
node-csr-v_aD9gjQZWLwmDOCoaBLwWh0atqfnESaeRKQ1G09miw 32m system:bootstrap:acs4mc Approved,Issued
node-csr-bKCTmXdewYFkEoBCPtjo1BF22fU87ZUSIS4E7v2sEv0 32m system:bootstrap:4mx8qs Approved,Issued
node-csr-qRePTjnLbxlBj-P9bVr_LhInhSELUiNg-1MBCt1CM6U 32m system:bootstrap:tl2cow Approved,Issued
所有节点均 ready:
[admin@k8s-admin ~]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-node01 Ready <none> 5m39s v1.13.4
k8s-node02 Ready <none> 5m39s v1.13.4
k8s-node03 Ready <none> 5m39s v1.13.4
k8s-node04 Ready <none> 5m39s v1.13.4
k8s-node05 Ready <none> 5m39s v1.13.4
kube-controller-manager 为各 node 生成了 kubeconfig 文件和公私钥:
[root@k8s-node01 ~]# ll /etc/kubernetes/kubelet.kubeconfig
-rw------- 1 root root 2299 3月 6 22:16 /etc/kubernetes/kubelet.kubeconfig
[root@k8s-node01 ~]# ll /etc/kubernetes/cert|grep kubelet
-rw------- 1 root root 1269 3月 6 22:16 kubelet-client-2019-03-06-22-16-19.pem
lrwxrwxrwx 1 root root 59 3月 6 22:16 kubelet-client-current.pem -> /etc/kubernetes/cert/kubelet-client-2019-03-06-22-16-19.pem
- 没有自动生成 kubelet server 证书。
基于安全性考虑,CSR approving controllers 默认不会自动 approve kubelet server 证书签名请求,需要手动 approve。
添加新的 worker 节点,系统会自动 approve client cert csr,只需要手动 approve server cert csr 即可。
[admin@k8s-admin ~]$ kubectl certificate approve csr-lk6kv
certificatesigningrequest.certificates.k8s.io/csr-lk6kv approved
[admin@k8s-admin ~]$ kubectl certificate approve csr-nrhtt
certificatesigningrequest.certificates.k8s.io/csr-nrhtt approved
...
# 一次性 approve 所有 Pending 状态的 server cert csr
[admin@k8s-admin ~]$ for i in $(kubectl get csr|grep "Pending"|awk '{print $1}'); do kubectl certificate approve $i; done
[admin@k8s-admin ~]$ kubectl get csr
NAME AGE REQUESTOR CONDITION
csr-lk6kv 8m41s system:node:k8s-node01 Approved,Issued
csr-nrhtt 8m41s system:node:k8s-node03 Approved,Issued
csr-sfn5g 8m41s system:node:k8s-node02 Approved,Issued
csr-67bk4 8m41s system:node:k8s-node05 Approved,Issued
csr-789cc 8m41s system:node:k8s-node04 Approved,Issued
node-csr-8sGRRYc4C4A2NIyFdQ1f_7KY3MmDc7oYkBN8EW1D_qA 36m system:bootstrap:9hecth Approved,Issued
node-csr-YWLcuZW6nRTCydJe_wi7_OVE3tOlpIMH80p0D-ztBgw 36m system:bootstrap:1cieie Approved,Issued
node-csr-v_aD9gjQZWLwmDOCoaBLwWh0atqfnESaeRKQ1G09miw 36m system:bootstrap:acs4mc Approved,Issued
node-csr-bKCTmXdewYFkEoBCPtjo1BF22fU87ZUSIS4E7v2sEv0 36m system:bootstrap:4mx8qs Approved,Issued
node-csr-qRePTjnLbxlBj-P9bVr_LhInhSELUiNg-1MBCt1CM6U 36m system:bootstrap:tl2cow Approved,Issued
[root@k8s-node01 ~]# ll /etc/kubernetes/cert|grep kubelet
-rw------- 1 root root 1269 3月 6 22:16 kubelet-client-2019-03-06-22-16-19.pem
lrwxrwxrwx 1 root root 59 3月 6 22:16 kubelet-client-current.pem -> /etc/kubernetes/cert/kubelet-client-2019-03-06-22-16-19.pem
-rw------- 1 root root 1313 3月 6 22:24 kubelet-server-2019-03-06-22-24-51.pem
lrwxrwxrwx 1 root root 59 3月 6 22:24 kubelet-server-current.pem -> /etc/kubernetes/cert/kubelet-server-2019-03-06-22-24-51.pem
kublet 启动后监听多个端口,用于接收 kube-apiserver 或其它组件发送的请求:
[root@k8s-node01 ~]# ss -lnptu|grep kubelet
tcp LISTEN 0 128 127.0.0.1:10248 *:* users:(("kubelet",pid=5686,fd=20))
tcp LISTEN 0 128 192.168.99.101:10250 *:* users:(("kubelet",pid=5686,fd=19))
tcp LISTEN 0 128 127.0.0.1:37712 *:* users:(("kubelet",pid=5686,fd=8))
- 10248: healthz http 服务;
- 10250: https API 服务;注意:未开启只读端口 10255;
例如执行 kubectl exec -it nginx-ds-5rmws -- sh
命令时,kube-apiserver 会向 kubelet 发送如下请求:
POST /exec/default/nginx-ds-5rmws/my-nginx?command=sh&input=1&output=1&tty=1
kubelet 接收 10250 端口的 https 请求:
- /pods、/runningpods
- /metrics、/metrics/cadvisor、/metrics/probes
- /spec
- /stats、/stats/container
- /logs
- /run/、"/exec/", "/attach/", "/portForward/", "/containerLogs/" 等管理;
详情参考:https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/server/server.go#L434:3
由于关闭了匿名认证,同时开启了 webhook 授权,所有访问 10250 端口 https API 的请求都需要被认证和授权。
预定义的 ClusterRole system:kubelet-api-admin
授予访问 kubelet 所有 API 的权限(kube-apiserver 使用的 kubernetes 证书 User 授予了该权限):
[admin@k8s-admin ~]$ kubectl describe clusterrole system:kubelet-api-admin
Name: system:kubelet-api-admin
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
nodes/log [] [] [*]
nodes/metrics [] [] [*]
nodes/proxy [] [] [*]
nodes/spec [] [] [*]
nodes/stats [] [] [*]
nodes [] [] [get list watch proxy]
kublet 配置了如下认证参数:
authentication.anonymous.enabled
:设置为 false,不允许匿名�访问 10250 端口;authentication.x509.clientCAFile
:指定签名客户端证书的 CA 证书,开启 HTTPs 证书认证;authentication.webhook.enabled
:设置为true,开启 HTTPs bearer token 认证;
同时配置了如下授权参数:
authroization.mode=Webhook
:开启 RBAC 授权;
kubelet 收到请求后,使用 clientCAFile 对证书签名进行认证,或者查询 bearer token 是否有效。如果两者都没通过,则拒绝请求,提示 Unauthorized:
[admin@k8s-admin ~]$ curl -s --cacert cert/ca.pem https://192.168.99.101:10250/metrics
Unauthorized
[admin@k8s-admin ~]$ curl -s --cacert cert/ca.pem -H "Authorization: Bearer 123456" https://192.168.99.101:10250/metrics
Unauthorized
通过认证后,kubelet 使用 SubjectAccessReview API 向 kube-apiserver 发送请求,查询证书或 token 对应的 user、group 是否有操作资源的权限(RBAC)。
# 权限不足的证书;
[admin@k8s-admin ~]$ curl -s --cacert cert/ca.pem --cert cert/kube-controller-manager.pem --key cert/kube-controller-manager-key.pem https://192.168.99.101:10250/metrics
Forbidden (user=system:kube-controller-manager, verb=get, resource=nodes, subresource=metrics)
# 使用部署 kubectl 命令行工具时创建的、具有最高权限的 admin 证书;
[admin@k8s-admin ~]$ curl -s --cacert cert/ca.pem --cert cert/kubectl.pem --key cert/kubectl-key.pem https://192.168.99.101:10250/metrics|head
# HELP apiserver_audit_event_total Counter of audit events generated and sent to the audit backend.
# TYPE apiserver_audit_event_total counter
apiserver_audit_event_total 0
# HELP apiserver_audit_requests_rejected_total Counter of apiserver requests rejected due to an error in audit logging backend.
# TYPE apiserver_audit_requests_rejected_total counter
apiserver_audit_requests_rejected_total 0
# HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request.
# TYPE apiserver_client_certificate_expiration_seconds histogram
apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="21600"} 0
--cacert
、--cert
、--key
的参数值必须是文件路径,如cert/kubectl.pem
或./kubelet.pem
,./kubelet.pem
不能省略./
,否则返回401 Unauthorized
;
创建一个 ServiceAccount,将它和 ClusterRole system:kubelet-api-admin 绑定,从而具有调用 kubelet API 的权限:
[admin@k8s-admin ~]$ kubectl create serviceaccount kubelet-api-test
serviceaccount/kubelet-api-test created
[admin@k8s-admin ~]$ kubectl create clusterrolebinding kubelet-api-test --clusterrole=system:kubelet-api-admin --serviceaccount=default:kubelet-api-test
clusterrolebinding.rbac.authorization.k8s.io/kubelet-api-test created
[admin@k8s-admin ~]$ SECRET=$(kubectl get secrets | grep kubelet-api-test | awk '{print $1}')
[admin@k8s-admin ~]$ TOKEN=$(kubectl describe secret ${SECRET} | grep -E '^token' | awk '{print $2}')
[admin@k8s-admin ~]$ curl -s --cacert cert/ca.pem -H "Authorization: Bearer ${TOKEN}" https://192.168.99.101:10250/metrics|head
# HELP apiserver_audit_event_total Counter of audit events generated and sent to the audit backend.
# TYPE apiserver_audit_event_total counter
apiserver_audit_event_total 0
# HELP apiserver_audit_requests_rejected_total Counter of apiserver requests rejected due to an error in audit logging backend.
# TYPE apiserver_audit_requests_rejected_total counter
apiserver_audit_requests_rejected_total 0
# HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request.
# TYPE apiserver_client_certificate_expiration_seconds histogram
apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="21600"} 0
cadvisor 统计所在节点各容器的资源(CPU、内存、磁盘、网卡)使用情况,分别在自己的 http web 页面(4194 端口)和 10250 以 promehteus metrics 的形式输出。
浏览器访问 https://192.168.99.101:10250/metrics 和 https://192.168.99.101:10250/metrics/cadvisor 分别返回 kublet 和 cadvisor 的 metrics。
注意:
- kublet.config.json 设置
authentication.anonymous.enabled
为 false,不允许匿名证书访问 10250 的 https 服务; - 参考A.浏览器访问kube-apiserver安全端口.md,创建和导入相关证书,然后访问上面的 10250 端口。
从 kube-apiserver 获取各 node 的配置:
$ # 使用部署 kubectl 命令行工具时创建的、具有最高权限的 admin 证书;
[admin@k8s-admin ~]$ source environment.sh
[admin@k8s-admin ~]$ sudo yum install epel-release
[admin@k8s-admin ~]$ sudo yum install jq
[admin@k8s-admin ~]$ curl -s \
--cacert cert/ca.pem \
--cert cert/kubectl.pem \
--key cert/kubectl-key.pem \
${KUBE_APISERVER}/api/v1/nodes/k8s-node01/proxy/configz \
|jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"'
{
"syncFrequency": "1m0s",
"fileCheckFrequency": "20s",
"httpCheckFrequency": "20s",
"address": "192.168.99.101",
"port": 10250,
"rotateCertificates": true,
"serverTLSBootstrap": true,
"authentication": {
"x509": {
"clientCAFile": "/etc/kubernetes/cert/ca.pem"
},
"webhook": {
"enabled": true,
"cacheTTL": "2m0s"
},
"anonymous": {
"enabled": false
}
},
"authorization": {
"mode": "Webhook",
"webhook": {
"cacheAuthorizedTTL": "5m0s",
"cacheUnauthorizedTTL": "30s"
}
},
"registryPullQPS": 0,
"registryBurst": 10,
"eventRecordQPS": 0,
"eventBurst": 10,
"enableDebuggingHandlers": true,
"healthzPort": 10248,
"healthzBindAddress": "127.0.0.1",
"oomScoreAdj": -999,
"clusterDomain": "cluster.local",
"clusterDNS": [
"10.16.0.2"
],
"streamingConnectionIdleTimeout": "4h0m0s",
"nodeStatusUpdateFrequency": "10s",
"nodeStatusReportFrequency": "1m0s",
"nodeLeaseDurationSeconds": 40,
"imageMinimumGCAge": "2m0s",
"imageGCHighThresholdPercent": 85,
"imageGCLowThresholdPercent": 80,
"volumeStatsAggPeriod": "1m0s",
"cgroupsPerQOS": true,
"cgroupDriver": "cgroupfs",
"cpuManagerPolicy": "none",
"cpuManagerReconcilePeriod": "10s",
"runtimeRequestTimeout": "15m0s",
"hairpinMode": "promiscuous-bridge",
"maxPods": 220,
"podPidsLimit": -1,
"resolvConf": "/etc/resolv.conf",
"cpuCFSQuota": true,
"cpuCFSQuotaPeriod": "100ms",
"maxOpenFiles": 1000000,
"contentType": "application/vnd.kubernetes.protobuf",
"kubeAPIQPS": 1000,
"kubeAPIBurst": 2000,
"serializeImagePulls": false,
"evictionHard": {
"imagefs.available": "15%",
"memory.available": "100Mi",
"nodefs.available": "10%",
"nodefs.inodesFree": "5%"
},
"evictionPressureTransitionPeriod": "5m0s",
"enableControllerAttachDetach": true,
"makeIPTablesUtilChains": true,
"iptablesMasqueradeBit": 14,
"iptablesDropBit": 15,
"failSwapOn": true,
"containerLogMaxSize": "10Mi",
"containerLogMaxFiles": 5,
"configMapAndSecretChangeDetectionStrategy": "Watch",
"enforceNodeAllocatable": [
"pods"
],
"kind": "KubeletConfiguration",
"apiVersion": "kubelet.config.k8s.io/v1beta1"
}
或者参考代码中的注释:https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/config/types.go