詳解kubelet?創(chuàng)建pod流程代碼圖解及日志說(shuō)明-魔扣目錄

正文

本文將從如下方面介紹kubelet創(chuàng)建pod的過(guò)程

kubernetes調(diào)度pod簡(jiǎn)介
kubelet 創(chuàng)建pod代碼圖解說(shuō)明（本文重點(diǎn)）
kubelet 調(diào)用cri創(chuàng)建容器說(shuō)明（本文重點(diǎn)）
通過(guò)日志來(lái)分析kubelet真實(shí)創(chuàng)建日志的全過(guò)程（本文重點(diǎn)）

kubernetes調(diào)度pod簡(jiǎn)介

kubernetes（后面簡(jiǎn)稱(chēng)k8s）主要有三種管理（創(chuàng)建）pod的方式：

一種是直接申明創(chuàng)建一個(gè)裸pod
另一種是通過(guò)controller 來(lái)申明創(chuàng)建pod：比如，deployments、replicationcontrollers、daemonsets或者replicasets
還有一種是static（靜態(tài)） pod 這種用的比較少，一般是把pod的申明文件放在對(duì)應(yīng)的kubernetes/manifest 目錄下，通常用來(lái)創(chuàng)建apiserver，controller-manager，scheduler這類(lèi)k8s管理組件的pod。

k8s推薦使用controller來(lái)管理pod，這符合k8s管理pod的習(xí)慣，便于使用k8s相關(guān)功能，比如彈性擴(kuò)縮容，pod故障自動(dòng)拉起等。我們也以controller管理的pod為例，簡(jiǎn)單梳理下k8s創(chuàng)建及調(diào)度pod流程，如下圖

詳解kubelet?創(chuàng)建pod流程代碼圖解及日志說(shuō)明

客戶端請(qǐng)求apiserver創(chuàng)建replicasets，apiserver通過(guò)認(rèn)證、鑒權(quán)、準(zhǔn)入后，會(huì)把請(qǐng)求相關(guān)信息持久化至etcd
Controller-manager 管理的replicaset controller 通過(guò)list-watch機(jī)制，watch到有replicasets創(chuàng)建請(qǐng)求，通過(guò)label selector發(fā)現(xiàn)集群中與這個(gè)replicasets 關(guān)聯(lián)的pod當(dāng)前狀態(tài)與期望狀態(tài)不一致，則會(huì)進(jìn)行調(diào)協(xié)（reconcile）向apiserver發(fā)起創(chuàng)建pod請(qǐng)求
Scheduler 通過(guò)list-watch機(jī)制來(lái)發(fā)現(xiàn)未綁定的pod，并通過(guò)預(yù)選及優(yōu)選策略算法，來(lái)計(jì)算出pod最終可調(diào)度的node節(jié)點(diǎn)，并通過(guò)apiserver將數(shù)據(jù)更新至etcd
Kubelet 通過(guò)list-watch發(fā)現(xiàn)有新的pod bound到本node上，則會(huì)發(fā)起創(chuàng)建pod相關(guān)流程

kubelet 創(chuàng)建pod代碼及圖解說(shuō)明

kubelet 簡(jiǎn)介

Kubelet 有點(diǎn)和controller類(lèi)似，也是通過(guò)list-watch相關(guān)信息，或者輪詢(xún)本地pod相關(guān)信息及事件，來(lái)觸發(fā)相關(guān)動(dòng)作，使pod處于”期望狀態(tài)”，并且向apiserver上報(bào)本node（宿主機(jī)）及node里所有pod的狀態(tài)信息。

kubelet 不同于其他controller的一點(diǎn)就是，它是部署在每個(gè)node節(jié)點(diǎn)上的agent，它需要與apiserver 打交道同樣也需要與cri(contain-runtime-interface)打交道來(lái)管理node上的容器。所以它需要通過(guò)apiserver來(lái)watch到對(duì)本地pod變更的事件，也需要不斷輪詢(xún)pod狀態(tài)信息，將狀態(tài)及時(shí)同步給apiserver，所以Kubelet整體工作邏輯是loop監(jiān)聽(tīng)各類(lèi)生產(chǎn)者產(chǎn)生的消息或者定時(shí)觸發(fā)消息，來(lái)調(diào)用相應(yīng)的消費(fèi)者（不同的子模塊）完成不同的操作，比如watch 到apiserver的請(qǐng)求，PLEG(pod lifecycle event generator)產(chǎn)生的事件，定時(shí)觸發(fā)的任務(wù)等

kubelet創(chuàng)建及啟動(dòng)pod流程

kubelet 創(chuàng)建pod代碼調(diào)用圖解

詳解kubelet?創(chuàng)建pod流程代碼圖解及日志說(shuō)明

kubelet 創(chuàng)建pod詳細(xì)說(shuō)明

1.kubelet 會(huì)listwatch所有namespace下、綁定到本node上的pod，并將信息傳入updatechannel。kubelet 的SyncLoop（是kubele的主循環(huán)函數(shù)，來(lái)控制例行循環(huán)往復(fù)的事情：同步接收、更新、處理pod變更相關(guān)信息）下的syncLoopIteration方法會(huì)監(jiān)聽(tīng)多方消息，會(huì)監(jiān)聽(tīng)各個(gè)消息源，來(lái)觸發(fā)相應(yīng)的操作，這個(gè)方法會(huì)接收前面listwatch到的updatechannel信息，交由對(duì)應(yīng)的handler:如pod創(chuàng)建：調(diào)用HandlePodAdditions處理，pod刪除調(diào)用HandlePodUpdates處理（DELETE is treated as a UPDATE because of graceful deletion.）
2.HandlePodAdditions 會(huì)對(duì)pods 進(jìn)行排序，判斷，準(zhǔn)入校驗(yàn)，之后調(diào)用dispatchWork 把對(duì)某個(gè)pod的操作分配給 podWorkers 做異步操作（pod創(chuàng)建、刪除、更新）處理
3.異步操作會(huì)調(diào)用kubelet syncPod（syncPod is the transaction script for the sync of a single pod.）方法,syncPod會(huì)做一些pod創(chuàng)建前的準(zhǔn)備工作

a.如果pod updateType 為podkill，立即執(zhí)行并返回(走pod刪除流程)

b.pod準(zhǔn)入檢查檢查pod是否能運(yùn)行在本節(jié)點(diǎn)

c.更新?tīng)顟B(tài)給 status manager ，status manager將pod狀態(tài)上報(bào)給apiserver

d.檢查網(wǎng)絡(luò)插件是否就緒

e.創(chuàng)建并更新pod cgroups配置

f.為pod創(chuàng)建對(duì)應(yīng)的目錄：pod目錄，volume目錄

g.等待pod sepc中的volme都被attach/mount

h.從apiserver中獲取pull secrets

i.調(diào)用 containerRuntime 的 SyncPod 方法開(kāi)始創(chuàng)建容器
復(fù)制代碼

4.containerRuntime 的 SyncPod 會(huì)做如下主要工作

a.創(chuàng)建sandbox

b.Create ephemeral containers

c.Create init containers

d.Create normal containers
復(fù)制代碼

其中創(chuàng)建sandbox是關(guān)鍵，sandbox可以理解為pod的運(yùn)行環(huán)境，是業(yè)務(wù)pod的父容器，在k8s里就是pause 容器,所有容器創(chuàng)建前都需要?jiǎng)?chuàng)建pause容器。首先會(huì)生成podsandbox相關(guān)配置：如dnsconfig，podhostname，設(shè)置sysctl，cgroups以及namespace

然后會(huì)調(diào)用CRI（container-runtime-interface）來(lái)調(diào)用底層container runtime來(lái)真實(shí)操作容器,之后還會(huì)調(diào)用CNI插件來(lái)為容器設(shè)置網(wǎng)絡(luò)。

5.我們?cè)賮?lái)看下創(chuàng)建sandbox：RunpodSandbox的步驟 (ds *dockerService) RunPodSandbox 是在是一個(gè)cri的是實(shí)現(xiàn)，所以在dockershim下dockershim是內(nèi)置在kubelet里的cri實(shí)現(xiàn)，用來(lái)銜接kubelet與docker,dockershim翻譯為docker"墊片"，很形象）。kubelet通過(guò)grp call調(diào)用的dockershim來(lái)實(shí)現(xiàn)容器的創(chuàng)建管理。

a.調(diào)用docker API Pull the image for the sandbox.
(kubelet 的sandbox鏡像：defaultSandboxImage = "k8s.gcr.io/pause:3.2")

b. 調(diào)用docker Create the sandbox container.

c.Create Sandbox Checkpoint.

d.調(diào)用docker Start the sandbox container.

e.Rewrite resolv.conf file generated by docker.

f. Setup networking for the sandbox. 調(diào)用cni插件為容器設(shè)置網(wǎng)絡(luò)

kubelet 調(diào)用cri說(shuō)明

我們目前container-runtime為docker，docker并不支持CRI，所以要想調(diào)用docker 操作容器，k8s內(nèi)置了dockershim來(lái)調(diào)用docker，dockershim可以理解為一個(gè)滿足CRI標(biāo)準(zhǔn)的容器運(yùn)行時(shí)，kubelet通過(guò)grpc call 來(lái)調(diào)用dockershim，dockershim收到kubelet的請(qǐng)求后，將其轉(zhuǎn)化為REST API請(qǐng)求，再發(fā)送給docker daemon，docker daemon 在通過(guò)組裝請(qǐng)求，調(diào)用docker API來(lái)完成container的最終創(chuàng)建、啟動(dòng)等相關(guān)操作。

這塊有兩個(gè)地方需要說(shuō)明下：

1是為啥會(huì)有dockershim？這里有個(gè)小故事，首先k8s再具有一定市場(chǎng)規(guī)模后，想與docker 解耦，不想強(qiáng)依賴(lài)docker,同時(shí)為了支持多種container-runtime，故制定了CRI，只有滿足CRI，kubelet便可以直接完成調(diào)用來(lái)管理container,然而docker一開(kāi)始并不支持CRI，故k8s想了個(gè)這種的方式，開(kāi)發(fā)了一個(gè)dockershim（docker "墊片"）來(lái)轉(zhuǎn)發(fā)請(qǐng)求,這樣k8s也完成了對(duì)docker的解耦,當(dāng)然這看起來(lái)較繁瑣且影響性能，故在kubernetes 1.24后，kubernetes宣布啟用dockershim，需要我們?cè)谠摪姹竞笾鲃?dòng)配置container-runtime。

2.docker這面也很早就做了應(yīng)對(duì)，docker抽離出了支持CRI標(biāo)準(zhǔn)的containerd，通過(guò)containerd來(lái)管理容器。

所以如下圖，調(diào)用docker API創(chuàng)建容器后，docker還會(huì)調(diào)用docker-containerd來(lái)管理創(chuàng)建容器，docker-containerd通過(guò)docker-containerd-shim來(lái)間接管理container，這樣一個(gè)好處就是升級(jí)或重啟docker，我們的業(yè)務(wù)容器依然可以正常運(yùn)行，最終docker-containerd-shim通過(guò)runc來(lái)創(chuàng)建container，runc是docker做的基于oci的實(shí)現(xiàn)就是以前的libcontainer，用于容器創(chuàng)建。

kubelet創(chuàng)建pod整體架構(gòu)圖

（container-runtime="docker"，大多數(shù)企業(yè)目前應(yīng)該都是使用的這種方式）

詳解kubelet?創(chuàng)建pod流程代碼圖解及日志說(shuō)明

kubelet創(chuàng)建pod日志說(shuō)明

我們通過(guò)實(shí)戰(zhàn)，開(kāi)啟debug日志來(lái)看下kubelet在創(chuàng)建pod時(shí)做了哪些工作

注：日志僅保留主要輸出及過(guò)濾敏感信息

1.收到新pod創(chuàng)建時(shí)間，寫(xiě)入updatechannel通道
I0921 18:10:00.486345 26075 config.go:414] Receiving a new pod "opslk1-xxx_lktest01(xxx-3995-11ed-80a8-48df37244930)"

2.syncLoop: 收到add事件
I0921 18:10:00.757557 26075 kubelet.go:2007] SyncLoop (ADD, "api"): opslk1-xxx_lktest01(xxx-3995-11ed-80a8-48df37244930)

3.準(zhǔn)入驗(yàn)證pod fit success
I0921 18:10:00.759786 26075 predicates.go:986] Pod: opslk1-xxx fit success. Node: xx.xx.10.9 has enough resources.

4.流轉(zhuǎn)至syncPod，SyncPodType=create
I0921 18:10:00.759956 26075 kubelet.go:1498] syncPod "xxx-3995-11ed-80a8-48df37244930" updateType:{{ } types.SyncPodType=create)

5.獲取pod狀態(tài)
I0921 18:10:00.760128 26075 kubelet_pods.go:1529] Generating status for "opslk1-xxx_lktest01(xxx-3995-11ed-80a8-48df37244930)"
I0921 18:10:00.760148 26075 kubelet_pods.go:1494] pod waiting > 0, pending
I0921 18:10:00.760174 26075 kubelet.go:1603] apiPodStatus.Phase:Pending pod:"opslk1-xxx_lktest01(xxx-3995-11ed-80a8-48df37244930)"

6.配置cgroupConfig,設(shè)置cpu，內(nèi)存
I0921 18:10:00.760200 26075 kubelet_resources.go:149] Newest cgroupConfig for pod:"opslk1-5sfjn_lktest01(739e1c1a-3175-11ed-aff8-48df37244926)"
are kubelet.cgroupResource{cpuShares:xxx, cpuQuota:xxx, memoryLimit:xxx, memoryLimitSwap:xxx}.

7.等待pod相關(guān)volume attach及掛載
I0921 18:10:00.768211 26075 volume_manager.go:350] Waiting for volumes to attach and mount for pod "opslk1-xxx_lktest01(xxx-3995-11ed-80a8-48df37244930)"

8.向apiserver同步狀態(tài)，先GET后PATCH
I0921 18:10:00.791361 26075 round_trippers.go:419] curl -k -v -XGET 'https://xxx/api/v1/namespaces/lktest01/pods/opslk1-xxx'
I0921 18:10:00.794250 26075 round_trippers.go:419] curl -k -v -XPATCH 'https://xxx/api/v1/namespaces/lktest01/pods/opslk1-xxx/status'
I0921 18:10:00.798998 26075 status_manager.go:506] Status for pod "opslk1-xxx_lktest01(xxx-3995-11ed-80a8-48df37244930)" updated successfully: (1, {Phase:Pending Conditions:[{Type:Initialized

9.根據(jù)期望狀態(tài)開(kāi)始調(diào)協(xié)，Reconcile Pod "Ready" condition if necessary. Trigger sync pod for reconciliation.
I0921 18:10:00.799365 26075 kubelet.go:2020] SyncLoop (RECONCILE, "api"): "opslk1-xxx_lktest01(xxx-3995-11ed-80a8-48df37244930)"

10.mount volume
I0921 18:10:02.177479 26075 operation_generator.go:506] MountVolume.WaitForAttach succeeded for volume "volume" DevicePath "/dev/mapper/docker-xxx_3995_11ed_80a8_48df37244930"
I0921 18:10:03.136754 26075 operation_generator.go:527] MountVolume.MountDevice succeeded for volume "volume" device mount path "/export/kubelet/pods/xxx-3995-11ed-80a8-48df37244930/volumes/kubernetes.io~lvm/volume"
I0921 18:10:03.136851 26075 operation_generator.go:567] MountVolume.SetUp succeeded for volume "volume" (UniqueName: "flexvolume-kubernetes.io/lvm/xxx_3995_11ed_80a8_48df37244930") pod "opslk1-xxx"

11.volumes attached、mounted 完畢
I0921 18:10:03.168555 26075 volume_manager.go:384] All volumes are attached and mounted for pod "opslk1-xxx_lktest01(xxx-3995-11ed-80a8-48df37244930)"

12.調(diào)用 containerRuntime 的 SyncPod 方法開(kāi)始創(chuàng)建容器
I0921 18:10:03.168568 26075 kuberuntime_manager.go:468] Syncing Pod "opslk1-xxx_lktest01(xxx-3995-11ed-80a8-48df37244930)": &Pod{}

13.創(chuàng)建sandbox容器:Setting cgroup parent,RunPodSandbox,Calling network plugin cni to set up pod
I0921 18:10:03.168833 26075 kuberuntime_manager.go:398] No sandbox for pod "opslk1-xxx_lktest01(xxx-3995-11ed-80a8-48df37244930)" can be found. Need to start a new one"opslk1-xxx_lktest01(xxx-3995-11ed-80a8-48df37244930)"
I0921 18:10:03.168885 26075 kuberuntime_manager.go:605] SyncPod received new pod "opslk1-xxx_lktest01(xxx-3995-11ed-80a8-48df37244930)", will create a sandbox for it
I0921 18:10:03.168891 26075 kuberuntime_manager.go:614] Stopping PodSandbox for "opslk1-xxx_lktest01(xxx-3995-11ed-80a8-48df37244930)", will start new one
I0921 18:10:03.168901 26075 kuberuntime_manager.go:841] Stop app containers for pod:"opslk1-xxx_lktest01(xxx-3995-11ed-80a8-48df37244930)".
I0921 18:10:03.168913 26075 kuberuntime_manager.go:666] Creating sandbox for pod "opslk1-xxx_lktest01(xxx-3995-11ed-80a8-48df37244930)"
I0921 18:10:03.170818 26075 docker_service.go:460] Setting cgroup parent to: "/kubepods/burstable/podxxx-3995-11ed-80a8-48df37244930"
I0921 18:10:03.170827 26075 docker_sandbox.go:108] RunPodSandbox PodName:opslk1-xxx PodUID:xxx-3995-11ed-80a8-48df37244930 NameSpace:lktest01
I0921 18:10:04.297831 26075 plugins.go:377] Calling network plugin cni to set up pod "opslk1-xxx_lktest01"
I0921 18:10:04.298323 26075 manager.go:1011] Added container: "/kubepods/burstable/podxxx-3995-11ed-80a8-48df37244930/805dda102e017247685240c2f740295396edcb7071dfe211979215eac0870e0b"
I0921 18:10:04.298535 26075 container.go:448] Start housekeeping for container "/kubepods/burstable/podxxx-3995-11ed-80a8-48df37244930/805dda102e017247685240c2f740295396edcb7071dfe211979215eac0870e0b"
I0921 18:10:04.298693 26075 cni.go:337] Got netns path /proc/26876/ns/net
I0921 18:10:04.298701 26075 cni.go:338] Using podns path lktest01
I0921 18:10:04.298820 26075 cni.go:307] About to add CNI network cni-loopback (type=loopback)
I0921 18:10:04.301399 26075 cni.go:337] Got netns path /proc/26876/ns/net
I0921 18:10:04.301405 26075 cni.go:338] Using podns path lktest01
I0921 18:10:04.301466 26075 cni.go:307] About to add CNI network cni (type=cni)
I0921 18:10:04.392172 26075 kuberuntime_manager.go:680] Created PodSandbox "805dda102e017247685240c2f740295396edcb7071dfe211979215eac0870e0b" for pod "opslk1-xxx_lktest01(xxx-3995-11ed-80a8-48df37244930)"
I0921 18:10:04.396981 26075 kuberuntime_manager.go:699] Determined the ip "xx.xx.226.17" for pod "opslk1-xxx_lktest01(xxx-3995-11ed-80a8-48df37244930)" after sandbox changed

14,創(chuàng)建常規(guī)容器
I0921 18:10:04.397114 26075 kuberuntime_manager.go:750] Creating container &Container{} in pod opslk1-xxx_lktest01(xxx-3995-11ed-80a8-48df37244930)
I0921 18:10:04.398859 26075 kuberuntime_container.go:108] Generating ref for container opslk: &v1.ObjectReference{Kind:"Pod", Namespace:"lktest01", Name:"opslk1-xxx"}
I0921 18:10:04.398883 26075 kuberuntime_container.go:117] To determine whether to restart the old container. Pod:opslk1-xxx_lktest01 PodIP: PodSandboxId: NameSpace:lktest01
I0921 18:10:04.398888 26075 kuberuntime_container.go:258] pod:opslk1-xxx default KeepRootDirForPod: true
I0921 18:10:04.398935 26075 server.go:471] Event(v1.ObjectReference{Kind:"Pod", Namespace:"lktest01", Name:"opslk1-xxx", UID:"xxx-3995-11ed-80a8-48df37244930", APIVersion:"v1", ResourceVersion:"19846024411", FieldPath:"spec.containers{opslk}"})

以上就是詳解kubelet 創(chuàng)建pod流程代碼圖解及日志說(shuō)明的詳細(xì)內(nèi)容，更多關(guān)于kubelet創(chuàng)建pod流程的資料請(qǐng)關(guān)注其它相關(guān)文章！