PAGE\*ROMANPAGE\*ROMANII邊緣云原生虛擬化研究報告2024年1月目 次前 言 III技術(shù)與需求概述 1虛擬機和容器 1OpenStack與Kubernetes 1融合管理的演進：K8s環(huán)境下運行虛擬機 3開源項目簡介 4技術(shù)實踐 9生命周期管理 9鏡像管理 10存儲管理 13網(wǎng)絡(luò)能力 15PAGEPAGE10技術(shù)與需求概述高可靠低延遲的設(shè)備接入和海量數(shù)據(jù)的實時計算問題，云技術(shù)有力的保障和推動了邊緣計算的應(yīng)用。虛擬機和容器虛擬機和容器是云計算中最常用到的應(yīng)用部署和運行方式。虛擬機是伴隨著虛擬化的技術(shù)出現(xiàn)的，容器則云原生技術(shù)的典型特征之一，他們的架構(gòu)對比如下圖所示：1：虛擬機與容器架構(gòu)對比圖如上圖所示，虛擬化技術(shù)一般通過虛擬化層（hypervisor）來實現(xiàn)，通過虛擬化技術(shù)，虛擬機可以CPU、內(nèi)存、IO（客戶機操作系統(tǒng)兩種技術(shù)的特點對比如下表：表1：虛擬機與容器技術(shù)特點對比對比項虛擬機技術(shù)容器技術(shù)安全隔離性強，操作系統(tǒng)級別弱，進程級別對宿主機操作系統(tǒng)依賴無有，需要相同操作系統(tǒng)內(nèi)核啟動時間慢，分鐘級快，秒級磁盤占用大（GB）?。∕B）虛擬化性能損耗大1小OpenStack與KubernetesOpenStack2與Kubernete（K8）3OpenStackOpenStackOpenStack系統(tǒng)必KeystonGlancNovNeutronCinde（SwifHorizoCeilomete）、Hea）、Trov）等。OpenStack（服務(wù)API下圖是OpenStack圖2OpenStackKubernetesKubernetsKubernetesKuberneteskube-apiservrkube-schedulerkube-controller-manageetc行時引擎、kubelet（節(jié)點代理程序）kube-proxy（網(wǎng)絡(luò)代理程序）。Kubernetes的設(shè)計原則包括：安KubernetesAPIKubernetes實現(xiàn)了CN容器運行時接口）、CSI（容器存儲接口）等接口和CRD（用戶自定義資源）等，便于實現(xiàn)功能的擴展。下圖是Kubernetes圖3：KubernetesOpenStackKubernetesK8s融合管理的演進：K8sNFV（networkfunctionvirtualization）網(wǎng)絡(luò)功能虛擬化的場景：將傳統(tǒng)的網(wǎng)元虛擬化，使用虛擬機會比使用容器更方便，因為容器在接入多網(wǎng)卡方面比起虛擬機的能力來說還有一定的差距；GPUGPUIP的變化，在虛擬機里部署可以有一個固定的IP，會更加方便；很多進程的應(yīng)用：在容器使用上，有個核心概念就是部署任務(wù)單一的進程，比如一個簡單的api服務(wù)進程加一個日志收集的進程組合成為了一個容器，有些多進程的應(yīng)用就不適合放在容器中運行了。計算資源：從管理角度來說計算資源的管理不同的平臺的管理方法也是截然不同的，比如OpenStack是通過projectquota來管理，而K8s則通過request/limit來管理，管理人員必須完全了解2套機制才能完全很好的管理起來；網(wǎng)絡(luò)資源：同樣，對于網(wǎng)絡(luò)管理來說，不同的平臺也是完全不同的，K8s使用CNI來管理網(wǎng)絡(luò)，同時OpenStack通過neutron來管理網(wǎng)絡(luò)，他們的使用理念上也是截然不同的，這樣很大程度上增加了學(xué)習(xí)成本；監(jiān)控/日志：各種平臺都有自己的完整的監(jiān)控/日志收集系統(tǒng)，它們會占用大量的計算、存儲和網(wǎng)絡(luò)資源，同時部署這樣2套平臺，從資源使用的角度上來說也是一種很大的浪費；K8s和OpenStac從各個方面來看，企業(yè)內(nèi)部的虛擬機平臺和容器平臺合并成為同一套平臺來進行管理是一個趨勢。那么是將K8s合并到OpenStack呢？還是反過來呢？OpenStackK8sOpenStack的zuOpenStackOpenStackAPI從云技術(shù)的現(xiàn)狀和發(fā)展來看，容器的應(yīng)用越來越廣泛，而且K8s在容器編排領(lǐng)域成為了業(yè)內(nèi)事實上的標準。在邊緣環(huán)境下，K8s的適用范圍也更加廣泛，因此，本文將進一步探討在K8s環(huán)境下運行虛擬機的技術(shù)和實踐范例。開源項目簡介本節(jié)介紹在K8s環(huán)境下運行虛擬機的相關(guān)開源項目。當(dāng)前這些項目還在發(fā)展之中，功能還在不斷地迭代和完善。KubVirtKubeVir4是一個K8s插件，由Redhat開源，云原生計算基金會（CNCF）贊助的開源項目。KubeVirt插件可以在K8s平臺上調(diào)度傳統(tǒng)的虛擬機。KubeVirt使用自定義資源（CRD）將VM管理接口接入到K8sPod去使用libvirtd管理VMPod與VM擬機KubeVirt4KubeVirt架構(gòu)圖KubeVirvirt-apkubevirt是以CRVMPodvirt-apistart、stop等操作。virt-controllevirt-controller會根據(jù)VMICRDvirt-launcherPodCRD的狀態(tài)。K8sapi-serveVMIvirt-handlevirt-handler會以deamonsetVMISpe與相應(yīng)libvirlibvirSpec的變化；VMISpecvirt-launcher：每個virt-launcherPod對應(yīng)著一個VMI，kubelet只負責(zé)virt-launcherPod運行狀態(tài)，不VMIvirt-handlerCRDvirt-launchelibvirtd實例來啟動VMI，隨著Pod的生命周期結(jié)束，virt-lanuncher也會去通知VMI去執(zhí)行終止操作；其次在每個virt-launcherPod中還對應(yīng)著一個libvirtd，virt-launcher通過libvirtd去管理VM的生命周期，不再是以前的虛擬機架構(gòu)那樣一個libvirtd去管理多個VM。是kubevirt自帶類似kubectlvirt-launcherPod這一層去直接管理VM虛擬機，可以控制VM的start、stop、restart。KubeVir利用CRDVirtualMachineInstance（VMI）：類似于KubernetesPod，是管理虛擬機的最小資源。一個VirtualMachineInstanceVirtualMachine（VM）：為群集內(nèi)的VirtualMachineInstance提供管理功能，例如開機/關(guān)機/重啟虛擬機，確保虛擬機實例的啟動狀態(tài)，與虛擬機實例是1:1的關(guān)系，類似與spec.replica為1的StatefulSet。VirtualMachineInstanceMigrations：提供虛擬機遷移的能力，雖然并不會指定具體遷移的目的節(jié)點，但要求提供的存儲支持RWX讀寫模式。VirtualMachineInstanceReplicaSeReplicaSetVirtualMachineInstanceVirtualMachineInstanceHPA。KubeVirVMDataVolume/PVCHarborIPVMIVMLauncherPodVM//熱刪VM安全的同時解決業(yè)務(wù)存儲空間需求和主機異常Hung等問題。VMCPU/MEM虛擬機刪除：對虛機資源進行回收，但VM所屬的磁盤數(shù)據(jù)仍將保留、具備恢復(fù)條件。KataContainerKataContainer5社區(qū)由OpenStackFoundatio（OSKataContainerKataContainerI/OKubernetesCRILiunx簡單：易于集成和使用。圖：KataContainerKataContainerkata-agent：在虛擬機內(nèi)kata-agent作為一個daemon進程運行，并拉起一個或多個容器的進程。kata-agent使用VIRTIO或VSOCK接口（QEMU在主機上暴露的socket文件）在guest虛擬機中運行g(shù)RPC服kata-runtime通過grpckata-agentkata-agent例如DockerEngineI/（stdoustderrstdinVIRTIOKataContainersproxy(kata-proxy)IOkata-runtimKataContainersruntime(kata-runtime)通過QEMU/KVM兼容OCIruntimespecification標準，支持Kubernetes的ContainerRuntimeInterface(CRI)接口，可替換CRIshimruntime(runc)通過K8sPodkata-proxkata-prox提供了kata-shim和kata-runtimeVM中的kata-agentShim：kata-shim類似Docker的containerd-shim或CRI-O的conmon，主要用來監(jiān)控和回收容器的進kata-shimIOstdout,stdinandstderKataContainerKataShimVcontainerd-shim-kata-v2ContainerdRuntimeV2(ShimAPI)kata-runtimkata-shimkata-proxy圖：KataShimV2演進圖Hypervisokata-container通過QEM/KVMhypervisor。Kube-OVNKube-OVN6是一款CNCF旗下的企業(yè)級云原生網(wǎng)絡(luò)編排系統(tǒng)，將SDN的能力和云原生結(jié)合，提供豐Kube-OVNKuberneteKuberneteKubernete生態(tài)Runtime的穩(wěn)定性和易用性。Kube-OVN的設(shè)計原則和思路是，平移OpenStack網(wǎng)絡(luò)的概念和功能到Kubernetes。OpenStack的網(wǎng)絡(luò)已經(jīng)發(fā)展了很多年，很多設(shè)計和概念也基本成了SDN的標準。Kube-OVN通過引入高級功能和成熟的網(wǎng)絡(luò)概念，從整體上增強Kubernetes網(wǎng)絡(luò)的能力，并通過OVN實現(xiàn)網(wǎng)絡(luò)的數(shù)據(jù)平面，簡化維護工作。Kube-OVN的組件可以大致分為三類：上游OVN/OVS組件。Agent。監(jiān)控，運維工具和擴展組件。圖7：Kube-OVN架構(gòu)圖上游OVN/OVS組件OVN/OVSKube-OVOVN/OVSKubernete之內(nèi)。ovn-central：Deployment運行OVN的管理平面組件，包括ovn-nb、ovn-sb和ovn-northd。多個ovn-centraRaftovn-nbAPIkube-ovn-controllerovn-nb進行交互配置虛擬網(wǎng)絡(luò)。ovn-sbovn-nbovn-northdovn-nbovn-sbovs-ovn：ovs-ovn以DaemonSet形式運行在每個節(jié)點，在Pod內(nèi)運行了openvswitch、ovsdb和ovn-controlleovn-centralAgen核心控制器和Agent該部分為Kube-OVN的核心組件，作為OVN和Kubernetes之間的一個橋梁，將兩個系統(tǒng)打通并將網(wǎng)絡(luò)概念進行相互轉(zhuǎn)換。kube-ovn-controller：該組件為一個Deployment執(zhí)行所有Kubernetes內(nèi)資源到OVN資源的翻譯工作，其作用相當(dāng)于整個Kube-OVN系統(tǒng)的控制平面。kube-ovn-controller監(jiān)聽了所有和網(wǎng)絡(luò)功能相關(guān)資源的事件，并根據(jù)資源變化情況更新OVN內(nèi)的邏輯網(wǎng)絡(luò)。主要監(jiān)聽的資源包括：Pod、Service、Endpoint、Node、NetworkPolicy、VPC、Subnet、Vlan、ProviderNetwork。kube-ovn-cni：該組件為一個DaemonSet運行在每個節(jié)點上，實現(xiàn)CNI接口，并操作本地的OVS配置單機網(wǎng)絡(luò)。kube-ovn-cni會配置具體的網(wǎng)絡(luò)來執(zhí)行相應(yīng)流量操作。監(jiān)控，運維工具和擴展組件Kube-OVN的核心網(wǎng)絡(luò)能力進行擴展，并簡化日常運維操作。kube-ovn-speakeDaemonSetIP訪問容器。kube-ovn-pinger：該組件為一個DaemonSet運行在每個節(jié)點上收集OVS運行信息，節(jié)點網(wǎng)絡(luò)質(zhì)量，網(wǎng)絡(luò)延遲等信息，收集的監(jiān)控指標可參考Kube-OVN監(jiān)控指標。kube-ovn-monitor：該組件為一個Deployment收集OVN的運行信息，收集的監(jiān)控指標可參考Kube-OVN監(jiān)控指標。kubectl-ko：該組件為kubectl插件，可以快速運行常見運維操作。技術(shù)實踐本章通過一些典型地范例介紹對于在K8s環(huán)境下運行虛擬機的功能增強的技術(shù)實踐。生命周期管理2.1.1 K8s在K8s中啟動的虛擬機都是在一個Pod里面運行著libvirtd和qemu等依賴組件，這樣kube-scheduler不需要感知Pod里是一個虛擬機還是一個容器，都按照統(tǒng)一的方式進行管理。既然虛擬機運行在了K8s平臺上，那么我們管理虛擬有可以通過kubectl進行管理。創(chuàng)建虛擬機通過kubectlcreate-fvm1.yamyam通過kubectleditvm-nnamespace1vimyaml文件。刪除虛擬機通過kubectldeletevmvm1nnamespace1來刪除在namespace1下的一個虛擬機vm1。虛擬機熱調(diào)整資源由于K8s最近的版本已經(jīng)支持Pod原地擴容了，可以利用了這個功能，實現(xiàn)kubevirt的虛擬機的cpu和memorcpu社區(qū)的熱擴容的實現(xiàn)：社區(qū)目前之實現(xiàn)了通過了livemigration（熱遷移）功能來實現(xiàn)的，這樣的實現(xiàn)依賴虛擬機是否可以進行熱遷移，比如虛擬機如果有g(shù)pu掛載的話，就不能實現(xiàn)熱遷移，也就不能熱擴容。圖8：社區(qū)版熱擴容改進的實現(xiàn)：首先使用了1.27.x版本的特性Pod原地擴容的特性Podin-placeVPA)先將外部的virt-launcherPod的limitlibvirtap圖9：改進版虛擬機熱擴容鏡像管理從Harbor成本Harbor虛擬機的鏡像了。Harbo7是由VMwareDockerRegistr(RBACLDAP、日志審核、管理界面、自我注冊、鏡像復(fù)制和中文支持等功能。#kubectlgetcm-nmec-imageshci#kubectlgetcm-nmec-imageshci-controller-config-oyaml#kubectleditcm-nmec-imageshci-controller-config-oyamlapiVersion:v1data:config.yaml:|healthzPort:8080resyncPeriod:leaderElection:leaderElect:trueleaseDuration:30srenewDeadline:resyncPeriod:5srresourceName:hci-controllerresourceLock:endpointsleasesresourceNamespace:mec-imagescontrollerConfig:baseImageNamespace:mec-imagessnapshotClass:csi-rbdplugin-snapclass#nameofVolumeSnapshotClassglanceBaseURL:00:9292registryAddr:harbor.mec.localkind:metadata:annotations:kubectl.kubernetes.io/last-applied-configuration:|{"apiVersion":"v1","data":{"config.yaml":"healthzPort:8080\nresyncPeriod:creationTimestamp:"2022-07-06T14:44:34Z"name:hci-controller-confignamespace:mec-imagesresourceVersion:"18229389"uid:3de8bcfc-f87d-4be5-9c85-d84198866133#kubectlcreate-fevm-registry-f#kubectlcreate-fevm-registry-fedora.yaml#catevm-registry-fedora.yamlapiVersion:mececs.io/v1beta1kind:EnhancedVirtualMachinemetadata:name:ecs-registry-fedoranamespace:wq-testspec:template:spec:running:truetemplate:metadata:labels:kubevirt.io/vm:ecs-registry-fedoraannotations:ovn.kubernetes.io/allow_live_migration:'true'K8.cf.io/networks:mec-nets/attachnet1attachnet1.mec-nets.ovn.kubernetes.io/logical_switch:subnet-ipv4attachnet1.mec-nets.ovn.kubernetes.io/default_route:'true'attachnet1.mec-nets.ovn.kubernetes.io/allow_live_migration:'true'spec:domain:cpu:sockets:4cores:1threads:1memory:guest:clock:timezone:"Asia/Shanghai"timer:rtc:present:devices:disks:disk:bus:virtioname:interfaces:bridge:{}name:resources:requests:cpu:2memory:2048MidnsPolicy:"None"dnsConfig:nameservers:-14options:name:value:"5"hostname:"ecs-registry-fedora"networks:name:attachnet1multus:networkName:mec-nets/attachnet1volumes:name:cloudinitdiskcloudInitNoCloud:userData:|-#cloud-configpassword:ssh_pwauth:Truechpasswd:{expire:False}source:registryImageURL:kubevirt/fedora36:latestbootVolume:resources:requests:storage:10Gi##kubectlgetevmecs-registry-fedora-nwq-testNAME AGENAME AGEecs-registry-fedora3h1m#kubectlgetvmecs-registry-fedora-nwq-testNAME AGESTATUSREADYecs-registry-fedora3h1mRunningTrue#kubectlgetvmiecs-registry-fedora-nwq-testNAME AGEPHASE IPecs-registry-fedora3h1mRunning6ecs8NODENAMEREADYTrue存儲管理KataContainerKatephrbIO性能。RBDimagvirtio-fsKataContainer可以添加卷直通功能，分為半直通（塊直通）和全直通（rbd直通）兩個模式。qmp命令直KataContainer中，KataContainer再通過moun??rbdimageqmKataContainerKataContainer再通過mounPVCannotationsPVCannotations字段。volume.katacontainers.io/direct-mode值為“blockvolume.katacontainers.io/direct-mode值為“rb在字段的值不為上述兩個或者不添加該字段時為原有模式。[root@deployersimple-test]#kubectl[root@deployersimple-test]#kubectlexec-itcentos-blk-test--bash[root@centos-blk-test/]#df-hFilesystemSizeUsedAvailUse%Mountedon/dev/sdb4.9G265M4.4G6%/tmpfs64M064M0%/devtmpfs998M0998M0%/sys/fs/cgroup/dev/sdc2.0G6.0M1.9G1%/data-rbd/dev/sdd2.0G6.0M1.9G1%/data-blockshm998M0998M0%/dev/shm[root@centos-blk-test/]#lsblkNAMEMAJ:MINRMSIZEROTYPEMOUNTPOINTsda8:005G0disksdb8:1605G0disk/sdc8:3202G0disk/data-rbdsdd8:4802G0disk/data-blockpmem0259:00382M1disk`-pmem0p1259:10380M1part半直通模式，創(chuàng)建Pod之前通過lsblk查看host的塊設(shè)備信息，如果為半直通則創(chuàng)建完P(guān)od之后host會多出來?個rbd的塊設(shè)備[[root@host1~]#lsblkNAMEMAJ:MINRMSIZEROTYPEMOUNTPOINTsr011:01486K0romrbd1251:1602G0diskvda252:0050G0disk└─vda1252:1050G0part/KataContainer和OpenEBSKatOpenEBSIO性能。OpenEBS8CNCFOpenEBS是K8s本地超融合存儲解OpenEBS支持兩大類卷，本地卷和復(fù)制卷。管理員和開發(fā)人員可以使用kubectl、Helm、Prometheus、Grafana、WeaveScop等K8OpenEB。KataContainer使用OpenEBOpenEB的lvm-localpvCSInodeDriveNodePublishVolumtargetPath。NodeUnPublishVolumetargetPat卸載。NodeExpandVolume：對volumNodeGetVolumeStats修改方案依賴PVCannotations實現(xiàn)是否為直通卷的判斷，在CSIkata進?適配：NodePublishVolum：通過annotations判斷是否為直通卷；通過targetPath目錄下的文件判斷該直通卷是否已經(jīng)掛載，已經(jīng)掛載則直接重新調(diào)用kata-runtimedirect-volumeadd即可重啟后kata-runtimeadd；如果未掛載則先對塊設(shè)備進?格式化（OpenEBS通過調(diào)用K8s的庫實現(xiàn)格式化并掛載）；kata-runtimedirect-volumeadd命令；在targetPath創(chuàng)建?個文件用于判斷是否為直通卷（因為annotations只會在stageVolume、publishVolum?NodeUnPublishVolume：通過targetPatkata-runtimedirect-volumedelete命令進刪除；NodeExpandVolume：通過targetPat如果為直通卷則在使用lvextend對文件系統(tǒng)進?擴容；kata-runtimedirect-volumeresizeNodeGetVolumeStats：通過targetPatkata-runtimedirect-volumestatsvolume狀態(tài)。網(wǎng)絡(luò)能力DPDKOVSDPDK是X86DPDKDPDK數(shù)據(jù)，不經(jīng)過內(nèi)核直接轉(zhuǎn)發(fā)到網(wǎng)卡，實現(xiàn)加速目的。DPDK加速的OVS與原始OVS的區(qū)別在于，從OVSopenvswitch.kDPDKPMDovs-vswitchd/容器不能利用DPDKspace來交互通訊，大大提高的了可以處理的網(wǎng)絡(luò)流量。圖10：DPDK加速的OVSDPDK加速的OVS數(shù)據(jù)流轉(zhuǎn)發(fā)的大致流程如下：OV的ovs-vswitchdOVS目的IP、源/目的MAC、端口等信息。OVS在用戶態(tài)查看精確流表和模糊流表，如果命中，則直接轉(zhuǎn)發(fā)。SDNOpenFlo控制器下發(fā)新的流表，該數(shù)據(jù)包重新發(fā)起選路、匹配和報文轉(zhuǎn)發(fā)?？偨Y(jié)：主要區(qū)別在于流表的處理，普通OVS流表轉(zhuǎn)發(fā)在內(nèi)核態(tài)，而OVS-DPDK流表轉(zhuǎn)發(fā)在用戶態(tài)。看Pod,vmi##kubectlgetPodNAMEREADYSTATUSRESTARTSAGEvirt-launcher-vm-dpdk-6nrqt2/2Running03m13s#kubectlgetvmi-ANAME AGEPHASE IP NODENAMEREADYvm-dpdk3m9sRunning73node173True#kubectlkovsctlnode173show340aee64-1e86-486a-a99e-a62481e9d67aBridgebr-intfail_mode:securedatapath_type:netdevPort#kubectlkovsctlnode173show340aee64-1e86-486a-a99e-a62481e9d67aBridgebr-intfail_mode:securedatapath_type:netdevPort"1e9da8d5d0d7_h"Interface"1e9da8d5d0d7_h"Portovn0Interfaceovn0type:internalPort"43d5339cda9a_h"Interface"43d5339cda9a_h"PPort"99e85593750e_h"Interface"99e85593750e_h"Portd6653a5bbdc2_hInterfaced6653a5bbdc2_hPortede503e0_net2_hInterfaceede503e0_net2_htype:dpdkvhostuserclientoptions:{vhost-serverpath="/var/run/openvswitch/vhost_sockets/578dd327-f9ba-4a1b-8bd3-1a55351e07ab/vhostuser-sockets/net2"}KataContainerSR-IOVDPDKKataContainer默認采用QEMU作為hypervisor，而QEMU不支持veth，所以一般默認方案是采用TAP來為VMK8環(huán)境下KataContainer使用MellanoxDPDKDPDSRIOsinglerootIOvirtualization)技術(shù)同樣也為了提高虛擬機/容器的網(wǎng)絡(luò)性能，SRIOV技術(shù)可以將一張物理網(wǎng)卡變?yōu)槿舾蓚€虛擬的物理網(wǎng)卡，并直接接入虛擬機/容器從而提高網(wǎng)絡(luò)性能。內(nèi)核編譯通常linux啟動時先加載kernel，再加載initrd.img，initrd.img通常被用來加載驅(qū)動模塊。但是在KataContainer中，不能通過啟動時加載驅(qū)動模塊，所以需要在編譯kernel時將需要的驅(qū)動模塊通過配置編譯到內(nèi)核文件中。CONFIG_DCBnetworkingoptions->DataCCONFIG_DCBnetworkingoptions->DataCenterBridgingsupportCONFIG_INFINIBANDdeviceDrivers->InfiniBandsupportCONFIG_DYNAMIC_MEMORY_LAYOUTProcessortypeandfeatres->RandomizethekernelmemorysectionsCONFIG_COMPATBinaryEmulations->IA32EmulationCONFIG_NUMAProcessortypeandfeatures->NUMAMemoryAllocationandSchedulerSupportCONFIG_PAGE_POOLGeneralsetup->PageallocatorrandomizationCONFIG_MODULESenable_loadablemodulesupportCONFIG_PCIdevicedriver->UserspaceI/OdriversCONFIG_MLX5devicedriver->networkdevicesupport->Ethernetdriversupport->MellanoxdevicesKernel-header包制作Mellanox驅(qū)動安裝需要依賴kernel-header模塊，所以需要提前準備和鏡像制作系統(tǒng)相同內(nèi)核的Kernel-heade$$makedeb-pkg編譯Mellanox驅(qū)動FROM***WORKDIRDocFROM***WORKDIR##加入Kernel-heade模塊包ADD*.deb./#安裝Kernel-heade模塊包RUNdpkg*.deb#安裝編譯需要的軟件和庫RUNapt-getupdate&&apt-getinstall-y\gcc\#下載MellanoADDMLNX_OFED_LINUX-5.7--ubuntu20.04-x86_64./MLNX_OFED_LINUX-5.7--ubuntu20.04-x86_64#MellanoRUN./mlnxofedinstall--upstream-libs--dpdk#運行DPDK相關(guān)應(yīng)用KataContainerSR-IOV設(shè)備和DPDKIPv4v6開啟雙?？梢宰孠8s平臺上的虛擬機/容器能夠同時支持ipv4/ipv6（可選）雙協(xié)議棧。虛擬機開啟IPv4v6雙棧需要K8s和Kube-ovn都開啟雙棧。K8s啟用雙棧IPv4vIPv6DualStacK8s采用kubeadmyam文件。##vim/etc/kubernetes/manifests/kube-apiserver.yaml---feature-gates=IPv6DualStack=trueservice-cluster-ip-range=/18,fd00:10:96::/112kube-controller-manageIPv4v6,并增加Pod/serviceIPv6CID。##vim/etc/kubernetes/manifests/kube-controller-manager.yaml---feature-gates=IPv6DualStack=trueservice-cluster-ip-range=/18,fd00:10:96::/112cluster-cidr=/16,fc00::/48--node-cidr-mask-size-ipv4=24--node-cidr-mask-size-ipv6=64#vim/etc/sys#vim/etc/sysconfig/kubelet#vim/var/lib/kubelet/config.yamlKUBELET_EXTRA_ARGS="--feature-gates=IPv6DualStack=true"kube-proxy：啟用IPv4v6雙棧特性,并增加PodIPv6CIDR。##kubectl-nkube-systemeditcmkube-proxydata:config.conf:|-featureGates:IPv6DualStack:trueclusterCIDR:/16,fc00::/48Kube-ovn啟用雙棧##viminstall.sh開啟雙棧部署好Koube-ovn后，在配置?網(wǎng)雙棧時，需要設(shè)置?網(wǎng)CIDR格式為cidr=<IPv4apiVersion:kubeovn.io/v1kind:SubnetmetadatapiVersion:kubeovn.io/v1kind:Subnetmetadata:name:namespace:sgbjspec:vpc:vpc-bj1-sgprotocol:IPv4default:falsecidrBlock:/24,fd00:10:18::/64excludeIps:--fd00:10:18::1gateway:,fd00:10:18::1gatewayNode:""disableGatewayCheck:truegatewayType:distributednatOutgoing:trueprivate:false正常指定?網(wǎng)啟動Pod。#kubectl#kubectlexec-itPod--ipa52:eth0@if53:<BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN>mtu1400qdiscnoqueuestate#catapiVersion:v1kind:Podmetadata:name:Podnamespace:defaultannotations:ovn.kubernetes.io/logical_switch:ovn-defaultK8.cf.io/networks:mec-nets/attachnetsg,mec-nets/attachnet1sgattachnetsg.mec-nets.ovn.kubernetes.io/logical_switch:subnet1-bj1attachnet1sg.mec-nets.ovn.kubernetes.io/logical_switch:subnet2-bj1spec:containers:-name:spec-subnet4command:["/bin/ash","-c","trap:TERMINT;sleep36000&wait"]image:rancher/curlUPUPlink/ether00:00:00:5d:bd:30brdff:ff:ff:ff:ff:ffinet8/16brd55scopeglobaleth0valid_lftforeverpreferred_lftforeverinet6fd00:10:16::1c/64scopeglobalvalid_lftforeverpreferred_lftforeverinet6fe80::200:ff:fe5d:bd30/64scopelinkvalid_lftforeverpreferred_lftforever54:net1@if55:<BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN>mtu1400qdiscnoqueuestateUPlink/ether00:00:00:2f:af:e4brdff:ff:ff:ff:ff:ffinet/24brd55scopeglobalnet1valid_lftforeverpreferred_lftforeverinet6fd00:10:18::5/64scopeglobalvalid_lftforeverpreferred_lftforeverinet6fe80::200:ff:fe2f:afe4/64scopelinkvalid_lftforeverpreferred_lftforever56:net2@if57:<BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN>mtu1400qdiscnoqueuestateUPlink/ether00:00:00:09:8a:5dbrdff:ff:ff:ff:ff:ffinet0/24brd55scopeglobalnet2valid_lftforeverpreferred_lftforeverinet6fd00:10:17::a/64scopeglobalvalid_lftforeverpreferred_lftforeverinet6fe80::200:ff:fe09:8a5d/64scopelinkvalid_lftforeverpreferred_lftforever#kubectlexec-itPod--ip-6routeshowfd00:10:16::/64deveth0metric256fd00:10:17::/64devnet2metric256fd00:10:18::/64devnet1metric256#kubectl#kubectlkonbctlshowswitchportaddresses:["00:00:00:E6:21:AEfd00:10:18::3"]portaddresses:["00:00:00:2F:AF:E4fd00:10:18::5"]porttype:routerrouter-port:#kubectlkonbctllr-route-listovn-clusterIPv4Routes src-ipIIPv6Routesfd00:10:16::4fd00:100:64::3src-ip##ipa8:ovn0:<BROADCAST,MULTICAST,UP,LOWER_UP>mtu1400qdiscnoqueuestateUNKNOWNgroupdefaultqlen1000link/ether00:00:00:f0:ac:c6brdff:ff:ff:ff:ff:ffinet/16brd55scopeglobalovn0valid_lftforeverpreferred_lftforeverinet6fd00:100:64::2/64scopeglobalvalid_lftforeverpreferred_lftforeverinet6fe80::200:ff:fef0:acc6/64scopelinkvalid_lftforeverpreferred_lftforeverKataContainer使用IPv4v6雙棧#catapiVersi#catapiVersion:v1kind:Podmetadata:name:Pod10namespace:annotations:ovn.kubernetes.io/logical_switch:ovn-defaultK8.cf.io/networks:mec-nets/attachnetsg,mec-nets/attachnet1sgattachnetsg.mec-nets.ovn.kubernetes.io/logical_switch:subnet1-bj1attachnet1sg.mec-nets.ovn.kubernetes.io/logical_switch:subnet2-bj1spec:runtimeClassName:kata-qemunodeName:mastercontainers:-name:Pod7command:["/bin/ash","-c","trap:TERMINT;sleep36000&wait"]image:rancher/curldnsPolicy:dnsConfig:nameservers:#ipa2:eth0:<BROADCAST,MULTI#ipa2:eth0:<BROADCAST,MULTICAST,UP,LOWER_UP>mtu1400qdiscfqstateUPqlen1000link/ether00:00:00:75:b3:f2brdff:ff:ff:ff:ff:ffinet/16brd55scopeglobaleth0valid_lftforeverpreferred_lftforeverinet6inet6fd00:10:16::4/64scopeglobalvalid_lftforeverpreferred_lftforeverinet6fe80::200:ff:fe75:b3f2/64scopelinkvalid_lftforeverpreferred_lftforever3:net1:<BROADCAST,MULTICAST,UP,LOWER_UP>mtu1400qdiscfqstateUPqlen1000link/ether00:00:00:a8:1a:bbbrdff:ff:ff:ff:ff:ffinet/24brd55scopeglobalnet1valid_lftforeverpreferred_lftforeverinet6fd00:10:18::2/64scopeglobalvalid_lftforeverpreferred_lftforeverinet6fe80::200:ff:fea8:1abb/64scopelinkvalid_lftforeverpreferred_lftforever4:net2:<BROADCAST,MULTICAST,UP,LOWER_UP>mtu1400qdiscfqstateUPqlen1000link/ether00:00:00:c7:37:2ebrdff:ff:ff:ff:ff:ffinet/24brd55scopeglobalnet2valid_lftforeverpreferred_lftforeverinet6fd00:10:17::2/64scopeglobalvalid_lftforeverpreferred_lftforeverinet6fe80::200:ff:fec7:372e/64scopelinkvalid_lftforeverpreferred_lftforeverVM使用IPv4v6雙棧#kubectlexec-itPod10-nsgbj--sh[fedo#kubectlexec-itPod10-nsgbj--sh[fedora@vm1~]$ipa2:eth0:<BROADCAST,MULTICAST,UP,LOWER_UP>mtu1400qdiscfq_codelstateUPgroupdefaultqlen1000link/ether00:00:00:72:cc:56brdff:ff:ff:ff:ff:ffaltnameenp1s0inet/24brd55scopeglobaldynamicnoprefixrouteeth0valid_lft86313494secpreferred_lft86313494secinet6fd00:10:18::9/128scopeglobaldynamicnoprefixroutevalid_lft86313495secpreferred_lft86313495secinet6fe80::200:ff:fe72:cc56/64scopelinknoprefixroutevalid_lftforeverpreferred_lftforever3:eth1:<BROADCAST,MULTICAST,UP,LOWER_UP>mtu1400qdiscfq_codelstateUPgroupdefaultqlen1000link/ether00:00:00:86:e6:babrdff:ff:ff:ff:ff:ffaltnameenp2s0inet/24brd55scopeglobaldynamicnoprefixrouteeth1valid_lft86313494secpreferred_lft86313494secinet6fd00:10:17::9/128scopeglobaldynamicnoprefixroutevalid_lft86313495secpreferred_lft86313495secinet6fe80::200:ff:fe86:e6ba/64scopelinknoprefixroutevalid_lftforeverpreferred_lftforever[f[fedora@vm1~]$ip-6r::1devloprotokernelmetric256prefmediumfd00:10:17::9deveth1protokernelmetric101prefmediumfd00:10:17::/64deveth1protorametric101prefmediumfd00:10:17::/64deveth1protorametric101prefmediumfd00:10:18::9deveth0protokernelmetric100prefmediumfd00:10:18::/64deveth0protorametric100prefmediumfe80::/64deveth0protokernelmetric100prefmediumfe80::/64deveth1protokernelmetric101prefmediumdefaultviafe80::200:ff:fed6:6258deveth0protorametric100prefmediumdefaultviafe80::200:ff:fe2b:192ddeveth1protorametric101prefmediumNicPort面體CR叫做NicPorCRI單獨創(chuàng)建，然后由管理員決定將創(chuàng)建好的實列分配給需要的虛擬機，從而達到良好的用戶體驗。創(chuàng)建NicPor創(chuàng)建NicPor時選擇v創(chuàng)建NicPorv創(chuàng)建NicPort創(chuàng)建NicPort需要選擇??，可以通過kubectlgetsubnet查看??信息和CIDRIP時要滿?IP在所選??的CIDR???動分CNkube-ov，network_typebridge。##kubectlgetsubnetNAMEPROVIDERVPCPROTOCOLCIDRPRIVATENATEXCLUDEIPSsubnet1-bj1DEFAULTGATEWAYTYPEV4USEDV4AVAILABLEV6USEDV6AVAILABLEovnvpc-bj1-sgIPv4false truefalse distributed2 251 0/240[""]?式?：使?APIecs1NicPorvmsubnetsubnet1-bj1IP00MAC為9e:13:e7:31:56:1d，示例如下：ccurl-v-XPOST-H"Accept:application/json"-H"Content-Type:application/json"'https://ecsvip.lab.ecs.io:6443/apis/kubevirt.chinaunicom.com/v1/namespaces/default/nicports'\--cacert/etc/kubernetes/pki/ca.crt\--cert/etc/kubernetes/pki/apiserver-kubelet-client.crt\--key/etc/kubernetes/pki/apiserver-kubelet-client.key\--data'{"apiVersion":"/v1","kind":"NicPort","metadata":{"name":"nicport","namespaces":"default"},"spec":{"subnet":"subnet1-bj1","ip":"00","cni":"kubeovn","mac":"9e:13:e7:31:56:1d","network_type":"bridge","vmi":""}}'}'GETNicPort中的NicPor時綁定vcurldata中的vmivm。?式?：使?kubectlapiVersion:kubevirapiVersion:/v1kind:NicPortmetadata:name:spec:subnet:"subnet1-bj1"ip:"21"cni:"kube-ovn"mac:"9e:13:e7:31:56:1f"network_type:"bridge"vmi:""http://kubectlapply-fnicport.yaml獲取NicPort?式?：使?APIcurl-v-XGET-H"Accept:applicurl-v-XGET-H"Accept:application/json"-H"Content-Type:application/json"https://ecsvip.lab.ecs.io:6443/apis/kubevirt.chinaunicom.com/v1/namespaces/default/nicports/nicport’\--cacert/etc/kubernetes/pki/ca.crt\--cert/etc/kubernetes/pki/apiserver-kubelet-client.crt\--key/etc/kubernetes/pki/apiserver-kubelet-client.key?式?：使?kubectl通過kubectldescribenicport<nicport_name或者kubectlgetnicport<nicport_name>-oyam查看NicPor信息。Spec:Cni:kube-ovnIp:00Mac:9e:13:e7:31:56:1dnetworSpec:Cni:kube-ovnIp:00Mac:9e:13:e7:31:56:1dnetwork_type:bridgeSubnet:subnet1-bj1Vmi:Status:dhcp_advertising_ip:Stat:phase0.1//NicPort<none>列出NicPort?式?：使?APIcurl-v-XGET-curl-v-XGET-H"Accept:application/json"-H"Content-Type:application/json"'https://ecsvip.lab.ecs.io:6443/apis/kubevirt.chinaunicom.com/v1/namespaces/default/nicports?limit=500'\--cacert/etc/kubernetes/pki/ca.crt\--cert/etc/kubernetes/pki/apiserver-kubelet-client.crt\--key/etc/kubernetes/pki/apiserver-kubelet-client.key?式?：使?kubectl使用kubectlgetnicportsNicPort列表。更新NicPort可以更新NicPor的vmi設(shè)置vmi-targevmi-target（""。?式?：使?APIcurl-v-Xcurl-v-XPATCH-H"Accept:application/json"-H"Content-Type:application/merge-patch+json"'https://ecsvip.lab.ecs.io:6443/apis/kubevirt.ch/v1/namespaces/default/nicports/nicport'\--cacert/etc/kubernetes/pki/ca.crt\--c