流批混部未來展望…ServerlessFlink…vn-agentvn-agent↓VirtualCluster標(biāo)準(zhǔn)化多租VirtualAPIServer輕量化多租VirtualAPIServer輕量化多租……↓↓↓agentansmagentansmflink-reserveflink-reserve-capacityflinkflink-pay-as-you-go…… …………AjobjobAjobjob …………replicas:2-ntopologySpreadConstra-labelSelector:matchLabels:/network-service-id:abcdmaxSkew:1topologyKey:topology.kubernetes.io/zonewhenUnsatisfiable:DoNotScheduleaffinity:nodeAffinity:requiredDuringSchedulingIgnoredDuri-matchExpressions:-key:topology.kubernetes.io/zoneoperator…………XXXXansmappswatchappswatchwatchwatch……pyxispyxis-managerTHANKYOUFlinkonYarnFlinkonK8sFlinkon20182023↓7.create6.createvalidate2.spec5.create...jobmanagerjobmanager2jobmanager1jobmanager1........taskmanager1....taskmanager1taskmanager2!!4.4.3.上傳jar1.提交jar3.上傳jar1.提交jar—學(xué)alerttalos!aggregatealerttalos!aggregateapply 核THANKYOUServerlessFlink:小紅書在云上k8sServerlessFlink:Xiaohongshu'sJourneyinMigratingandDeployingFlinkClusters………………各個(gè)子業(yè)務(wù)使用不同的獨(dú)占資源池，維護(hù)上百資源池人力成本巨大各個(gè)子業(yè)務(wù)使用不同的獨(dú)占資源池，維護(hù)上百資源池人力成本巨大Flink獨(dú)占池?cái)U(kuò)縮容流程復(fù)雜，涉及預(yù)算審批、云廠商開關(guān)機(jī)、機(jī)器環(huán)境初單次流程耗時(shí)在天級(jí)別nodeFlinkPodFlinkPodnodeFlinkPodFlinkPodnodeFlinkPodnodeFlinkPodFlinkPodnodeFlinkPodFlinkPodnodeFlinkPodFlinkPodnodeFlinkPodFlinkPodFlink獨(dú)占集群公共混部集群FlinkQuotaFlinkQuotaB其他業(yè)務(wù)其他業(yè)務(wù)quotaCFlinkQuotaAFlinkQuotaAnodeFlinkPodFlinkPodnodeFlinkPodFlinkPodnodeFlinkPodFlinkPodFlinkPodnodeFlinkPodFlinkPodFlinkPodnodenodeFlinkPodnodenodeFlinkPodPodFlinkPodquota擴(kuò)容quota縮容啟動(dòng)quota擴(kuò)容BaremetalSupplyBaremetalSupply標(biāo)準(zhǔn)混部池標(biāo)準(zhǔn)混部池Bu?erBu?erControllerAPIServerWorkload畫像服務(wù)Workload畫像服務(wù)ControllerMetricsAwareController更新預(yù)測(cè)指標(biāo)獲取歷史監(jiān)控監(jiān)控系統(tǒng)resourcePro?lerMetricsAware監(jiān)控系統(tǒng)resourcePro?lerMetricsAware匯總畫像，更新NodeMetrics監(jiān)聽監(jiān)聽MetricsAwareAgentAgentAgentAgentresourcePro?lerlMetricsAwareresourcePro?lerlMetricsAware實(shí)時(shí)負(fù)載感知調(diào)度SchedulerDescheduler動(dòng)態(tài)更新規(guī)則實(shí)時(shí)負(fù)載感知二次調(diào)度容器服務(wù)容器服務(wù)Node維度匯總待驅(qū)逐的pod構(gòu)造Flink請(qǐng)求URL請(qǐng)求驅(qū)逐Pod列表請(qǐng)求驅(qū)逐Pod列表校驗(yàn)pod是否在用返回RequestId或者結(jié)果根據(jù)RequestId輪訓(xùn)結(jié)果申請(qǐng)冗余pod資源ready后拉黑pod并觸發(fā)task重啟返回是否可驅(qū)逐根據(jù)返回結(jié)果執(zhí)行驅(qū)逐或放棄釋放空閑pod根據(jù)返回結(jié)果執(zhí)行驅(qū)逐或放棄收到驅(qū)逐請(qǐng)求收到驅(qū)逐結(jié)果請(qǐng)求生成并返回RequestId返回生成并返回RequestId返回RequestId對(duì)應(yīng)的請(qǐng)求狀態(tài)結(jié)束根據(jù)請(qǐng)求傳入的Pod結(jié)束根據(jù)請(qǐng)求傳入的Pod數(shù)量，申請(qǐng)對(duì)應(yīng)的新Pod否是否申請(qǐng)完成 是否申請(qǐng)完成 是否超時(shí)是觸發(fā)checkpoint觸發(fā)checkpoint獲取Pod上的Task集合主動(dòng)重啟Task上述流程正常上述流程正常執(zhí)行否設(shè)置驅(qū)逐請(qǐng)求為失敗否設(shè)置驅(qū)逐請(qǐng)求為失敗是設(shè)置驅(qū)逐請(qǐng)求為成功結(jié)束設(shè)置驅(qū)逐請(qǐng)求為成功結(jié)束使用CSIEphemeralInlineVolum資源擴(kuò)縮容耗時(shí)資源擴(kuò)縮容耗時(shí)擴(kuò)大池化資源覆蓋度年底遷移60kcore左右總體成本收益接近10k年底遷移60kcore左右總體成本收益接近10kcore應(yīng)用AutoScaler進(jìn)一步THANKYOUjobjob1job2job3...managerapiserverflinkmanagerapiserverflink/cancel等操作fetchjarjobjob-修復(fù)線程池設(shè)置bugrest服務(wù)就緒慢-不等待service關(guān)聯(lián)pod-池化復(fù)用RestClient序列化操作頻繁-刪除不必要的序列化操作operator可調(diào)諧的單集群任務(wù)數(shù)達(dá)到2w提交作業(yè)吞吐可達(dá)1000app/分鐘，全程穩(wěn)定整體吞吐1000整體吞吐1000app/m單app啟動(dòng)時(shí)延小于1.5minwatchindex優(yōu)化watcherswatchers單機(jī)Pod數(shù)與系統(tǒng)內(nèi)存關(guān)系(G)864208單機(jī)Pod數(shù)與系統(tǒng)內(nèi)存關(guān)系(G)86420850nodeCache:一輪調(diào)度中同質(zhì)作業(yè)的調(diào)度cache,nodeCache:一輪調(diào)度中同質(zhì)作業(yè)的調(diào)度cache,cachecachejobjobjoballocatejobache3allocatejobache3 源assignableexclusivewatchwatchapiserverwatchTaint/cordon/drainwatchTHANKYOULarge-ScaleCloud-NativePracticeOfFlinkatBilStreaming數(shù)量:6500+Batch數(shù)量:3000+Streaming數(shù)量:6500+Batch數(shù)量:3000+流批混部、Spark混部/opt/flink/opt/flinkconfconfpluginsusrlibL──tableapi等依賴jar物理機(jī)遷移機(jī)器從YARN物理機(jī)遷移機(jī)器從YARN集群遷移為Pod掛載物理磁盤磁盤異構(gòu)物理機(jī)磁盤數(shù)量不統(tǒng)一磁盤故障，權(quán)限問題等磁盤異構(gòu)物理機(jī)磁盤數(shù)量不統(tǒng)一磁盤故障，權(quán)限問題等磁盤負(fù)載不均RocksDB默認(rèn)隨機(jī)選擇磁盤磁盤掛載在/mnt子目錄Flink進(jìn)程啟動(dòng)時(shí)篩選可用目錄磁盤優(yōu)化殘留數(shù)據(jù)在殘留數(shù)據(jù)在Pod退出時(shí)，不會(huì)刪除數(shù)據(jù)增加Podlifecycle，Pod退出時(shí)通過preStop清理數(shù)據(jù)開啟HA時(shí)：任務(wù)可以從正確的Checkpoint恢復(fù)，但遇PodPod處于失敗狀態(tài)Pod處于Pod處于Running狀態(tài)啟動(dòng)時(shí)根據(jù)numberOfTaskSlots值調(diào)整閾值大小，避免固定參數(shù)的不足Pid數(shù)量限制優(yōu)化服務(wù)器無法創(chuàng)建線程優(yōu)化部分作業(yè)中線程泄漏，引起所在機(jī)器可用線程數(shù)不足，影響機(jī)器上所有作