B站運維系統(tǒng)從無到有的演進之路梁曉聰B站devops嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibiliB站彈幕視頻網(wǎng)站1億活躍用戶100萬活躍UP主“最大的同性交友網(wǎng)站”嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili故事的開始B站炸了嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibiliWeNeed標準化高質(zhì)量交付彈性伸縮嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili在運維人員緊缺的情況,如何標準化文檔轉(zhuǎn)化成可執(zhí)行代碼嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili堡壘機執(zhí)行

asdocker_shdocker/site.yml嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili如何提高交付質(zhì)量標準化發(fā)布,快速反饋嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili測量VS監(jiān)控youcan’toptimisewhatyoucan’tmeasure嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili測量系統(tǒng)場景接口訪問量,平均訪問耗時最近成功支付的數(shù)量采樣統(tǒng)計cache的key大小等等等….嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili如何彈性伸縮基于docker的可彈性伸縮web集群嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili負載均衡難點自動伸縮/故障轉(zhuǎn)移/代碼迭代造成的ip變更tenginereload的性能影響解決tengine+upsync+consul嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili監(jiān)控開源方案cadvisorprometheus考慮接入成本大新坑最終自研監(jiān)控agent嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili監(jiān)控influxdbmonitoragentindockermetriccpumemdiskionetworktagsmesosidcontaineripcontainernamelabels嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili注冊中心consulregisterinfo服務發(fā)現(xiàn)containeridipenv…ippoolipamplugin可用地址已用地址網(wǎng)關(guān)tengineupstreamupsyncip:portmaxfailstimeout…嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili網(wǎng)絡需求:ippercontainer業(yè)務網(wǎng)絡隔離盡量簡單可以排除bridge,host方式嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili網(wǎng)絡站隊:cni(ContainerNetworkInterface)cnm(ContainerNetworkModel)親兒子嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili網(wǎng)絡方案選型:xvlan技術(shù)overlay沒有多租戶隔離需求排障成本大沒有性能優(yōu)化能力3層路由caliconode-mesh方式

導致大量路由node-router方式

核心設備bgppeer太多2層隔離macvlan簡單性能損耗低合并到docker1.12release嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili網(wǎng)絡ipam-plugin:consulflaskapiRequestAddressReleaseAddressRequestPoolReleasePooldockerrun-itd\

--nethost\

--restartalways\

--nameipam-plugin\

-eCONSUL_HOST=x.x.x.x\

-eCONSUL_PORT=8500\

docker-reg.bilibili.co/ipam-plugin:0.10嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili日志中心logdriversyslogflumekafkahdfs/elasticsearch嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili自動伸縮mesos+marathonmesoshookeventbusautoscaler嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili策略{"app":"/web/app_name","autoscale_multiplier":0.2,"limit":"mean_cpu_usage>300ormean_memory_percent>80","max_instances":15,"min_instances":6,"times":3,"counter":0}嗶哩嗶哩-(゜-゜)つロ乾杯~-bilibili四todoswarmmesos結(jié)合更細化的autoscaling策略可遷移ip的ipam-plugin替換ten