Mobike大數(shù)據(jù)基礎(chǔ)平臺(tái)建設(shè)目錄?Mobike介紹?團(tuán)隊(duì)介紹?平臺(tái)演變?平臺(tái)架構(gòu)?平臺(tái)建設(shè)?未來(lái)展望Mobike介紹-全球第一大互聯(lián)網(wǎng)出行服務(wù)平臺(tái)?10+個(gè)國(guó)家200個(gè)城市?日訂單量3000萬(wàn)+?700萬(wàn)+摩拜單車?2億+用戶團(tuán)隊(duì)介紹?職責(zé)A提供一站式大數(shù)據(jù)服務(wù)平臺(tái)A提供大數(shù)據(jù)存儲(chǔ)計(jì)算平臺(tái)A提供實(shí)時(shí)搜索服務(wù)平臺(tái)A標(biāo)準(zhǔn)化數(shù)據(jù)計(jì)算邏輯和管理?用戶_____數(shù)據(jù)團(tuán)隊(duì)、研發(fā)工程師平臺(tái)歷程業(yè)務(wù)?日志處理?數(shù)據(jù)倉(cāng)庫(kù)?報(bào)表?紅包車?反作弊?實(shí)時(shí)計(jì)算?車輛調(diào)度數(shù)據(jù)量?規(guī)模>2Hadoopcluster>100+nodes?存儲(chǔ)??1PB>?日增6TB?計(jì)算>800+jobs/day>400000+Tasks/day?工具，以Hadoop為核心，Kafka>HBaseHive、Spark>Storm>ELK平臺(tái)架構(gòu)service+logstashservice+logstashservice+logstash平臺(tái)架構(gòu)數(shù)據(jù)輸出可視化展示自動(dòng)化報(bào)表運(yùn)菅決策業(yè)務(wù)診斷服務(wù)接口HTTPJSONAPI數(shù)據(jù)應(yīng)用用戶畫像推薦數(shù)據(jù)挖掘商業(yè)分析報(bào)表統(tǒng)計(jì)蛔空數(shù)據(jù)計(jì)算MapReduceSparkHiveShellOLAPSparkstreamingStormYARN蜒稀者HDFS專輸SqoopKafkaMysql/MongoLog集群portal數(shù)據(jù)管理系統(tǒng)監(jiān)控用戶管理資源調(diào)度集群狀態(tài)機(jī)器管理工具管理枳限管理流程管理配額管理數(shù)據(jù)采集平臺(tái)建設(shè)?日志收集>Logstash+Kafka+Flume-ng?離線處理>HDFSHA+RMHA/AllonYarn>Hive數(shù)倉(cāng)>SparkMllib模型訓(xùn)練?實(shí)時(shí)處理>Storm>Sparkstreaming?Es實(shí)時(shí)搜索服務(wù)?全鏈路實(shí)時(shí)監(jiān)控全鏈路監(jiān)控EventsReceivedRate(/s)0IEventsReceivedRate240,522.87/s?EventLatency1.39msEventsEmittedRate(/s)0IEventsEmittedRate240,505.1/s全鏈路監(jiān)控kafkasum40,000,00030,000,00020,000,00010,000,000010:3011:0011:30?timestampper5minuteskafk>nll>num10,000,0008,000,0006,000,0004,000,0002,000,000010:0010:3011:0011:3010:00@timestampper10minutes0?topi與.HANDdu...?topi^i'VANDclu...?topiL?_ANDclu.“?topics^ANDdu...0-to卩.■帥JANDcIil,■tojnkNDdiL,?topifll『"NDclu."?topicbUJANDclu...12:0012:3013:0013:30全鏈路監(jiān)控PartitionLogSizeOffsetOwner88341851989341875111034184961113414934112341855811334160781143415324115341870511634187571173418755341851934187483418495341493234185573416076341532334187043418757341875510,1.110.189?consum餘6-6be88375?db7b?48c8?8fd2?620a9926053110.1,110J89-consumeM-6be88375?db7b-48c8-8fd2-620a9926053110丄110,189-consume卜6?6be88375?db7b?48c8?8fd2?620a99260531101110.189-consumer-6-6be88375-db7b-48c8-8fd2-620a9926053110丄110.189-consume卜6-6be88375?db7b-48c8-8fd2-620a9926053110丄110J89?consumeM?6be88375?db7b?48c8?8fd2?620a9926053110丄110J89?consum餘6?6be88375?db7b?48c8?8fd2?620a9926053189-consume卜6-6be88375?db7b-48c8-8fd2-620a9926053110丄110,189-consume卜6-6be88375?db7b-48c8-8fd2-620a9926053110丄110J89?consumer66be88375?db7b?48c8?8fd2?620a99260531CreatedModify2017-11-2207:03:212017-11-2207:03:212017-11-2207:03:192017-11-2207:03:192017-11-2207:03:212017-11-2207:03:212017-11-2207:03:212017-11-2207:03:212017-11-2207:03:202017-11-2207:03:202017-11-2207:03:202017-11-2207:03:202017-11-2207:03:212017-11-2207:03:212017-11-2207:03:222017-11-2207:03:222017-11-2207:03:222017-11-2207:03:222017-11-2207:03:212017-11-2207:03:21全鏈路監(jiān)控IV?槌PMFtJog11I，x」ogX■illogw姍BAK_FIKE_TYPE_SIZEBAK_FIKE_TYPE_NUM01I0300Jt800\BAKFILETOTALNUMSIZEBAKFILETYPETOTALSIZE00200&MN?5—UJ-Hu.IMvm卜LOL卜LON9LOL卜LONLnLOL卜LON寸LOL卜LONELOL卜LONOLLELON寸。LL卜LONLrn。LZ.LON6LOELON8LOLZ.LON卜L。ELON9LOLZLONSLOLZ.LON寸LOELONfnLOLSLONonunu42c?TIMETIME平臺(tái)建設(shè)?Yum源?Puppet?Ansible?Zabbix?Ganglia平臺(tái)建設(shè)?Puppet配置統(tǒng)一管理?Ansible自動(dòng)化部署Puppet(Hadoop)〔Agentl][Agent2......]平臺(tái)建設(shè)?HDFS，約束目錄規(guī)范，嚴(yán)禁私自創(chuàng)建目錄>Quota限額，文件數(shù)、存儲(chǔ)?YARN>按作業(yè)類型劃分固定的Queue(online/offline/bi/dw/default)>Jobname規(guī)范格式zhangsan—xxx?集群賬號(hào)>按部門、按業(yè)務(wù)分配用戶>個(gè)人帳號(hào)僅供測(cè)試，組帳號(hào)上生產(chǎn)調(diào)度，Job提交需指定queue?數(shù)據(jù)權(quán)限>HDFSAcl?Job管理a工作流定義方式，通過(guò)配置文件，GitLabC