1、阿里新一代實(shí)時計(jì)算引擎Blink技術(shù)王峰資深技術(shù)專家每日免費(fèi)獲取報告1、每日微信群內(nèi)分享5+最新重磅報告；2、每日分享當(dāng)日華爾街日報、3、每周分享經(jīng)濟(jì)學(xué)人4、每月匯總500+份當(dāng)月重磅報告（增值服務(wù)）；掃一掃二維碼關(guān)注公號回復(fù)：研究報告加入“起點(diǎn)財(cái)經(jīng)”微信群。姓名：王峰花名：莫問2006.4 碩士畢業(yè)于北航計(jì)算機(jī)系2006.4 目前 阿里巴巴集團(tuán)資深技術(shù)專家負(fù)責(zé)實(shí)時計(jì)算北京團(tuán)隊(duì)1.流式計(jì)算架構(gòu)2.Apache Flink3.Alibaba BlinkBlink應(yīng)用案例4.1.流式計(jì)算架構(gòu)2.Apache Flink3.Alibaba BlinkBlink應(yīng)用案例4.流式計(jì)算模型User Cod

2、eUser CodeUser Code對于不間斷的數(shù)據(jù)流持續(xù)執(zhí)行計(jì)算Producer有狀態(tài)的流式計(jì)算什么是有狀態(tài)的計(jì)算?需要處理跨多條記錄信息的計(jì)算User code狀態(tài)一致性exactly onceat least oncestate狀態(tài)管理checkpoint, recoveryupgrade, version counters: sum, max, min, average, etc. windows: time/count window machine learning: model parameters流式計(jì)算中靈活的Window機(jī)制Time-driven (examples: ev

3、ery 30 seconds)Data-driven (examples: every 1000 records) Sliding window (no overlap) Tumbling window(with overlap) Session window (punctuated by a gap ofinactivity)流式計(jì)算中亂序數(shù)據(jù)處理Event-Time(CreatedTime)Out-of-OrderRecordsCoreFeatureisWatermark1.流式計(jì)算架構(gòu)2.Apache Flink3.Alibaba BlinkBlink應(yīng)用案例4.Latency down

4、 to the millisecondsLatencyVolume/ ThroughputState & Accuracy10s of millions events/sec for stateful applicationsExactly-once semantics Event time processingFlink as a “true”StreamingComputing Engine完備的Streaming語義真正的流式執(zhí)行引擎，高吞吐，低延遲提供Exactly-Once的一致性保證靈活豐富的window支持支持Out-of-Order亂

5、序數(shù)據(jù)處理Programming ModelHigh-level Language部署在多種調(diào)度系統(tǒng)Declarative DSLStandaloneYARN, MesosCore APIs豐富的狀態(tài)存儲支持Java Heap + HDFSRocksdb + HDFSLow-level Building Block(streams, state, event time)Stateful Streaming ProcessingDataStream / DataSet APITable APISQLFlinkCheckpoints&Recovery(Chandy - Lamport)FlinkS

6、treamingApplicationsourcestate (embedded key-value store)operatorSinkoperatoroffsetsnapshotpersistent storage (HDFS)1.流式計(jì)算架構(gòu)2.Apache Flink3.Alibaba BlinkBlink應(yīng)用案例4.Blink在阿里的成長路線 2015.8 2015.10 2016.11 2017.5 調(diào)研Flink啟動BlinkBlink雙11上線Blink統(tǒng)一阿里實(shí)時流計(jì)算架構(gòu)最好的流式計(jì)算引擎尚未在大規(guī)模生產(chǎn)中驗(yàn)證完善Flink架構(gòu)和穩(wěn)定性將Flink在阿里生產(chǎn)上線搜索推薦全實(shí)

7、時鏈路實(shí)時商品、報表、在線學(xué)習(xí)統(tǒng)一服務(wù)阿里集團(tuán)實(shí)時計(jì)算業(yè)務(wù)基于Blink的StreamCompute-2.0Blink = Alibabas Flink VersionBlink擁抱Apache Flink開源社區(qū)Blink全面兼容Apache Flink APIBlink將長期保持和Flink主流分支發(fā)展Blink在阿里巴巴的場景對Flink的優(yōu)化改進(jìn)，一直在不斷回饋開源社區(qū)Flink計(jì)算和調(diào)度架構(gòu)分離，可以更大規(guī)模的部署到Y(jié)ARN/Mesos以及阿里內(nèi)部的調(diào)度系統(tǒng)Failover優(yōu)化，提升大規(guī)模生產(chǎn)穩(wěn)定性Incremental Checkpoint，提升大規(guī)模State場景下的性能Asyn

8、chrous I/O，提升計(jì)算吞吐量完善Streaming SQL語義和實(shí)現(xiàn)Blink在Flink社區(qū)影響力Blink團(tuán)隊(duì)中已經(jīng)培養(yǎng)出4名Flink CommitterBlink團(tuán)隊(duì)已經(jīng)連續(xù)3次贊助、參加Flink Forward，并在大會上給出TalkBlink實(shí)時計(jì)算平臺High-level APISQLDSLTable APICore APIDataStream APIDataSet APIRuntimeDistributed Streaming DataflowDeployAlibaba Apsara FuxiApache Hadoop YARNStorageApache Hadoop

9、 HDFSAlibaba Apsara PanguBlinkArchitectureSubmit/MonitorYARN (Resource Management)Launch AMRequest TMLaunch TMBlink Developing PlatformBlink RuntimeDebugYARN App MasterTask ManagerIn/outResource ManagerAlibaba Data LakeConnectorsTasksCheckpoint CoordinationMetricsMetric ReporterAlibaba MonitorJob ManagerState BackendTask ScheduleCheckpoint / RecoveryHDFS (Persistent Storage)Aliba