1、,Software and Services Group,Hadoop: the Intel Way,(Hadoop的英特爾之道),Bring New Analytics Capabilities to Hadoop Stack,何京翔,英特爾亞太研發(fā)有限公司總經(jīng)理,#,Security & Trust,Workload Consolidation,Cloud and IOT: More Users, More Device, More Data Immersive Experiences Cloud,Connectivity Data Analytics Software and Servi

2、ces Group,Open Cloud Architecture,#,Software and Services Group,Intels Vision,This decade we will create and extend computing technology to connect and enrich the lives of every person on earth,Software and Services Group # 4,Sensor Readin g,Log,Tabl e,Image,Document,Existing IT & Data,RDBM S EDW,Da

3、ta Marts,Systems BI All of Your Big Data (Structured & Unstructured),Our Big Data Goal: Make Hadoop the Foundation of Next-Gen Data Analytics Platform Data Mining and Analytics,Business Intelligenc e,Statistic Modeling,Machine Learning,# 5,HBase,HDFS,Hive,Base Station s,3G,Instantaneous query of 3G

4、records by subscribers Software and Services Group,User,Segmentation MapReduce ETL,Hadoop in Telecom Carrier Network Optimizations,Hive,Instantaneous,query (e.g., road image),Legacy applications,MapReduce HBase Stream processing (e.g., real-time road conditions) Software and Services Group # 6,Hadoo

5、p in Smart City Data mining (e.g., vehicle,tracking),Hadoop的英特爾之道,更易用 (Reduced Complexity) 更高效,企業(yè)級(jí)解決方案 Enterprise-Grade Solution 即時(shí)分析 (Instantaneous Analysis) 英特爾Hadoop發(fā)行版, 穩(wěn)定的企業(yè)級(jí)軟件產(chǎn) 品 針對(duì)垂直行業(yè)的功能 增強(qiáng),前沿技術(shù)開(kāi)發(fā) Advanced Development “Project Panthera”, Advanced development and path-finding Open source and

6、community driven,(Improved Efficiency) Bring New Analytics Capabilities to Hadoop Stack Software and Services Group # 7,英特爾Hadoop發(fā)行版 優(yōu)化的大數(shù)據(jù)處理軟件產(chǎn)品,英特爾,Hadoop Manage r 安裝、部署、 配置、監(jiān)控、 告警和訪問(wèn)控 制,利用硬件新技術(shù)進(jìn)行優(yōu)化 針對(duì)行業(yè)的功能增強(qiáng)，應(yīng)對(duì)不同行業(yè)的大數(shù)據(jù)挑戰(zhàn) 數(shù)據(jù)分析、統(tǒng)計(jì)和挖掘,Mahout,機(jī)器學(xué)習(xí),R 數(shù)據(jù)統(tǒng)計(jì),Hive,Pig,數(shù)據(jù)流處理語(yǔ)言,可靠的分布式文件系統(tǒng) Software and Servi

7、ces Group # 8,穩(wěn)定的企業(yè)級(jí)Hadoop發(fā)行版 為Hadoop提供即時(shí)數(shù)據(jù)處理能力 數(shù)據(jù)處理,工具集,from Revolution Analytics 交互式數(shù)據(jù)倉(cāng)庫(kù) MapReduce 穩(wěn)定高效的分布式計(jì)算框架 分布式、高維數(shù)據(jù)庫(kù)HBase HBase 0.94的改進(jìn)和創(chuàng)新，提供即時(shí)數(shù)據(jù)處理 HDFS,Sqoop 關(guān)系數(shù)據(jù)ETL工具 Flume 日志收集工具 Zookeeper 分布式協(xié)作服務(wù),SQL engine for,Hive/MapReduce, Better integration with existing infrastructure using SQL,HBas

8、e, Document,semantics & significantly speedup query processing on,HBase Software and Services Group # 9, Efficient utilization,of new HW platform technologies,“Project Panthera” Open source initiatives to enable advanced analytics capabilities on Hadoop Document store on,#,Software and Services Grou

9、p,即時(shí)分析 (Instantaneous Analysis),10,Instantaneous analysis with greatly enhanced HBase Stream new data into HBase for analysis in real time, Support high update rate workloads (to keep the system always up to,date), Allow very low latency, online data serving Etc.,#,11,Interactive Query on HBase (英特爾

10、Hadoop發(fā)行版) 10X faster than MapReduce For certain queries on HBase (e.g., group-by aggregation),HBase Query Engine Layer, ,Fast, distributed aggregations directly inside HBase Parallel scanning over multiple regions Advanced, distributed filtering (CRC32 comparator, fuzzy row,filter, etc.),HBase Quer

11、y Engine as New Hive,Backend Most “SELECT” automatically optimized to use HBase Query Engine “WHERE” using advanced scanner/filter “GROUP-BY” using distributed,aggregations “JOIN” stills go to MapReduce Software and Services Group,#,12,A Document Store on HBase (“Project Panthera”) Up-to 3x storage

12、reduction and 3x query speedup For Hive/MapReduce query processing on HBase (See and HBASE-6800) DOT (Document Oriented Table) on HBase, ,Each row contains a collection of documents Each document contains a collection of fields A document is mapped to a HBase column and serialized using Avro Complet

13、e transparent to existing HBase applications Software and Services Group,#,Software and Services Group,更易用 (Reduced Complexity),13, Better data mining and statistics capabilities, Full-text indexing and search, Statistic modeling with R language, Better integration with existing infrastructures, Geo

14、-distributed datacenters Full SQL support for OLAP,#,14,Full-Text Indexing and Search (英特爾Hadoop發(fā)行版) Full-text indexing and near real-time search for advanced data mining (E.g., log and click stream analysis, healthcare record analysis, etc.),Incremental full-text indexing on HBase Full-text indexin

15、g for semi-structured data (text, strings, numbers, etc.) Index incrementally built when records inserted or updated Support very high data insertion / update rate,Near real-time search Distributed, keyword or logical expression based search Zero delay of searching latest data that are just inserted

16、 Software and Services Group,#,Software and Services Group,Bring R Statistics into Hadoop,(英特爾Hadoop發(fā)行版),15,Distributed Statistic Modeling on Hadoop using R language,16,Data Center A Virtual Big Table,Cross-Datacenter BigTable/HBase (英特爾Hadoop發(fā)行版) A virtual Big Table overlaid over existing geo-distr

17、ibuted data centers, ,Global table view Data stored in geo-distributed, ,data centers Better locality & higher availability Data transfer eliminated through distributed,aggregation Data Center C Data Center B Async Replication Software and Services Group #,17,An analytical SQL engine for Hive/MapRed

18、uce (“Project Panthera”) Goal: Provide Full SQL support for OLAP in Hadoop Required by business users, enterprise applications, 3rd party tools (e.g., BI applications), etc. (See and HIVE-3472),Hive Parser,Hive-AST,HiveQL,Driver,Query,* Software and Services Group #,(Open Source) SQL Parser*,SQL- AS

19、T,SQL-AST Analyzer & Translator Subquery Multi-Table Unnesting SELECT ,Hive Semantic Analyzer INTERSECT MINUS Support Support,Hadoop MR,SQL,Hive- AST,#,Software and Services Group,更高效 (Improved Efficiency),18, Performance benchmarks & tools, Efficient utilizing of new HW platform technologies (e.g., SSD,infiniband),#,19,英特爾Hadoop發(fā)行版高效支撐海量移動(dòng)上網(wǎng) 記錄分析 聯(lián)通全國(guó)移動(dòng)用戶上網(wǎng)記錄查詢分析系統(tǒng), ,國(guó)內(nèi)首個(gè)基于Hadoop/HBase的商用電信服務(wù) 系統(tǒng) 系統(tǒng)部署 英特爾Hadoop發(fā)行版 滿足高性能的數(shù)據(jù)導(dǎo)入和快速查詢。 穩(wěn)定、易于部署和管理的企業(yè)級(jí)方案。 180+節(jié)點(diǎn)Hadoop/HBase集群 系統(tǒng)性能指標(biāo) 上網(wǎng)記錄入庫(kù)時(shí)間：一般小于30分鐘，實(shí) 際約10分鐘 具備存儲(chǔ)全國(guó)移動(dòng)用戶不小于6個(gè)月的原始 上網(wǎng)記錄能力 統(tǒng)計(jì)分析的中間報(bào)表數(shù)據(jù)保存不小于5年 上網(wǎng)記錄查詢速度：不高于1秒 支持并發(fā)查詢數(shù)目：1000請(qǐng)求/秒 Softwar