TheJourneytoaData-DrivenEnterprisePivotalBigDataSuite2.0介紹以及OpenDataPlatform陳威Email:cwei@pivotal.ioMobile:139-1100-2375AgendaOverallBDS2.0詳解基于BDS的大數(shù)據(jù)平臺架構(gòu)ODP&OpenSource簡單案例介紹為什么Pivotal是您值得信賴的伙伴Overall新的數(shù)據(jù)與應(yīng)用時代的特征開放數(shù)據(jù)驅(qū)動的應(yīng)用云與大數(shù)據(jù)的融合避免技術(shù)鎖定依賴生態(tài)環(huán)境共同創(chuàng)新新的互聯(lián)網(wǎng)時代的準(zhǔn)則開放敏捷云親和縮短創(chuàng)新的周期降低TCO加速TimeToMarket專注解決業(yè)務(wù)問題避免IaaS鎖定安全性World’sFirst

OpenSourced,Enterprise-ClassDataPortfolio

+

OpenDataPlatformPIVOTALBIGDATASUITE開放敏捷云親和ModernDataInfrastructure

+

AdvancedAnalytics

+

AppsatScaleMultipleCloudDeploymentModels

+

BigDataSuiteonPivotalCloudFoundry數(shù)據(jù)驅(qū)動型企業(yè)的漫漫征途自動化部署和橫向擴(kuò)展分析型的應(yīng)用并零運(yùn)維通過高級深度智能分析挖掘遠(yuǎn)見與洞察現(xiàn)代化的數(shù)據(jù)基礎(chǔ)架構(gòu)數(shù)據(jù)驅(qū)動型企業(yè)的征途STOREStructuredUnstructuredHighVolumeHighVelocityANALYZEPredictiveAnalyticsMachineLearningAdvanceDataScienceRealtimeAnalyticsDEVELOPAdvancedAnalyticPipelinesRealtimeAnalyticalApplicationsGlobalScaleData-DrivenApplicationsEnterprise,Consumer,IoT,andMobileINNOVATEAgileDevExpertiseDevOpsHybridCloudContinuousDeliveryClosedLoopApplicationsAGILEDEVELOPMENTBIGDATAPREDICTIVEANALYTICSENTERPRISEPAASBigDataSuites的設(shè)計原則ModernizeYourDataInfrastructureLeverageAdvancedAnalyticsDeployData-drivenAppsatScaleAGILEDATAFlexibilityInnovationPerformanceandScaleFOUNDATIONFORALLPLAYSLambda大數(shù)據(jù)架構(gòu)

12345………………ftpJDBChdfsAPI…h(huán)ttpRESTSocketTCP/UDPMQJDBC…APIJDBChdfsAPIJDBChdfsAPIJDBChdfsRESThttp…BDS2.0BigDataSuite2015:更敏捷的數(shù)據(jù)全棧敏捷的數(shù)據(jù)自動化部署與擴(kuò)展深度智能分析現(xiàn)代化架構(gòu)SpringXDSparkPivotalHD&OpenDataPlatformPivotalGreenplumDatabasePivotalHAWQRabbitMQRedisPivotalGemFirePivotalBDSonPCFPivotalCloudFoundry數(shù)據(jù)驅(qū)動型企業(yè)征途-withPivotalBDSSTOREStructuredUnstructuredHighVolumeHighVelocityANALYZEPredictiveAnalyticsMachineLearningAdvanceDataScienceRealtimeAnalyticsDEVELOPAdvancedAnalyticPipelinesRealtimeAnalyticalApplicationsGlobalScaleData-DrivenApplicationsEnterprise,Consumer,IoT,andMobileINNOVATEAgileDevExpertiseDevOpsHybridCloudContinuousDeliveryClosedLoopApplicationsAGILEDEVELOPMENTBIGDATAPREDICTIVEANALYTICSENTERPRISEPAASSpringXDSparkPivotalHD&

OpenDataPlatformSpringXDPivotalGreenplumDatabasePivotalHAWQSpringXDPivotalGemFireRedisRabbitMQSpringIOGroovyPivotalBDSonPCFPivotalCloudFoundryPivotalLabsDataScienceDataEngineering真正變成一個數(shù)據(jù)驅(qū)動的企業(yè)很難傳統(tǒng)的數(shù)據(jù)架構(gòu)很難擴(kuò)展和處理大數(shù)據(jù)挑戰(zhàn)目前有限的技術(shù)提供商不能夠提供全面的解決方案全部自己用開源的產(chǎn)品來搭建需要極高的技能水平數(shù)據(jù)驅(qū)動型企業(yè)靈活的存儲和處理所有的數(shù)據(jù)DeployanalyticappsatscaleApplyanalyticalgorithmsDiscoverinsightsStoreanytype

andsizeofdata從不靈活的架構(gòu)遷移到敏捷的數(shù)據(jù)架構(gòu)Basedonscaled-up,proprietarysystemsSiloedandscattered

datasetsStoragelimitedwith

rigiddatamodelsElastic,Scale-out

storageandprocessingHandleanydata

typeorsourceCloudfriendlyand

open-sourcebasedAnalytics-optimized,ODP–basedHadoopdistributionIn-memory,distributedprocessingfromApacheScale-outanalyticspipelinemanagementwithdataingestionandprocessing敏捷，開放的數(shù)據(jù)存儲與處理方式Aswemovetocombineallthedatageneratedbyouractivityandtoleverageadvancedanalyticsinrealtime,there’snobetterwaytodothatthanthroughtheflexibilityandchoiceprovidedbyPivotal’sBigDataSuite.–SylvainLeBorne,EVPDataPlatforms“”DeployanalyticappsatscaleStoreanytype

andsizeofdata數(shù)據(jù)驅(qū)動型企業(yè)戰(zhàn)略性的運(yùn)用高級分析帶來的遠(yuǎn)見ApplyanalyticalgorithmsDiscoverinsights從傳統(tǒng)的BI向數(shù)據(jù)科學(xué)家方向轉(zhuǎn)型LongrunningqueriesStaledashboardsSummarizeddatasetsMassiveparallelprocessingofqueriesFreshdataMachinelearningandadvancedanalyticstools0101010101010101001010101010101011001010101010101010高級分析能力-敏捷性與性能LeadinganalyticdatawarehouseMostadvancedanalytical

SQLengineonHadoop100Xperformanceimprovementanalyzingtrendsamong500millionjobpostings.“”數(shù)據(jù)驅(qū)動型企業(yè)的基礎(chǔ)架構(gòu)需要高可擴(kuò)展性Storeanytype

andsizeofdataApplyanalyticalgorithmsDiscoverinsightsDeployanalyticappsatscale從缺乏橫向擴(kuò)展性的分析型應(yīng)用轉(zhuǎn)向橫向擴(kuò)展型應(yīng)用ServicefailureordatalossatscaleLonginnovationcyclesPoorexperienceatscaleResilient,scale-outmessagingandprocessingAgiledevelopmentwithCF-baseddataservicesLow-latency,in-memorycomputingDistributed,in-memorydatabaseforhigh-scaleNoSQLapplicationsIn-memory,datastructureserverforfastreadandwriteapplicationsRobustmessagingforhigh-scaleapplicationsLeverageBigDataSuitedataserviceswithinPivotalCloudFoundryapplicationsDeployandmanageBigDataSuitewithPivotalCloudFoundryFoundation低延遲高可用的數(shù)據(jù)存儲分析與消息驅(qū)動型應(yīng)用300%improvementinticket-servingcapacityledto30%increaseine-ticketsales.“”當(dāng)今其他大數(shù)據(jù)處理產(chǎn)品領(lǐng)域的現(xiàn)狀-紛繁復(fù)雜且固定的模式應(yīng)用的類型定價的模式業(yè)界對應(yīng)的產(chǎn)品數(shù)據(jù)量/CPU節(jié)點節(jié)點CPUs和其他約束條件CPUs和其他約束條件CPU?GreenplumOracle/DB2/XXXDatabaseCloudera/PHDHortonworks/XXXXXHadoopDistributedFileSystemImpala/Stinger/XXXXXParallelQueryEngineoverHDFSIn-MemoryDataGridSQLFire/XXXXXIn-MemoryDataGridwithSQLLayerGemFire/Terracotta/XXIn-MemorySQLOverHDFS134256Spark/Shark/GemfireXD/XXXX企業(yè)需要單獨(dú)采購，用量固定，無法靈活的組裝整個數(shù)據(jù)架構(gòu)1Pivotal化繁為簡-整合的統(tǒng)一定價模式應(yīng)用的類型GreenplumDBDatabasePivotalHDHadoopDistributedFileSystemHAWQParallelQueryEngineoverHDFSIn-MemoryDataGridSQLFireIn-MemoryDataGridwithSQLLayerGemFireIn-MemorySQLOverHDFS定價的模式Pivotal對應(yīng)全線產(chǎn)品SKU134256UnitofMeasurePriceGemFireXD整合各種需求，統(tǒng)一定價，總量固定，簡化企業(yè)的選擇和流程ODP&OpenSourceOpenDataPlatformInitiativeAcommoncorefortheHadoopecosystemEnterprise-gradesoftwarequalityRapidlyacceleratedcertificationsStrongecosystemdevelopmentODPOpenDataPlatformandHortonworksODP-OpenDataPlatformTodevelopODPCore,notadistributionCurrentlyPivotal,Hortonworks,IBM,Verizon,Telstra,andmanymoreHortonworksPartnershipPivotal&HortonworksreferralandGTMPartnershipCo-developmentofPHD3

towardODPCoreHAWQonHortonworks(GemFiretofollow)PHDCertificationsupportAllianceOverviewJointengineeringPivotalHDandHDPbasedonacommoncorePivotalBDScertifiedonHDPJointmarketingandmutualreferralHortonworks(forHadoop)Pivotal(forSQL,Analytics&App-Dev)PivotalHDescalationsupportbyHortonworksCo-foundersofOpenDataPlatformPIVOTALANDHORTONWORKSARESTRONGDRIVERSOFOPENSOURCESOFTWAREPIVOTALANDHORTONWORKSARESTRONGDRIVERSOFOPENSOURCESOFTWAREODP#(EnterpriseHardening)HortonworksDataPlatform(HDP)PivotalHadoopDistributionOtherapps/tools**ex.Analyticsappsandvisualizationtools#O基于BDS的大數(shù)據(jù)平臺架構(gòu)大數(shù)據(jù)平臺參考架構(gòu)DataFeedsTransactionalAppsAnalyticAppsDataStreamPipelineDistributedComputingReal-TimeDataExpertSystems&

MachineLearningAdvancedAnalyticsHDFSData

Lake大數(shù)據(jù)平臺參考架構(gòu)-續(xù)DataFeedsTransactionalAppsAnalyticAppsDataStreamPipelineHDFSData

LakeGemFireHAWQGPDBSpringXDIoT大數(shù)據(jù)架構(gòu)實例INGESTIONJSON/HTTPSTREAMPROCESSINGSpringXDTransformEnrichDATALAKEPivotalHDSinkADVANCEDANALYTICSHAWQREAL-TIMEDATAINSIGHTSGemFireMOBILESERVICESMICROSERVICESPivotalCFDashboardAnalyticsAppSimulatorIoTAPPSRabbitMQPUSH簡單案例介紹大數(shù)據(jù)分析在各個行業(yè)的應(yīng)用更豐富、更全面的客戶管理更個性、更智慧的營銷管理更高效、更準(zhǔn)確的風(fēng)險管理Customer360oSingleViewKeyLifeEvents&NextBestActionStressTest&CreditAssessmentsZionsBank大數(shù)據(jù)分析ZIONS是美國西部的金融服務(wù)企業(yè)，下屬17家銀行，500家分支機(jī)構(gòu)，分布在美國西部、西南部10個州；ZionsBank根據(jù)業(yè)務(wù)系統(tǒng)記錄的客戶消費(fèi)內(nèi)容、消費(fèi)時間、消費(fèi)地點、消費(fèi)方式等信息，并結(jié)合用戶在社交媒體上的言論，勾畫客戶360o立體畫像；設(shè)計產(chǎn)品推薦引擎，利用客戶畫像判斷客戶喜好，當(dāng)前客戶需要的金融產(chǎn)品，進(jìn)行產(chǎn)品推薦在客戶畫像基礎(chǔ)上，設(shè)計了客戶流失模型，分析客戶交易行為，判斷客戶忠誠度，針對流失客戶進(jìn)行挽留在PivotalPlatform上實現(xiàn)KeyLifeEvent營銷場景交易明細(xì)客戶/賬戶數(shù)據(jù)社交媒體GPDB股市行情……外部數(shù)據(jù)業(yè)務(wù)系統(tǒng)GPHD客戶關(guān)系管理平臺數(shù)據(jù)集成處理（MPPSQL）數(shù)據(jù)交換平臺數(shù)據(jù)倉庫平臺大數(shù)據(jù)平臺數(shù)據(jù)交換平臺GPDB客戶管理數(shù)據(jù)集市營銷管理客戶行為分析模型挖掘預(yù)測數(shù)據(jù)挖掘平臺（SAS、R……貸款風(fēng)險預(yù)測模型最佳產(chǎn)品推薦模型分析目標(biāo)客戶確定營銷方案執(zhí)行產(chǎn)品營銷個性化產(chǎn)品推薦MRMPPSQLJPMorgan大數(shù)據(jù)應(yīng)用JPMorgan總部設(shè)于美國紐約市，是一家跨國金融服務(wù)機(jī)構(gòu)，也是美國最大的銀行之一。其業(yè)務(wù)遍及60多個國家，包括投資銀行、金融交易處理、投資管理、商業(yè)金融服務(wù)、個人銀行等。為將欺詐交易帶來的損失降到最低，JPMorgan根據(jù)大數(shù)據(jù)平臺累積的客戶長期交易歷史，建立事件驅(qū)動型的實時欺詐檢測模型，實現(xiàn)下面三類檢測：規(guī)則檢測根據(jù)反欺詐專家的經(jīng)驗，預(yù)先定義判斷規(guī)則，實時監(jiān)測欺詐。欺詐消費(fèi)檢測指利用數(shù)據(jù)挖掘手段從消費(fèi)歷史數(shù)據(jù)庫已經(jīng)收集的欺詐消費(fèi)記錄中挖掘出某些統(tǒng)一的欺詐模式，并利用已有的欺詐模式對新的消費(fèi)行為進(jìn)行分析和判斷。異常消費(fèi)檢測利用基于用戶長期交易歷史建立的正常消費(fèi)模型，實時捕獲新的消費(fèi)行為，判斷是否異常；多數(shù)用戶行為具有漸變性，因此應(yīng)當(dāng)使用新的交易記錄及時更新到正常消費(fèi)模型，以適應(yīng)用戶行為的變化。交易欺詐的實時偵測場景Jane'sdataiscollectedconcerningthedemandAge:28Sex:FemaleOccupation:ITCreditrating:A……FacebookAcct:Jane95SocialHub:Alice,Tom，etc.Colleagues:Jack……Jane360degreesofportraitJane'sdataisfedthrough22componen