簡(jiǎn)介：張粵磊(簡(jiǎn)介：張粵磊(Jackson)@me：vzyueleiQQ:416988515飛谷云()創(chuàng)始人(2014-2016.3)平安付大數(shù)據(jù)平臺(tái)架構(gòu)師(2012-2014)外匯交易中心ETL項(xiàng)目開(kāi)發(fā)經(jīng)理(2010-2012)HPTRAM項(xiàng)目ETL開(kāi)發(fā)組長(zhǎng)(2005-2010)DBA10余年一線(xiàn)數(shù)據(jù)業(yè)務(wù)(制造，咨詢(xún)服務(wù)，互聯(lián)網(wǎng)金融)及數(shù)據(jù)處理技術(shù)實(shí)踐經(jīng)驗(yàn)工具:數(shù)數(shù)據(jù)處理的哪些事2.大數(shù)據(jù)環(huán)境下對(duì)于公共數(shù)據(jù)及行為數(shù)據(jù)的數(shù)據(jù)3.由傳統(tǒng)數(shù)據(jù)倉(cāng)庫(kù)到大數(shù)據(jù)數(shù)據(jù)倉(cāng)庫(kù)的數(shù)據(jù)處理傳統(tǒng)數(shù)據(jù)倉(cāng)庫(kù)的數(shù)據(jù)處理技術(shù)傳統(tǒng)數(shù)據(jù)倉(cāng)庫(kù)的數(shù)據(jù)處理傳統(tǒng)數(shù)據(jù)倉(cāng)庫(kù)的數(shù)據(jù)處理技術(shù)是什么？該過(guò)程涵蓋傳統(tǒng)數(shù)據(jù)倉(cāng)庫(kù)的標(biāo)準(zhǔn)流程和數(shù)據(jù)處理方法和實(shí)踐同樣適用于數(shù)據(jù)平Portal傳統(tǒng)數(shù)據(jù)倉(cāng)庫(kù)的數(shù)據(jù)處理技術(shù)-概念定義PortalClientEmployees/EmployeeIdentityIdentityandAccessManagementReportingReportingRepositoriesVerticalReportingRepositoriesGDCPMCapacityManagement(Americas/EMEA)UpliftServicesReportingFunctionsReportCreationDataCollection,Extraction,andTransportationCRDWCRDWAvM/CpM/SLM(RegionalDeployment)BaseServicesInteractiveReportHandlingDataanalysisandForecastReportManagementAd-hocReportingServiceManagement,ReportingDWB1DMB2DMSLMDWExternalExternalandInternalDataSourcesServiceLinesCOTSComponentExternalComponentDevelopedComponentDevelopAppsExternalAppsCOTSAppsESM傳統(tǒng)數(shù)據(jù)倉(cāng)庫(kù)的數(shù)據(jù)處理技術(shù)-業(yè)務(wù)定義ironmenttionandDeliveryAccessAccessoricReportsisrecastpositoriesOtherpositoriestAvM/CpM/SLMoreeveloreevel)ReportingDatawarehouse(DW)yManagement(Americas/EMEA)aExtractionTransfer(DRS)menttionstionsrationAssetData:InfrastructuregegeftwareWebChatWeb/WebChatity(Antivirusity(Antivirus/PatchlianceseDeskephonyonandransportationjectsofManagement(DataSources)utingISomeromersfactiony傳統(tǒng)數(shù)據(jù)倉(cāng)庫(kù)的數(shù)據(jù)處理技術(shù)-邏輯r,,,,XLSicReportsConfig,RMTreeStructure,etc.On-boardingTools(*)ionCapacityUpliftServicesironmentSupportOperationsironmentManagementManagementandDeliverytseInteractiveReportscAd-hocalysisCreate/Manage/DeliverReportsReportingDataWarehousefilees0DataModel1Erwin2thenticationdataentityentityccessMgmtSLMReportsDataforanalysisnceDatacoreAdapterModelmTools(*)Users,roles,etc.ionscityandPerformanceAvM/CpM/SLMEMITools(OVPI,c33partytoolsOVR/HPReportersualizerCreate/UpdateFacts/DimensionsDataExtraction,Transport,Transform,LoadInformaticafilenceMetricsReportingandAnalyticsExtract/ReceiveSourceDataWorkflowWorkflowSOM/WorkflowOVSCSOM/WorkflowOVSC/SMAssetManagementOVAC/HPAMaItemsConfig.ManagementESL/JetWebChaticeDesklACDsystemscificiceLinesngTools(*)espsservicesTools(*)espsservicesetcObjectsofManagementVLAN2DBTierDataSourcesTCPIP/PortsVLAN2DBTierDataSourcesTCPIP/PortsMMITCPIP/PortsAppWebTierVLAN1AppTierOraclePort:UDPUDPTCPTCPI**ntranetVLAN3AJP1.3Port:8009ClientcertificateHTTPSPort:443/JDBC1521137138139445Oracle/JDBCPort:1521HTTPS/Port:443LDAP/Port:389esTCPIP/PortsHTTP8080AppOracle/JDBCPort:1521XXTrustedNetworkSiteMinderPorts:44001-44003sAuthorizationServer§Platform–HPUXAuthorizationServeryPolicyeLDAPLegacyOMCNetesCustomerNetworkersHTTPSPort:443HPSiteminderWebAgentISAPIJakartaPluginLoginWebPageNon-HPSiteminderWebagentFunction–LoginHTTPServerSoftware–MSIISPlatform–WindowsSiteminderPorts:44001-44003傳統(tǒng)數(shù)據(jù)倉(cāng)庫(kù)的數(shù)據(jù)處理技術(shù)-物理定義`AppAppETL:CreatesummarizeviewtobusinessrequirementsETL:ExtractdatafromSourceSystemETL:CreatesummarizeviewtobusinessrequirementsETL:ExtractdatafromSourceSystemngpriortoloadingintowarehouseLevel1LevelceelSummarydataforreportingETLETL:GenerateSurrogateKeysCreatederivedcolumnstosupportbusinessrequirements.Createadditionaltablestosupportvariousactivitytype.xtractfromthesourcengMayHPRestricted傳統(tǒng)數(shù)據(jù)倉(cāng)庫(kù)的數(shù)據(jù)處理技術(shù)-設(shè)計(jì)定義數(shù)據(jù)倉(cāng)庫(kù)DW數(shù)據(jù)倉(cāng)庫(kù)DW傳統(tǒng)數(shù)據(jù)倉(cāng)庫(kù)的數(shù)據(jù)處理技術(shù)-開(kāi)發(fā)實(shí)現(xiàn)數(shù)據(jù)處理工具數(shù)據(jù)處理工具傳統(tǒng)數(shù)據(jù)倉(cāng)庫(kù)的數(shù)據(jù)處理技術(shù)-開(kāi)發(fā)實(shí)現(xiàn)傳統(tǒng)數(shù)據(jù)倉(cāng)庫(kù)的數(shù)據(jù)處理技術(shù)不可用?無(wú)論選擇任何一種RDBMS，都無(wú)法涵蓋大量的非結(jié)構(gòu)化業(yè)務(wù)數(shù)據(jù)哪些數(shù)據(jù)未按統(tǒng)一格式存儲(chǔ)?基于RDBMS的數(shù)據(jù)存儲(chǔ)并不能真實(shí)反映業(yè)務(wù)數(shù)據(jù)本源格式，文本視頻，郵件在DB中的存儲(chǔ)哪些數(shù)據(jù)的值在信息含義上是沖突的?哪些數(shù)據(jù)和信息是不正確的，或者數(shù)據(jù)是超期的?不同RDBMS對(duì)數(shù)據(jù)類(lèi)型的定義精度各有區(qū)別哪些數(shù)據(jù)是重復(fù)數(shù)據(jù)或者數(shù)據(jù)的哪些屬性是重復(fù)的哪些關(guān)聯(lián)的數(shù)據(jù)缺失關(guān)聯(lián)性或者未建立索引準(zhǔn)確性傳統(tǒng)數(shù)據(jù)倉(cāng)庫(kù)的數(shù)據(jù)處理技術(shù)準(zhǔn)確性哪些數(shù)據(jù)和信息是不正確的，或者數(shù)據(jù)是超期的?不同RDBMS對(duì)數(shù)據(jù)類(lèi)型的定義精度各有區(qū)別當(dāng)源系統(tǒng)與目標(biāo)系統(tǒng)屬于不同RDBMS或字符集等情時(shí)分秒；oracle的Integer數(shù)據(jù)類(lèi)型是8字節(jié)38位精度，db2的Integer數(shù)據(jù)類(lèi)型是4字節(jié)10位精度等等。數(shù)數(shù)據(jù)處理的哪些事1.傳統(tǒng)數(shù)據(jù)倉(cāng)庫(kù)的數(shù)據(jù)處理技術(shù)及思考2.大數(shù)據(jù)環(huán)境下對(duì)于公共數(shù)據(jù)及行為數(shù)據(jù)的數(shù)3.由傳統(tǒng)數(shù)據(jù)倉(cāng)庫(kù)到大數(shù)據(jù)數(shù)據(jù)倉(cāng)庫(kù)的數(shù)據(jù)處理大數(shù)據(jù)處理及模型開(kāi)發(fā) load用E load用E Hql:清洗標(biāo)準(zhǔn)化數(shù)據(jù)的數(shù)據(jù)處理技術(shù)數(shù)據(jù)的數(shù)據(jù)處理技術(shù)數(shù)據(jù)的數(shù)據(jù)處理技術(shù)按數(shù)據(jù)特征分類(lèi)定義：目前其實(shí)專(zhuān)指的是關(guān)系模型數(shù)據(jù)，即以關(guān)系型數(shù)據(jù)庫(kù)表形式管理的數(shù)據(jù)。絕大多存放。簡(jiǎn)析：雖然從專(zhuān)業(yè)角度講，結(jié)構(gòu)化就是關(guān)系模型的說(shuō)法并不準(zhǔn)確。但針對(duì)目前業(yè)內(nèi)現(xiàn)狀，還是將其定義為關(guān)系模型數(shù)據(jù)為最為妥當(dāng)，因?yàn)樗逦鴾?zhǔn)確地代表了我們傳統(tǒng)上最熟定義：半結(jié)構(gòu)化與非結(jié)構(gòu)化常常一同被提及，兩者其實(shí)專(zhuān)指所有其他“非”結(jié)構(gòu)化數(shù)據(jù)。但如果想更加清晰地描述，可以將“半結(jié)構(gòu)化數(shù)據(jù)”定義為：那些非關(guān)系模型的、有基簡(jiǎn)析：此部分?jǐn)?shù)據(jù)可以用程序化格式解析處理，公共數(shù)據(jù)，行為數(shù)據(jù)多以此種格式定義：除去結(jié)構(gòu)化與半結(jié)構(gòu)化的所有數(shù)據(jù)，即沒(méi)有固定結(jié)構(gòu)模式的數(shù)據(jù)，例如WORD、簡(jiǎn)析：區(qū)分半結(jié)構(gòu)化與非結(jié)構(gòu)化數(shù)據(jù)的意義在于，目前在企業(yè)內(nèi)對(duì)兩者的處理方法(包括存儲(chǔ)、訪(fǎng)問(wèn)與分析)是不同的。非結(jié)構(gòu)化數(shù)據(jù)大多采用內(nèi)容管理的方法，展示上需要公共數(shù)據(jù)處理的注意點(diǎn)接口定義加入接口規(guī)范變更版本及內(nèi)容到數(shù)據(jù)字段中對(duì)于網(wǎng)站抓取或接口調(diào)用的變化版本記錄有利于對(duì)數(shù)據(jù)準(zhǔn)確和完整性的可追溯落地后的文件時(shí)間和成功標(biāo)志信息同樣參與數(shù)據(jù)處理合并到數(shù)據(jù)落地層(LEVEL0)后數(shù)據(jù)的落地時(shí)間和數(shù)據(jù)大小行數(shù)記錄到數(shù)據(jù)監(jiān)控表中在數(shù)據(jù)倉(cāng)庫(kù)處理和分析展示中添加數(shù)據(jù)處理的可追溯信息對(duì)于核心指標(biāo)及對(duì)應(yīng)元數(shù)據(jù)顯示和監(jiān)控，確保對(duì)于數(shù)據(jù)的理解和定義全局一致行為數(shù)據(jù)處理的注意點(diǎn)埋點(diǎn)數(shù)據(jù)一定要符合業(yè)務(wù)數(shù)據(jù)信息流才能保證數(shù)據(jù)處理的完整性和確保數(shù)據(jù)的業(yè)務(wù)可用性行為數(shù)據(jù)的標(biāo)識(shí)健(UID，DID)要與其它數(shù)據(jù)源統(tǒng)一關(guān)聯(lián)期，確保數(shù)據(jù)的一致性和關(guān)聯(lián)性。行為數(shù)據(jù)的元數(shù)據(jù)信息盡可能從源頭以字段化方式植入數(shù)據(jù)處理的數(shù)據(jù)文件中位詳細(xì)頁(yè)①簡(jiǎn)歷信息帖子地址DB職位信息位詳細(xì)頁(yè)①簡(jiǎn)歷信息帖子地址DB職位信息定時(shí)生成維度統(tǒng)計(jì)去重/計(jì)算/映射④②理元③計(jì)任務(wù)公共數(shù)據(jù)及行為數(shù)據(jù)的數(shù)據(jù)處理技術(shù)案例圖檢索頁(yè)飛谷調(diào)度系統(tǒng)飛谷監(jiān)控系統(tǒng)數(shù)數(shù)據(jù)處理的哪些事1.傳統(tǒng)數(shù)據(jù)倉(cāng)庫(kù)的數(shù)據(jù)處理技術(shù)及思考2.大數(shù)據(jù)環(huán)境下對(duì)于公共數(shù)據(jù)及行為數(shù)據(jù)的數(shù)據(jù)3.由傳統(tǒng)數(shù)據(jù)倉(cāng)庫(kù)到大數(shù)據(jù)數(shù)據(jù)倉(cāng)庫(kù)的數(shù)據(jù)處敏1數(shù)據(jù)處理的重點(diǎn)是全樣本數(shù)據(jù)的基于業(yè)務(wù)時(shí)間周期同步落地2基于RDBMS的敏感數(shù)據(jù)在進(jìn)入大數(shù)據(jù)平臺(tái)后進(jìn)行脫敏處理敏1數(shù)據(jù)處理的重點(diǎn)是全樣本數(shù)據(jù)的基于業(yè)務(wù)時(shí)間周期同步落地2基于RDBMS的敏感數(shù)據(jù)在進(jìn)入大數(shù)據(jù)平臺(tái)后進(jìn)行脫敏處理，確