數(shù)據(jù)挖掘?qū)д揚ang-ningTan,MichaelStieinbach,andVipinKumar著PearsonEducationLTD.人民郵電出版社2023/6/92數(shù)據(jù)挖掘?qū)д撝饕獏⒖紩鳭iaweiHan,MichelineKamberandJianPeiDataMining:ConceptsandTechniqus(thirdEdition),MonrganKaufmannPublishersInc.,2012范明,孟小峰譯數(shù)據(jù)挖掘:概念與技術(shù)（第二版）機(jī)械工業(yè)出版社,20072023/6/93數(shù)據(jù)挖掘?qū)д?023/6/94數(shù)據(jù)挖掘?qū)д?023/6/95數(shù)據(jù)挖掘?qū)д揓iaweiHan在數(shù)據(jù)挖掘領(lǐng)域做出杰出貢獻(xiàn)的鄭州大學(xué)校友——韓家煒2023/6/96數(shù)據(jù)挖掘?qū)д摰?章緒論英文幻燈片制作：Tan,Steinbach,Kumar中文幻燈片編譯：范明為什么挖掘數(shù)據(jù)？(商業(yè))大量數(shù)據(jù)被收集,存儲在數(shù)據(jù)庫\數(shù)據(jù) 倉庫中Webdata,e-commercepurchasesatdepartment/

grocerystoresBank/CreditCard

transactions計算機(jī)越來越便宜，功能越來越 強(qiáng)大競爭壓力越來越大Providebetter,customizedservicesforanedge(e.g.inCustomerRelationshipManagement)2023/6/98數(shù)據(jù)挖掘?qū)д摓槭裁赐诰驍?shù)據(jù)？(科學(xué))數(shù)據(jù)以極快的速度收集和存儲(GB/hour)remotesensorsonasatellitetelescopesscanningtheskiesmicroarraysgeneratinggeneexpressiondatascientificsimulationsgeneratingterabytes(千兆字節(jié))ofdata傳統(tǒng)的技術(shù)難以處理這些rawdata數(shù)據(jù)挖掘可能幫助科學(xué)家inclassifyingandsegmentingdatainHypothesisFormation2023/6/99數(shù)據(jù)挖掘?qū)д撏诰虼笮蛿?shù)據(jù)集：動機(jī)常常有些信息“隱藏”在數(shù)據(jù)中,并非顯而易見的人分析需要數(shù)周\數(shù)月,才能發(fā)現(xiàn)有用的信息許多數(shù)據(jù)根本未曾分析過TheDataGap2023/6/910數(shù)據(jù)挖掘?qū)д撌裁词菙?shù)據(jù)挖掘許多不同定義本書定義在大型數(shù)據(jù)存儲庫中，自動地發(fā)現(xiàn)有用信息的過程。Exploration&analysis,byautomaticorsemi-automaticmeans,oflargequantitiesofdatainordertodiscovermeaningfulpatternsJiaweiHan的定義從大型數(shù)據(jù)集中提取有趣的(非平凡的,蘊涵的,先前未知的并且是潛在有用的)信息或模式一個類似于JiaweiHan的定義Non-trivialextractionofimplicit,previouslyunknownandpotentiallyusefulinformationfromdata2023/6/911數(shù)據(jù)挖掘?qū)д撌裁?不)是數(shù)據(jù)挖掘

WhatisDataMining?CertainnamesaremoreprevalentincertainUSlocations(O’Brien,O’Rurke,O’Reilly…inBostonarea)Grouptogethersimilardocumentsreturnedbysearchengineaccordingtotheircontext(e.g.Amazonrainforest,A,)

WhatisnotDataMining?Lookupphonenumberinphonedirectory

QueryaWebsearchengineforinformationabout“Amazon”2023/6/912數(shù)據(jù)挖掘?qū)д摂?shù)據(jù)挖掘與KDD數(shù)據(jù)挖掘與知識發(fā)現(xiàn)數(shù)據(jù)挖掘是數(shù)據(jù)庫中知識發(fā)現(xiàn)（knowledgediscoveryindatabase,KDD）不可缺少的一部分KDD是將未加工的數(shù)據(jù)轉(zhuǎn)換為有用信息的整個過程2023/6/913數(shù)據(jù)挖掘?qū)д撘l(fā)數(shù)據(jù)挖掘的挑戰(zhàn)1可伸縮海量數(shù)據(jù)集越來越普遍數(shù)千兆字節(jié)(terabytes)為處理海量數(shù)據(jù)，算法必須是可伸縮的（scalable）可伸縮可能還需要新的數(shù)據(jù)結(jié)構(gòu)，以有效的方式訪問個別記錄例如，當(dāng)要處理的數(shù)據(jù)不能放進(jìn)內(nèi)存時，可能需要非內(nèi)存算法使用抽樣技術(shù)或開發(fā)并行和分布算法也可以提高可伸縮程度2023/6/914數(shù)據(jù)挖掘?qū)д撎魬?zhàn)2高維性具有數(shù)以百計或數(shù)以千計屬性的數(shù)據(jù)集生物信息學(xué)：涉及數(shù)千特征的基因表達(dá)數(shù)據(jù)不同地區(qū)溫度測量：維度（特征數(shù)）的增長正比于測量的次數(shù)為低維數(shù)據(jù)開發(fā)的數(shù)據(jù)分析技術(shù)不能很好地處理高維數(shù)據(jù)某些數(shù)據(jù)分析算法，隨著維度（特征數(shù)）的增加，計算復(fù)雜性迅速增加2023/6/915數(shù)據(jù)挖掘?qū)д撎魬?zhàn)3異種數(shù)據(jù)和復(fù)雜數(shù)據(jù)傳統(tǒng)的數(shù)據(jù)分析方法只處理包含相同類型屬性的數(shù)據(jù)集非傳統(tǒng)的數(shù)據(jù)類型的出現(xiàn)需要能夠處理異種屬性的技術(shù)半結(jié)構(gòu)化文本和超鏈接的Web頁面集具有序列和三維結(jié)構(gòu)的DNA數(shù)據(jù)地球表面不同位置上的時間序列測量值（溫度、氣壓等）的氣象數(shù)據(jù)數(shù)據(jù)中的聯(lián)系如時間和空間的自相關(guān)性、圖的連通性、半結(jié)構(gòu)化文本和XML文檔中元素之間的父子聯(lián)系2023/6/916數(shù)據(jù)挖掘?qū)д撎魬?zhàn)4數(shù)據(jù)的所有權(quán)與分布數(shù)據(jù)地理上分布在屬于多個機(jī)構(gòu)的資源中需要開發(fā)分布式數(shù)據(jù)挖掘技術(shù)分布式數(shù)據(jù)挖掘算法面臨的主要挑戰(zhàn)包括(1)如何降低執(zhí)行分布式計算所需的通信量？(2)如何有效地統(tǒng)一從多個資源得到的數(shù)據(jù)挖掘結(jié)果？(3)如何處理數(shù)據(jù)安全性問題？2023/6/917數(shù)據(jù)挖掘?qū)д撎魬?zhàn)5非傳統(tǒng)的分析傳統(tǒng)的統(tǒng)計學(xué)方法：假設(shè)-檢驗?zāi)Ｊ教岢鲆环N假設(shè)，設(shè)計實驗來收集數(shù)據(jù)，然后針對假設(shè)分析數(shù)據(jù)當(dāng)前的數(shù)據(jù)分析任務(wù)常常需要產(chǎn)生和評估數(shù)以千計的假設(shè)希望自動地產(chǎn)生和評估假設(shè)導(dǎo)致了一些數(shù)據(jù)挖掘技術(shù)的開發(fā)數(shù)據(jù)挖掘所分析的數(shù)據(jù)集通常不是精心設(shè)計的實驗的結(jié)果代表數(shù)據(jù)的時機(jī)性樣本（opportunisticsample）而不是隨機(jī)樣本（randomsample）數(shù)據(jù)集常常涉及非傳統(tǒng)的數(shù)據(jù)類型和數(shù)據(jù)分布2023/6/918數(shù)據(jù)挖掘?qū)д摂?shù)據(jù)挖掘的起源數(shù)據(jù)挖掘是多學(xué)科交叉領(lǐng)域利用了來自如下一些領(lǐng)域的思想：統(tǒng)計學(xué)的抽樣、估計和假設(shè)檢驗人工智能、模式識別和機(jī)器學(xué)習(xí)的搜索算法、建模技術(shù)和學(xué)習(xí)理論數(shù)據(jù)庫系統(tǒng)提供有效的存儲、索引和查詢處理支持分布式技術(shù)也能幫助處理海量數(shù)據(jù)最優(yōu)化、進(jìn)化計算、信息論、信號處理、可視化和信息檢索MachineLearning/Pattern

RecognitionStatistics/

AIDataMiningDatabasesystems2023/6/919數(shù)據(jù)挖掘?qū)д?/p>

數(shù)據(jù)挖掘任務(wù)預(yù)測vs.描述預(yù)測(Prediction)根據(jù)其他屬性的值，預(yù)測特定屬性的值描述(Description)導(dǎo)出概括數(shù)據(jù)中潛在聯(lián)系的模式2023/6/920數(shù)據(jù)挖掘?qū)д摂?shù)據(jù)挖掘任務(wù)分類（Classification）[Predictive]回歸（Regression）[Predictive]關(guān)聯(lián)規(guī)則發(fā)現(xiàn)（AssociationRuleDiscovery）[Descriptive]序列模式發(fā)現(xiàn)（SequentialPatternDiscovery）[Descriptive]聚類（Clustering）[Descriptive]異常/偏差檢測（Anomaly/DeviationDetection）[Predictive]2023/6/921數(shù)據(jù)挖掘?qū)д摲诸?定義給定一批記錄----訓(xùn)練集(trainingset)Eachrecordcontainsasetofattributes,oneoftheattributesistheclasslabel(類標(biāo)號).任務(wù):建立一個模型(model)類標(biāo)號屬性是其他屬性值的函數(shù)目標(biāo):previouslyunseenrecordsshouldbeassignedaclassasaccuratelyaspossible.Atestset(檢驗集)isusedtodeterminetheaccuracyofthemodel.Usually,thegivendatasetisdividedintotrainingandtestsets,withtrainingsetusedtobuildthemodelandtestsetusedtovalidateit2023/6/922數(shù)據(jù)挖掘?qū)д摲诸?例子categoricalcategoricalcontinuousclassTestSetTrainingSetModelLearnClassifier2023/6/923數(shù)據(jù)挖掘?qū)д摲诸?應(yīng)用1DirectMarketingGoal:Reducecostofmailingbytargetingasetofconsumerslikelytobuyanewcell-phoneproduct.Approach:Usethedataforasimilarproductintroducedbefore.Weknowwhichcustomersdecidedtobuyandwhichdecidedotherwise.This{buy,don’tbuy}decisionformstheclassattribute.Collectvariousdemographic,lifestyle,andcompany-interactionrelatedinformationaboutallsuchcustomers.Typeofbusiness,wheretheystay,howmuchtheyearn,etc.Usethisinformationasinputattributestolearnaclassifiermodel.2023/6/924數(shù)據(jù)挖掘?qū)д摲诸?應(yīng)用2FraudDetectionGoal:Predictfraudulentcasesincreditcardtransactions.Approach:Usecreditcardtransactionsandtheinformationonitsaccount-holderasattributes.Whendoesacustomerbuy,whatdoeshebuy,howoftenhepaysontime,etcLabelpasttransactionsasfraudorfairtransactions.Thisformstheclassattribute.Learnamodelfortheclassofthetransactions.Usethismodeltodetectfraudbyobservingcreditcardtransactionsonanaccount.2023/6/925數(shù)據(jù)挖掘?qū)д摲诸?應(yīng)用3SkySurveyCatalogingGoal:Topredictclass(starorgalaxy)ofskyobjects,especiallyvisuallyfaintones,basedonthetelescopicsurveyimages(fromPalomarObservatory).3000imageswith23,040x23,040pixelsperimage.Approach:Segmenttheimage.Measureimageattributes(features)-40ofthemperobject.Modeltheclassbasedonthesefeatures.SuccessStory:Couldfind16newhighred-shiftquasars,someofthefarthestobjectsthataredifficulttofind!2023/6/926數(shù)據(jù)挖掘?qū)д摲诸?應(yīng)用3Attributes:Imagefeatures,Characteristicsoflightwavesreceived,etc.EarlyIntermediateLateDataSize:72millionstars,20milliongalaxiesObjectCatalog:9GBImageDatabase:150GB

Class:StagesofFormationCourtesy:2023/6/927數(shù)據(jù)挖掘?qū)д摶貧w回歸(regression)Predictavalueofagivencontinuousvaluedvariablebasedonthevaluesofothervariables,assumingalinearornonlinearmodelofdependency.Greatlystudiedinstatistics,neuralnetworkfields.Examples:Predictingsalesamountsofnewproductbasedonadvertisingexpenditure.Predictingwindvelocitiesasafunctionoftemperature,humidity,airpressure,etc.Timeseriespredictionofstockmarketindices2023/6/928數(shù)據(jù)挖掘?qū)д撽P(guān)聯(lián)規(guī)則:定義關(guān)聯(lián)規(guī)則(associationrule)Givenasetofrecordseachofwhichcontainsomenumberofitemsfromagivencollection;Producedependencyruleswhichwillpredictoccurrenceofitemsbasedonoccurrencesofotheritems.RulesDiscovered:

{Milk}-->{Coke}{Diaper,Milk}-->{Beer}2023/6/929數(shù)據(jù)挖掘?qū)д撽P(guān)聯(lián)規(guī)則:應(yīng)用1MarketingandSalesPromotion:Lettherulediscoveredbe

{Bagels,…

}

-->

{PotatoChips}PotatoChips

asconsequent=>Canbeusedtodeterminewhatshouldbedonetoboostitssales.Bagelsintheantecedent=>Canbeusedtoseewhichproductswouldbeaffectedifthestorediscontinuessellingbagels.Bagelsinantecedent

and

Potatochipsinconsequent

=>CanbeusedtoseewhatproductsshouldbesoldwithBagelstopromotesaleofPotatochips!2023/6/930數(shù)據(jù)挖掘?qū)д撽P(guān)聯(lián)規(guī)則:應(yīng)用2Supermarketshelfmanagement.Goal:Toidentifyitemsthatareboughttogetherbysufficientlymanycustomers.Approach:Processthepoint-of-saledatacollectedwithbarcodescannerstofinddependenciesamongitems.Aclassicrule--Ifacustomerbuysdiaperandmilk,thenheisverylikelytobuybeer.So,don’tbesurprisedifyoufindsix-packsstackednexttodiapers!2023/6/931數(shù)據(jù)挖掘?qū)д摼垲?定義Givenasetofdatapoints,eachhavingasetofattributes,andasimilaritymeasureamongthem,findclusterssuchthatDatapointsinoneclusteraremoresimilartooneanother.Datapointsinseparateclustersarelesssimilartooneanother.SimilarityMeasures:EuclideanDistanceifattributesarecontinuous.OtherProblem-specificMeasuresIntraclusterdistancesareminimizedInterclusterdistancesaremaximized2023/6/932數(shù)據(jù)挖掘?qū)д摼垲?應(yīng)用1MarketSegmentation:Goal:subdivideamarketintodistinctsubsetsofcustomerswhereanysubsetmayconceivablybeselectedasamarkettargettobereachedwithadistinctmarketingmix.Approach:Collectdifferentattributesofcustomersbasedontheirgeographicalandlifestylerelatedinformation.Findclustersofsimilarcustomers.Measuretheclusteringqualitybyobservingbuyingpatternsofcustomersinsameclustervs.thosefromdifferentclusters.

2023/6/933數(shù)據(jù)挖掘?qū)д摼垲?應(yīng)用2DocumentClustering:Goal:Tofindgroupsofdocumentsthataresimilartoeachotherbasedontheimportanttermsappearinginthem.Approach:Toidentifyfrequentlyoccurringtermsineachdocument.Formasimilaritymeasurebasedonthefrequenciesofdifferentterms.Useittocluster.Gain:InformationRetrievalcanutilizetheclusterstorelateanewdocumentorsearchtermtoclustereddocuments2023/6/934數(shù)據(jù)挖掘?qū)д撐臋n聚類:例ClusteringPoints:3204ArticlesofLosAngelesTimes.SimilarityMeasure:Howmanywordsarecommoninthesedocuments(aftersomewordfiltering).2023/6/935數(shù)據(jù)挖掘?qū)д摦惓z測任務(wù)：識別其特征顯著不同于其他數(shù)據(jù)的觀測值這樣的觀測值稱為異常點（anomaly）或離群點（outlier）發(fā)現(xiàn)真正的異常點，而避免錯誤地將正常的對象標(biāo)注為異常點應(yīng)用信用卡欺詐檢測網(wǎng)絡(luò)入侵檢測2023/6/936數(shù)據(jù)挖掘?qū)д摂?shù)據(jù)挖掘的應(yīng)用數(shù)據(jù)挖掘的應(yīng)用數(shù)據(jù)庫分析和決策支持市場分析和管理針對銷售(targetmarketing),顧客關(guān)系管理,購物籃分析,交叉銷售(crossselling),市場分割(marketsegmentation)風(fēng)險分析與管理預(yù)測,顧客關(guān)系,改進(jìn)保險,質(zhì)量控制,競爭能力分析欺騙檢測與管理其它應(yīng)用文本挖掘(新聞組,email,文檔資料)流數(shù)據(jù)挖掘(Streamdatamining)Web挖掘.DNA數(shù)據(jù)分析2023/6/938數(shù)據(jù)挖掘?qū)д撌袌龇治雠c管理(1)用于分析的數(shù)據(jù)源在哪?信用卡交易,會員卡,打折優(yōu)惠卷,顧客投訴電話,(公共)生活時尚研究針對銷售(Targetmarketing)找出顧客群,他們具有相同特征:興趣,收入水平,消費習(xí)慣,等.確定顧客隨時間變化的購買模式個人帳號到聯(lián)合帳號的轉(zhuǎn)變:結(jié)婚,等.交叉銷售分析(Cross-marketanalysis)產(chǎn)品銷售之間的關(guān)聯(lián)/相關(guān)基于關(guān)聯(lián)信息的預(yù)測2023/6/939數(shù)據(jù)挖掘?qū)д撌袌龇治雠c管理(2)顧客分類(Customerprofiling)數(shù)據(jù)挖掘能夠告訴我們什么樣的顧客買什么產(chǎn)品(聚類或分類)識別顧客需求對不同的顧客識別最好的產(chǎn)品使用預(yù)測發(fā)現(xiàn)什么因素影響新顧客提供匯總信息各種多維匯總報告統(tǒng)計的匯總信息(數(shù)據(jù)的中心趨勢和方差)2023/6/940數(shù)據(jù)挖掘?qū)д摲ㄈ朔治龊惋L(fēng)險管理財經(jīng)規(guī)劃和資產(chǎn)評估現(xiàn)金流分析和預(yù)測臨時提出的資產(chǎn)評估交叉組合(cross-sectional)和時間序列分析(金融比率(financial-ratio),趨勢分析,等.)資源規(guī)劃:資源與開銷的匯總與比較競爭:管理競爭者和市場指導(dǎo)對顧客分類和基于類的定價在高度競爭的市場調(diào)整價格策略2023/6/941數(shù)據(jù)挖掘?qū)д撈垓_檢測和管理(1)應(yīng)用廣泛用于健康照料,零售,信用卡服務(wù),電訊(電話卡欺騙),等.方法使用歷史數(shù)據(jù)建立欺騙行為模型,使用數(shù)據(jù)挖掘幫助識別類似的實例例汽車保險:檢測這樣的人,他/她假造事故騙取保險賠償洗錢:檢測可疑的金錢交易(USTreasury'sFinancialCrimesEnforcementNetwork)醫(yī)療保險:檢測職業(yè)病患者,醫(yī)生和介紹人圈2023/6/942數(shù)據(jù)挖掘?qū)д撈垓_檢測和管理(2)檢測不適當(dāng)?shù)尼t(yī)療處置澳大利亞健康保險會(AustralianHealthInsuranceCommission)發(fā)現(xiàn)許多全面的檢查是請求做的,而不是實際需要的(每年節(jié)省100萬澳元).檢測電話欺騙電話呼叫模式:通話距離,通話時間,每天或每周通話次數(shù).分析偏離期望的模式.英國電訊(BritishTelecom)識別頻繁內(nèi)部通話的呼叫者的離散群,特別是移動電話,超過數(shù)百萬美元的欺騙.零售分析家估計,38%的零售業(yè)萎縮是由于不忠誠的雇員造成的.2023/6/943數(shù)據(jù)挖掘?qū)д撈渌鼞?yīng)用運動IBMAdvancedScout分析NBA的統(tǒng)計數(shù)據(jù)(阻擋投籃,助攻,和犯規(guī))獲得了對紐約小牛隊(NewYorkKnicks)和邁艾米熱隊(MiamiHeat)的競爭優(yōu)勢天文借助于數(shù)據(jù)挖掘的幫助,JPL和PalomarObservatory發(fā)現(xiàn)了22顆類星體(quasars)InternetWebSurf-AidIBMSurf-Aid將數(shù)據(jù)挖掘算法用于有關(guān)交易的頁面的Web訪問日志,以發(fā)現(xiàn)顧客喜愛的頁面,分析Web銷售的效果,改進(jìn)Web站點的組織,等.2023/6/944數(shù)據(jù)挖掘?qū)д摂?shù)據(jù)挖掘界簡史1989IJCAIWorkshoponKnowledgeDiscoveryinDatabases(Piatetsky-Shapiro)KnowledgeDiscoveryinDatabases(G.Piatetsky-ShapiroandW.Frawley,1991)1991-1994WorkshopsonKnowledgeDiscoveryinDatabasesAdvancesinKnowledgeDiscoveryandDataMining(U.Fayyad,G.Piatetsky-Shapiro,P.Smyth,