第四章文本自動(dòng)分類(lèi)技術(shù)1知識(shí)的結(jié)構(gòu)問(wèn)題和知識(shí)是孿生結(jié)構(gòu)本身也是知分類(lèi)體十進(jìn)制系統(tǒng)（分類(lèi)國(guó)會(huì)館 AMS（數(shù)學(xué)會(huì)）的數(shù)學(xué)知識(shí)體系專(zhuān)利內(nèi)容的類(lèi)別體WebYahoo以前的主 對(duì)象：可以是 事先定好的分類(lèi)體系：可能有 最適合：判斷4分類(lèi)體系一般人與分類(lèi)系統(tǒng)可以是分類(lèi)(multi- ?and聯(lián)賽體育MEDLINE(NationalLibraryofMedicine)$2million/yearformanualindexingofjournal難以保證一致性和準(zhǔn)確性(40%準(zhǔn)確率相對(duì)高(準(zhǔn)確率可達(dá)85來(lái)源于真實(shí)文本，可信度 TextCategorization用數(shù)學(xué)表示如下f:A

其中，A為待分類(lèi)的文本集B為分類(lèi)體系中的類(lèi)別集門(mén)戶(hù)（網(wǎng)頁(yè)/信息部門(mén)（處理減小人工分類(lèi)的 定義分類(lèi) 將預(yù)先分類(lèi)過(guò)的文檔作為 12345專(zhuān)家系統(tǒng)(late人工定義規(guī) 專(zhuān)家系統(tǒng)（人工定義規(guī)則 太難（最初看起來(lái)容易 一致性問(wèn)題(規(guī)則集很人口局十年人口統(tǒng)計(jì)資料的分析（2200萬(wàn)項(xiàng)資料232504$15millioniffullydoneby人工定義ExpertSystemDevelopmenttime: -monthspeople,8Accuracy=基于機(jī)器學(xué)習(xí)的(Creecy’92:1-Developmenttime: -Accuracy=自動(dòng)分類(lèi)Given:Collectionofexamplenewsstoriesalreadylabeledwithacategory(topic).Task:Predictcategoryfornewsstoriesnotyetlabeled.Forourexamplewe’llonlygettoseetheheadline（標(biāo)題）ofthenewsstory.We’llrepresentcategoriesusingcolors.(Allexampleswiththesamecolorbelongtothesame企業(yè)企業(yè)個(gè)人事事FiveBonusShareUnit事FiveBonusShareUnitLosesSixMlnCrownsinSaysOpenWouldHitSays1986SaidItSetsBonusIssue JoinsBidforSanItaly’sLatoReportNoDefendsU.S.FarmControl nsInterim nsInterimProfitExceedU.S.MandatoryFarmControlItaly’sLa能給一個(gè)賦予什么顏色??AmatilProposesAmatilProposesUnitLosesSixMlnCrownsinJapanMinistrySaysOpenFarmSays1986JardineMathesonSaidItSetsTwo-for-FiveBonusIssueRecing“B”P(pán)rofitExceedProfitExceedU.S.FarmControl nsItaly’sLaLoanSetAmatilProposesAmatilProposesUnitLosesSixMlnCrownsinJapanMinistrySaysOpenFarmSays1986JardineMathesonSaidItSetsTwo-for-FiveBonusIssueRecing“B”P(pán)rofitExceedProfitExceedU.S.FarmControl nsItaly’sLa得到分類(lèi)：事LoanSetAmatilProposesAmatilProposesUnitLosesSixMlnCrownsinJapanMinistrySaysOpenFarmSays1986JardineMathesonSaidItSetsTwo-for-FiveBonusIssueRecing“B”「率」（R,F 111 F Pmissrate=1-fallout=b/(b+d)=falsealarmBreakEvenPointBEPp=rinterpolated11pointaverageprecision所有類(lèi)的總體

Fi

11 宏平

MacroF

m1mim1m

PiPMicroF

(niFiFiimm布爾模向量空間概率模特征抽取(feature禁用詞(stopwords)去除、詞根還原(TF數(shù)據(jù)：去掉噪聲文檔或文檔內(nèi)數(shù)特征選擇(Feature向量空間模型(VectorSpace 每個(gè)文檔dj可以用標(biāo)引項(xiàng)向量 權(quán)重計(jì)算，N個(gè)訓(xùn)練 檔AM*N= 相似度較

D1=2T1+3T2+Q=0T1+0T2+2D2=3T1+7T2+ 相關(guān)詞cluster，wordclusterN-gram，NDavidLewis等一致地認(rèn)為：(英文分類(lèi)中)使布爾權(quán)重(booleanaij=1(TFij>0)orTF:TFij*log(N/TFij*log(N/ *log(N/DF2kkTFC:對(duì)上面進(jìn)行歸一 1.0)*log(N/DFkk 1.0)*log(N/DFkk 避免過(guò)擬合（overfitting），Over-fitting希望模型的表現(xiàn)對(duì)訓(xùn)練集和未知文檔基本一致通過(guò)降維，計(jì)算時(shí)間和空基本信念：除那些stopwords外，還有許多詞文檔頻率法（DF,信息增益法（information互信息法（mutualThe 基于DF的啟發(fā)式Term的DF稀有詞項(xiàng)的全局不代表性（這是一種adhoc方法，不依據(jù)什么最容易實(shí)現(xiàn)，可擴(kuò)展性 Robertson&SparckJonesRSJ(t,c)

cj中出現(xiàn)t的概率logP(t|cj P(t|c TSV(t,c)r*

P(t|cj

r為出現(xiàn)t的c類(lèi)文檔個(gè)j P(t|c j

logP(t|cj)log(1P(t|cj))log(1P(t|cj))logP(t|cjContingencytablep={p1,p2,…,Entropy(t)PilogiPici/Pi(AB)/nABCEntropy(t)P(ci|t)logP(ci|iterm類(lèi)別分布的CE(t)

P(c|t)

| P(c 信息增益(InformationGainGain(t)Gain(t)Entropy(S)ExpectedEntropy(StMi1P(c)logP(cii[P(t){MP(c|t)logP(c|iiP(tMP(c|t)logP(c|tii隨量X,Y相關(guān)程度的一種度量MI(X,Y)P(x,y)logP(x, P(x)P(其中P(x,y)是變量取值（x,y）不情況下，用t,c表示這兩個(gè)隨關(guān)心的P(t)P(c)P(t,c)都可以通過(guò)統(tǒng)互信息(MutualctctABCDI(t,c)logP(t logP(t|c)P(t)P(c)

A(AC)(AmIAVG(t)P(ci)I(t,cim (t) P(c)I(t,c MI(t,C) 隨

（卡方源于統(tǒng)計(jì)學(xué)的卡方分布(chi-從（類(lèi)，詞項(xiàng)）chi-2（卡方若AD<BCctABCD2(ctABCD

N(AD(AC)(BD)(AB)(C (t)

P(c)2(t,c2

i{2(t,citerms(t)= mIG:InformationmG(t)

i1

)log

(cipmP(cpmP(c|t)logP(c|t)P(t

m m

P|tP|t|t

I(t,c)2(t,c)

A(AC)(AN(ADCB)(AC)(BD)(AB)(C4決策樹(shù)（DecisionKNN算法(K-Nearest貝葉斯網(wǎng)絡(luò)（Bayes神經(jīng)網(wǎng)絡(luò)（NeuralIndependencyIndependencyM-creditcredit

yes yesCARTC4.5ID3發(fā)展而CHAIDAttributeSelectionMeasure:InformationGain(ID3/C4.5)選擇信息增益最ScontainssituplesofclassCifori={1,…,m}informationmeasuresinforequiredtoclassifyanyarbitrarytupleI(

m 1,s2,...,sm)

log AttributeSelectionMeasure:InformationGain(ID3/C4.5)entropyofattributeAwithvE(A)j

s1j...s

I(s1j,...,smjinformationgainedbybranchingonattributeAGain(A)I(s1,s2,...,sm)選擇信息增益最大的屬性作為判定的分

iCxij iCxij iCxij nnCw'

(di

)

xi

wcj221-Nearest回顧前面的LoanSetDidLoanSetJardineMathesonSaidItSetsTwo-for-FiveBonusIssueRecingJardineMathesonSaidItSetsTwo-for-FiveBonusIssueRecing“B”IsuzunsNoInterimItaly’sLaFondiariato1986ProfitsBidforSan JapanMinistrySaysOpenFarmTradeNorwayUnitCrownsinBonus1-NearestNeighborbasedbasedk=1,A類(lèi)帶權(quán)重計(jì)算，計(jì)算權(quán)重和最大的類(lèi)。k常取3或者5well-knownapproachtopatterninitiallybyFixandHodgestheoreticalerrorboundysisbyDuda&Hart(1957)appliedtotextcategorizationinearlyamongtop-performingmethodsinTCscalabletolargeTCkNNforTextCategorization(YangYM,SIGIR-1994)RepresentsaspointsDefineasimilaritymeasureforpairwises.Tuneparameterkforoptimizingclassificationeffectiveness.Chooseavotingscheme(e.g.,weightedsum)forscoringcategoriesThresholdonthescoresforclassificationdecisions（不是簡(jiǎn)單排序取NearestNeighbor“Similar”item:Weneeda“similarity”ifwewanttoapplythisautomatically.要考慮多少鄰居DoeseachneighborgetthesameK-NNusingaweighted-sumvotingSchemeCategoryScoringforThescoreforacategoryisthesumofthesimilarityscoresbetweenthepointtobeclassifiedandallofitsk-neighborsthatbelongtothegivencategory.Torestate:score(c|x)bc

sim(x,d)I(d,dkNNofxisthenewpoint;cisadisaclassifiedpointamongthek-nearestofsim(x,d)isthesimilaritybetweenxandI(d,c)=1iffpointdbelongstoclassI(d,c)=0簡(jiǎn)單(amongtop-5inbenarkevaluations)計(jì)算時(shí)間和空間（在一些場(chǎng)合不算太大kNN是懶散學(xué)習(xí)方法（lazylearning,基本openThen-dimensionalinputvectorxismappedintovariableybymeansofthescalarproductandanonlinearfunctionmap

outputfvector

vector

P(cj

|di)

P(dir

|cj)P(cjP(di

P(di|cj)P(cjP(di|cjP(wik|cj)，獨(dú)立性假kP(cj)

N(cjN(ckk

1N(cj|c|N(ckk

|cj)

wi在cj類(lèi)別文檔中出現(xiàn)在cj類(lèi)所有文檔中出現(xiàn)

1kBaggingBoosting基于SVMSVM(x1,y1),...(xl,yl)，xRn，y最優(yōu)分類(lèi)超平面(wx)b最優(yōu)分類(lèi)超平面xi)b1,when:yi(wxi)b1,when:yiyi[(wxib1，i1,...

2/||w

yi[(xiw)b]1,i1,2,...,（(w) w（2

(w)（wy[(x2b]1,i1,2,..., ：：lL(w,b,)ww){y[xw)b]

L(w0,b0,0)

0ii

i1,L,l L(w,b

)0wy0x i1,L,

i 11lW()

2ijyiyj(xixj

l

yiii i1,...,

i{yi

(w

0的樣本對(duì)分類(lèi)不起什么作用，有用的i0f(x)

i(xi

x)b* y*支持向y*而對(duì)于線(xiàn)性不可分的情況，通過(guò)引入i

(w,)

1(ww) l i

yi[(wxi