1、1,11 Discriminant Analysis判別分析,Sec. 1 介紹 Sec. 2 兩總體分類 Sec. 3 兩正態(tài)總體分類 Sec.4 評估判別函數(shù) Sec.5 多總體分類 Sec.6 典型判別函數(shù) Sec.7 logistic 回歸和判別分析 Sec.8 評注,2,The ideas associated with discriminant analysis can be traced back to the 1920s and work completed by the English statistician Karl Pearson, and others, on int

2、ergroup distances, e.g., coefficient of racial likeness (CRL), (Huberty, 1994). In the 1930s R. A. Fisher translated multivariate intergroup distance into a linear combination of variables to aid in intergroup discrimination. Methodologists from Harvard University contributed much to the interest in

3、 application of discriminant analysis in education and psychology in the 1950s and 1960s (Huberty, 1994). Klecka (1980) provided several historical references that deal mostly with early applications of DA.,歷史,2020/8/4,中國人民大學六西格瑪質(zhì)量管理研究中心,3,目錄 上頁 下頁 返回 結(jié)束,第四章 判別分析,回歸模型普及性的基礎(chǔ)在于用它去預(yù)測和解釋度量(metric)變量。但是對

4、于非度量(nonmetric)變量，多元回歸不適合解決此類問題。本章介紹的判別分析來解決被解釋變量是非度量變量的情形。在這種情況下，人們對于預(yù)測和解釋影響一個對象所屬類別的關(guān)系感興趣，比如為什么某人是或者不是消費者，一家公司成功還是破產(chǎn)等。 判別分析在主要目的是識別一個個體所屬類別的情況下有著廣泛的應(yīng)用。潛在的應(yīng)用包括預(yù)測新產(chǎn)品的成功或失敗、決定一個學生是否被錄取、按職業(yè)興趣對學生分組、確定某人信用風險的種類、或者預(yù)測一個公司是否成功。在每種情況下，將對象進行分組，并且要求使用這兩種方法中的一種可以通過人們選擇的解釋變量來預(yù)測或者解釋每個對象的所屬類別。,2020/8/4,中國人民大學六西格瑪

5、質(zhì)量管理研究中心,4,4.1 判別分析的基本理論,有時會遇到包含屬性被解釋變量和幾個度量解釋變量的問題，這時需要選擇一種合適的分析方法。比如，我們希望區(qū)分好和差的信用風險。如果有信用風險的度量指標，就可以使用多元回歸。但我們可能僅能判斷某人是在好的或者差的一類，這就不是多元回歸分析所要求的度量類型。 當被解釋變量是屬性變量而解釋變量是度量變量時，判別分析是合適的統(tǒng)計分析方法。 判別分析能夠解決兩組或者更多組的情況。 當包含兩組時，稱作兩組判別分析。當包含三組或者三組以上時，稱作多組判別分析（Multiple discriminant analysis）。 判別分析的假設(shè)條件 判別分析最基本的要

6、求是,分組類型在兩組以上；在第一階段工作是每組案例的規(guī)模必須至少在一個以上。解釋變量必須是可測量的，才能夠計算其平均值和方差，使其能合理地應(yīng)用于統(tǒng)計函數(shù)。,2020/8/4,中國人民大學六西格瑪質(zhì)量管理研究中心,5,目錄 上頁 下頁 返回 結(jié)束,4.1 判別分析的基本理論,判別分析的假設(shè)之一，是每一個判別變量（解釋變量）不能是其他判別變量的線性組合。即不存在多重共線性問題。 判別分析的假設(shè)之二，是各組變量的協(xié)方差矩陣相等。判別分析最簡單和最常用的形式是采用線性判別函數(shù)，它們是判別變量的簡單線性組合。在各組協(xié)方差矩陣相等的假設(shè)條件下，可以使用很簡單的公式來計算判別函數(shù)和進行顯著性檢驗。 判別分析

7、的假設(shè)之三，是各判別變量之間具有多元正態(tài)分布，即每個變量對于所有其他變量的固定值有正態(tài)分布。在這種條件下可以精確計算顯著性檢驗值和分組歸屬的概率。當違背該假設(shè)時，計算的概率將非常不準確。,6,Overview,Discriminant function analysis, a.k.a. discriminant analysis or DA, 主要用于分類. 好的判別函數(shù), 應(yīng)該正確判斷率比較高. Discriminant function analysis is found in SPSS under Analyze, Classify, Discriminant. One gets DA

8、or MDA from this same menu selection, depending on whether the specified grouping variable has two or more categories.,7,There are several purposes for DA and/or MDA: To classify cases into groups using a discriminant prediction equation. To test theory by observing whether cases are classified as p

9、redicted. To investigate differences between or among groups. To determine the most parsimonious way to distinguish among groups. To determine the percent of variance in the dependent variable explained by the independents. To determine the percent of variance in the dependent variable explained by

10、the independents over and above the variance accounted for by control variables, using sequential discriminant analysis. To assess the relative importance of the independent variables in classifying the dependent variable. To discard variables which are little related to group distinctions. To infer

11、 the meaning of MDA dimensions which distinguish groups, based on discriminant loadings.,8,Discriminant analysis has two steps: (1) F檢驗(Wilks lambda) 可以用于檢驗判別模型是否顯著, (2) 如F檢驗顯著, 然后考察獨立變量在類別之間的差異，以便對依賴變量進行分類。,Suppose an anesthesiologist needs to determine whether an anesthetic is safe for a person wh

12、o is having a heart operation. Based on these kinds of criteria, the anesthesiologist would like to know the following: can this knowledge be used to construct a rule that will classify new patients as to whether they are going to be safe or unsafe recipients of the anesthetic? what is the rule and

13、can the rule be used to classify new patients? what are the chances of making mistakes when using the rule?,麻劑,10,Discriminant analysis 為用來建立規(guī)則一種多元技術(shù)，該技術(shù)能幫助樣本進行適當分類。 Discriminant analysis 類似于回歸分析，但是其依賴變量或者被解釋變量為定性變量，而不是連續(xù)的。. Discriminant analysis is 也稱為分類分析.,目的: 從不同總體（或類別）中刻畫個體的特征。 盡量從不同類別使用判別器或分類器分

14、離開來. Goal of classification: 把不同個體分類到不同類別中. 問題是找到一個好的規(guī)則，能最優(yōu)的對新個體進行分類！,12,11.2 兩總體分類,主要問題 (1) 分類兩類個體 or (2) 把新個體指派到其中一個類別。記兩個類別為 1 and 2. The objects are separated or classified on the basis of measurements on p associated random variables X=X1, X2 , Xp. The observed values of X differ to some extent

15、 from one class to the other.,我們把第一類的個體看成一個總體 1 and 第二類的個體看成一個總體 2. 這兩個總體對應(yīng)的概率密度函數(shù)為f1(X) and f2(X), and consequently, 這樣可以就可以討論如何指定個體屬于那個類.,Example 11.1 考慮某城鎮(zhèn)中兩類人群: 1, 割草機擁有者, and 2, those 不擁有者. In order to identify the best prospect for an intensive sales campaign, 生產(chǎn)商 is interested in classifying

16、families as prospective owners or nonowners on the basis of x 1=income and x 2=lot size. Random samples of n 1=12 current owners and n 2=12 current nonowners are selected. The sample observations yield the scatter plot (Figure 11.1).,Remark 1. 一個好的判別方法應(yīng)該產(chǎn)生少數(shù)錯誤分類. 2. 要考慮先驗概率 . 3. 考慮誤判的成本或代價. (e.g. di

17、agnose disease),基本思想 令 f1(X) and f2(X) 分別為兩總體 1 and 2 對應(yīng)的密度函數(shù). 我們的目的是要把X指定給其中一個總體中.令 為全空間. 令R1 為x的一個集合， 當x屬于R1時，我們把對象x分配給總體 1 ，反之如果屬于 R2=-R1 則分配給總體 2. 假定 集合 R1 和 R2 互斥，構(gòu)成全空間.,令 p1 為 1的先驗概率 and p2 為 2的先驗概率, 其中 p1 + p2 = 1. 那么 P(觀測對象被正確地劃入 1) = P(X R1 | 1)P(1) = P(1|1) p1 P(觀測對象被錯誤劃入 1) = P(X R1 | 2 )

18、P(2 ) = P(1|2) p2 P(觀測對象被正確劃入2 ) = P(X R2 | 2 )P(2 ) = P(2 | 2) p2 P(觀測對象被錯誤劃入 2 ) = P(X R2 | 1)P(1) = P(2 |1) p1 （11-3）,錯分代價可以代價矩陣來表示 : 其中 c(2|1) 為屬于 1 被錯誤劃入 2 的代價 ，and c(1|2)為屬于 2 被錯誤劃入 1 的代價 .,那么平均的或期望的錯分代價為 (ECM) ECM = c(2|1)P(2 |1) p1 + c(1|2)P(1| 2) p2 (11-5) 一個合理的分類法則應(yīng)該有最小或盡可能小的ECM. 結(jié)論 11.1.

19、是ECM達到最小的區(qū)域R1 and R2 由下列不等式確定：,Proof of the result 11.1,We need to show that the regions R1 and R2 that minimize the ECM are defined by the vlues x for which the following inequalities hold: Substituting the expressions for P(2|1) and P(1|2) into (11-5) gives,We get the result 11.1.,11-7,假設(shè)有一個新觀測點 x

20、 0 ，其中 f 1(x 0 ) = .3 and f 2 (x 0 ) = .4. 問該點應(yīng)該劃入那個總體 ? Then 我們發(fā)現(xiàn) x 0 R1 ，因此應(yīng)該將其分入1,Other criteria,總錯誤概率（TPM）最小化原則 。 TPM = P(錯分 1 的觀測值或錯分 2 的觀測值) = p1 R1 f1 (x)dx +p2 R2 f2 (x)dx (11-8) 數(shù)學上這個問題等價于在錯分代價相同情況下師期望錯分代價最小化。因此，這種情況下的最優(yōu)區(qū)域由 (11-7)中的(b) 給出.,最大后驗概率原則,當 P(1 | x0) P(2 | x0)時, x0 劃入總體 1.,注釋 : 相當

21、于采用(11-7)中的總錯分概率的法則(b)，因為上式中分母相同. 但是，在觀測到x0 后再計算總體 1 和 2 的概率，這對識別不很明確的分配來說常常有用。,11.3 兩正態(tài)總體的分類,正態(tài)總體分類方法簡單高效。假定 f1(X) and f2(X) 為多元正態(tài)密度函數(shù)，, 分別有均值 1 and 協(xié)方差矩陣 1 and 均值向量 2 and 協(xié)方差矩陣 2.,（二）兩個總體距離判別法,先考慮兩個總體的情況，設(shè)有兩個協(xié)差陣相同的p維正態(tài)總體，對給定的樣本Y，判別一個樣本Y到底是來自哪一個總體，一個最直觀的想法是計算Y到兩個總體的距離。故我們用馬氏距離來給定判別規(guī)則，有：,1、方差相等,則前面的

22、判別法則表示為,當 和已知時， 是一個已知的p維向量，W（y）是y的線性函數(shù)，稱為線性判別函數(shù)。稱為判別系數(shù)。用線性判別函數(shù)進行判別分析非常直觀，使用起來最方便，在實際中的應(yīng)用也最廣泛。,Result 11.2,假定兩總體 1 and 2 具有 (11-10)的密度函數(shù). 這時使 ECM 最小化的分配法則如下： 把 x0 分配給 1 如 Allocate x0 to 2 otherwise.,上式中判別函數(shù)現(xiàn)在變成了一個線性函數(shù)了！,Proof of 11.2,Proof. Since the quantities in (11-11) are nonnegative for all x, w

23、e can take their natural logarithms and preserve the order of the inequalities. Moreover,Consequently, combine with (11-11),we get the results.,當總體參數(shù) 1, 2 , and 未知. Wald and Anderson suggest 建議將總體參數(shù)用樣本對應(yīng)量來代替.,1 : 正常人群 n1=30 2 : A型血友病犯者 n2=22,調(diào)查信息 因此代價相同，先驗概率相同情況下 得到,分配規(guī)則 如果 x0 = .210, .044, then y0

24、= 6.62 4.61. 我們把其分給 2.,假設(shè)先驗概率已知 : p 1 = .75, p 2 = .25. 并假定 c(1| 2) = c(2 |1).利用判別統(tǒng)計量 有 w = 6.62 (4.61) = 2.01 , Applying (11-18), we see that 這樣我們可以分配給 2 , an obligatory carrier.,協(xié)方差矩陣 12的分類,如果協(xié)方差矩陣不等，分配規(guī)則如下.,11.4 評估分類函數(shù),判斷分類方法優(yōu)劣的一個重要方法就是計算其誤判率或錯分率?？傚e分率為 通過適當選擇 R 1 and R 2得到該量的最小值, 稱為 最優(yōu)失誤率 (OER).,

25、其中R1和R2有（11-7）中的（b）確定。,樣本分類函數(shù)的效果可以用真實失誤率來評估 (AER),一般來說AER不能計算，因為它依賴未知的密度函數(shù)，但是用表現(xiàn)失誤率（APER）來替代，定義為訓練樣本中被錯分的比率。,11.5 多總體分類,1. 最小期望錯分代價法。 Let f i(X) be the density associated with population i, i=1,2,g. Let p i =the prior probability of population i, i=1,2,g. c (k|i)=the cost of allocating an item to k

26、when it belongs to i, for k, i= 1,2,g. Rk = the set of xs classified as k.,Result 11.5. 能使 ECM (11 -37)達到最小的分類域，可以通過將 x 分配給 k , k =1, 2,。, g, 如果下式最小： 不止一個最小，則將 x 分配給其中任意滿足要求的總體. 證明見張堯庭等（209）,正態(tài)總體分類,(1) 協(xié)方差不等時,二次判別函數(shù)，分配給第i個總體,(11.46),(2) 協(xié)方差矩陣相等時,相等時，判別得分為,因此可以定義線性判別得分,11.6 Fishers 判別函數(shù),Fishers idea

27、- 把多元變量 x 變成一元變量 y ，使得 ys 能盡量分類總體 1 and 2,A fixed linear combination of the xs takes the values y11 , y12, , y1n, for the observations from 1 and the values y21, y22, , y2n for the observations from 2 The separation of these two sets of univariate ys is assessed in terms of the difference between y1

28、 and y2 expressed in standard deviation units. That is,其平方后，分子相當于組間差,組內(nèi)差,67,典型判別函數(shù),典型判別函數(shù)的思想由 Fisher首次提出。典型判別分析通過對原始變量做線性變換來構(gòu)建新變量 。構(gòu)建的典型變量使得它們包含原始變量集中有用的信息 。換句話說，它們類似主成分和因子分析方法，當然計算方法有所不同 。,68,不考慮典型函數(shù)是否可以解釋，其優(yōu)點是它們可以簡化實際數(shù)據(jù)的維數(shù)，從而使得數(shù)據(jù)可以可視化. 典型函數(shù)允許研究人員開發(fā)簡單的判別規(guī)則 。,/stat/spss/dae/dis

29、crim.htm,69,典型分析的思想:,70,第一個典型判別函數(shù),假設(shè)研究人員獲得來自總體Gi的 ni 個樣本, 假設(shè)該總體服從分布為 Np(i,), for i=1,2,k. 并假設(shè)這些總體具有相同的協(xié)方差矩陣,71,Let,那么組間的離差為 B:,72,組內(nèi)變差為:,經(jīng)典判別分析的思想是，對原始數(shù)據(jù)進行投影使得變化后的樣本組間差別最大，組內(nèi)差別最小，即使得比值最大。,73,可以證明,E-1B的最大特征值. a1 為 E-1B 對應(yīng)的最大特征向量.線性組合y1=a1Tx 就是單個線性判別函數(shù) ，其提供了總體之間的最大差異.,這里F可以用于檢驗兩組之間的均值是否相同！,74,a1 is th

30、e largest eigenvalue, Proof(here we change some symbol),為什么V-1/2AV-1/2和V1A的特征根一樣，因為AB和BA的非0特征根相同！,75,V-1A的特征向量就是要找的系數(shù)a,76,77,X的有效判別可以基于 a1Tx, a1T1, , a1Tk, 令 di=|a1Tx-a1Ti| 如果 di.最小，則x應(yīng)該分配給第i個總體,第二典型判別函數(shù),y2=a2Tx,di2=(a1Tx-a1Ti)2+(a2Tx-a2Ti)2,Assign x to the population that gives the minimum value fo

31、r di2.,Result 11.6,80,81,82,83,Determining the dimensionality of the cannonical space,The dimensionality of the cannonical space s is bounded above y the minimum of p and m-1. We can construct SCREE plots of the eigenvalues or consider what proportion of the total variability is being accounted for

32、by each cannonical function and select enough to account for a large proportion of the total variability.,84,Example 7.3,Let data in iris become 1,2,3,then we use discriminant analysis Iris is grouping variable,85,Example 11.1,data gpa; infile T11-6.dat; input gpa gmat admit; proc discrim data = gpa

33、 pool=yes manova wcov pcov listerr crosslisterr; class admit ; var gpa gmat; run; 去掉先驗概率 /faculty/moser/exst7037/discrim.html Proc CanDisc Data=Iris All Out=OIris; Class Species; Var y1 y2; Run;,86,87,88,89,90,91,92,93,This shows a test for homogeneity of the variance-covarianc

34、e matrices for the three varieties.This test is significant and the hypothesis of equal would be rejected. The linear discriminant functions often work quite well even though the vaiance-c are unequal.If the prob of correct classification are high enough to satisfy the user,then the user should not

35、be too concerned that he is using a linear discriminant rule rather than a quadritic rule. SPSS cannot.,94,95,We see the eigenvalues of W-1B,as well as statistical tests for determining the dimensionality of the cannonical space.For this example, both eigenvalues are significant(p=0.0000).The first

36、accounts for 99.1% of total variability,so the second is not important.So, the means for these three varieties come close to lying on a straight line within the four-dim sample space .,96,Define standardized cannonical functions-these could be used on data that has been standardized to determine the

37、 projections of data points onto the cannonical space.,97,The first lists vectors that define unstandardized cannonical functions.For example, we could compute undstandardized cnnonical scores via Can1 =-.829*SL-1.5*SW+2.2*PL+2.8*PW-2.1 The location of the three variety means in the unstandardized c

38、annonical space are shown at the bottom.it is plot on the territorial map,their locations are given by the (*) on the plot.,98,99,100,Spss also locates the perpendicular bisectors between the variety means on this plot.SPSS calls this plot a territorial ma.Thess bisectors divide the cannonical space

39、 into three distinct regions.If the projection of a new data point falls into one of the regions ,then the new points is closets to the means (*) in that region which determines the variety to which the observation would be classified.,101,102,This table provide a listing of how each iris plant in t

40、he data set would be classified by the discriminant rule developed.The column labeled ACTUAL GROUP identifies the variety from which the observation came; the Highest shows the variety to which the observation would be assigned by the discriminatnt rule.,103,The first column labeled P(G/D) is the po

41、sterior probability for the group to which the observation is assigned. This posterior prob is 0.885 for case 3.SPSS gives posterior prob for only the best group and the second best group.The Second P(D/G) can be ignored. The last column, labeled DISCRIM SCORES, give the locations of the projections of the observations to the cannonical sp