LogisticRegressionThedata我們將建立一個(gè)邏輯回歸模型來(lái)預(yù)測(cè)一個(gè)學(xué)生是否被大學(xué)錄取。假設(shè)你是一個(gè)大學(xué)系的管理員，你想根據(jù)兩次考試的結(jié)果來(lái)決定每個(gè)申請(qǐng)人的錄取機(jī)會(huì)。你有以前的申請(qǐng)人的歷史數(shù)據(jù),你可以用它作為邏輯回歸的訓(xùn)練集。對(duì)于每一個(gè)培訓(xùn)例子，你有兩個(gè)考試的申請(qǐng)人的分?jǐn)?shù)和錄取決定。為了做到這一點(diǎn)，我們將建立一個(gè)分類模型，根據(jù)考試成績(jī)估計(jì)入學(xué)概率。#三大件importnumpyasnpimportpandasaspdimportmatplotlib.pyplotasplt%matplotlibinlineimportos#path='data'+os.sep+'LogiReg_data.txt'pdData=pd.read_csv("F:\\研究課程\\機(jī)器學(xué)習(xí)\\宇迪系列Wpython數(shù)據(jù)分析與機(jī)器學(xué)習(xí)實(shí)戰(zhàn)-全-E\\唐宇迪-機(jī)器學(xué)習(xí)課程資料\\機(jī)器學(xué)習(xí)算法配套案例實(shí)戰(zhàn)\梯度下降WdataWLogiReg_data.txt",header=None,names=['Exam1','Exam2','Admitted'])pdData.head()Ejcam1Exam2Admitted034.623G607&.0246930130.28671143.394393023584740972.90219803fflM32599&6.308552147^.03273&75.3413761pdData.shape(100r2)positive=pdData[pdData['Admitted']==1]#returnsthesubsetofrowssuchAdmitted=1,i.e.thesetof*positive*examplesnegative=pdData[pdData['Admitted']==0]#returnsthesubsetofrowssuchAdmitted=0,i.e.thesetof*negative*examplesfig,ax=plt.subplots(figsize=(10,5))ax.scatter(positive['Exam1'],positive['Exam2'],s=30,c='b',marker='o',label='Admitted')ax.scatter(negative['Exam1'],negative['Exam2'],s=30,c='r',marker='x',label='NotAdmitted')ax.legend()ax.set_xlabel('Exam1Score')ax.set_ylabel('Exam2Score')

<rflrtplotlibB1;errt,T?<tatftdtd?<rflrtplotlibB1;errt,T?<tatftdtd?抑出Q能Thelogisticregression目瑋：潼立分艾羞「我解匕三-卜譽(yù)頻必/h比?)設(shè)士iM直，根拒閾值対斷艮取結(jié)杲夏完成的模抉si?m3ii映射到鶴率旳匡竝nodcl:返回預(yù)則結(jié)果值cost：根擁蜃魏計(jì)茸損矢;radicn-t:計(jì)賓每/'■參數(shù)的；弟度右豈descait理訂參鎮(zhèn)吏韻I■iccurazj-：-?+耳蒂恿sigmoid函數(shù)鮒=7^defsigmoid(z):return1/(1+np.exp(-z))nums=np.arange(-10,10,step=l)#createsavectorcontaining20equallyspacedvaluesfrom-10to10fig,ax=plt.subplots(figsize=(12,4))ax.plot(nums,sigmoid(nums),'r')［SzrtpInLlili.liriEw.LAnEZDntQi：］Ib=iIb41d4eCI〉］SigmoidgR-211gm=0.5?削—=0g(十g)=Idefmodel(X,theta):returnsigmoid(np.dot(X,theta.T))（肉斫吐）xxi 乜pdData.insert(0,'Ones',1)#inatry/exceptstructuresoasnottoreturnanerroriftheblocksiexecutedseveraltimessetX(trainingdata)andy(targetvariable)orig_data=pdData.as_matrix()#convertthePandasrepresentationofthedatatoanarrayusefulforfurthercomputationscols=orig_data.shape[l]X=orig_data[:,0:cols-1]y=orig_data[:,cols-1:cols]converttonumpyarraysandinitalizetheparameterarraytheta#X=np.matrix(X.values)#y=np.matrix(data.iloc[:,3:4].values)#np.array(y.values)theta=np.zeros([1,3])X[:5]array([[1.? 34.6236E962,78.02469282],[1.r30.28S71077,43.89499752],[1.=35.84740877,72.90219803],[1.r60.18250939,86.3085521]r[1.r79.03273605,75.34437644]])y[:5]array([[0.],[0.],[0.],[1.].[1.]])

扌前失函數(shù)扌前失函數(shù)。陽(yáng)gy}=-ylag（咖⑷）一（I—叭Io威I一加⑴）defcost(X,y,theta):left=np.multiply(-y,np.log(model(X,theta)))right=np.multiply(l-y,np.log(l-model(X,theta)))returnnp.sum(left-right)/(len(X))cost(X,y,theta)0.69314718055994529計(jì)算梯度気二-誇山_隔肋夠defgradient(X,y,theta):grad=np.zeros(theta.shape)error=(model(X,theta)-y).ravel()forjinrange(len(theta.ravel())):#foreachparmeterterm=np.multiply(error,X[:,j])grad[0,j]=np.sum(term)/len(X)returngradGradientdescent比較3中不同梯度下降方法STOP_ITER=0STOP_COST=1STOP_GRAD=2defstopCriterion(type,value,threshold):returnvalue>thresholdreturnabs(value[-1]-value[-2])<thresholdreturnnp.linalg.norm(value)<threshold#returnvalue>thresholdreturnabs(value[-1]-value[-2])<thresholdreturnnp.linalg.norm(value)<thresholdiftype==STOP_ITER:eliftype==STOP_COST:eliftype==STOP_GRAD:importnumpy.random#洗牌defshuffleData(data):np.random.shuffle(data)cols=data.shape[1]X=data[:,0:cols-1]y=data[:,cols-1:]returnX,yimporttimedefdescent(data,theta,batchSize,stopType,thresh,alpha):#梯度下降求解init_time=time.time()i=0#迭代次數(shù)k=0#batchX,y=shuffleData(data)grad=np.zeros(theta.shape)#計(jì)算的梯度costs=[cost(X,y,theta)]#損失值whileTrue:grad=gradient(X[k:k+batchSize],y[k:k+batchSize],theta)k+=batchSize#取batch數(shù)量個(gè)數(shù)據(jù)ifk>=n:k=0X,y=shuffleData(data)#重新洗牌theta=theta-alpha*grad#參數(shù)更新costs.append(cost(X,y,theta))#計(jì)算新的損失i+=1ifstopType==STOP_ITER: value=ielifstopType==STOP_COST: value=costselifstopType==STOP_GRAD: value=gradifstopCriterion(stopType,value,thresh):breakreturntheta,i-1,costs,grad,time.time()-init_timedefrunExpe(data,theta,batchSize,stopType,thresh,alpha):#importpdb;pdb.set_trace();theta,iter,costs,grad,dur=descent(data,theta,batchSize,stopType,thresh,alpha)name="Original"if(data[:,1]>2).sum()>1else"Scaled"name+="data-learningrate:{}-".format(alpha)ifbatchSize==n:strDescType="Gradient"elifbatchSize==1:strDescType="Stochastic"else:strDescType="Mini-batch({})".format(batchSize)name+=strDescType+"descent-Stop:"ifstopType==STOP_ITER:strStop="{}iterations".format(thresh)elifstopType==STOP_COST:strStop="costschange<{}".format(thresh)else:strStop="gradientnorm<{}".format(thresh)name+=strStopprint("***{}\nTheta:{}-Iter:{}-Lastcost:{:03.2f}-Duration:{:03.2f}s".format(name,theta,iter,costs[-1],dur))fig,ax=plt.subplots(figsize=(12,4))ax.plot(np.arange(len(costs)),costs,'r')ax.set_xlabel('Iterations')ax.set_ylabel('Cost')ax.set_title(name.upper()+'-Errorvs.Iteration')returntheta不同的停止策略設(shè)定迭代次數(shù)#選擇的梯度下降方法是基于所有樣本的n=100runExpe(orig_data,theta,n,STOP_ITER,thresh=5000,alpha=0.000001)data.=leamon：iatr.le-Ofi-G-xadicntdrecent-Slap:ECCOLt匕匚乩tiinu[[rO-CDDZTn?HOJTOK320.00376111]]-Itei.5000-La^t-.0.S3-ImratiOD.LLB=ajrrayftC-ClKOaTia?!ClOJTDEg島0.OD37bT]L]]JORIGINALDATA-LEARNINGRATE:：IE-06亠GRADIENTDESCENT-STOP:5000ITERATIONS?Errorvs.Iteration根據(jù)損失值停止設(shè)定閾值1E-6,差不多需要110000次迭代runExpe(orig_data,theta,n,STOP_COST,thresh=0.000001,alpha=0.001)*f*Orisinaldata-leamincrate:0,OCd-GradientdMcwit_Stop-owtschanst<Le-05IhEta.[[-5. 0-W77L429D.O4UT23P7]]-Iter:-匚artmrt.也33-IXiiaUan. 47sarrsy(：[[-5.13364014. 0.0477142^U04072^7]])

根據(jù)梯度變化停止設(shè)定閾值0.05，差不多需要40000次迭代runExpe(orig_data,theta,n,STOP_GRAD,thresh=0.05,alpha=0.001)|I_K. Jz.1z- Lz.c.J.LL1->Lillclllk：：7-?. -J'J I 'U..■-Ibeta.[-1.37：E3D?1027216320.018990)]-Iter:?O45-LancostO.J；-ZuiuLi.u.It.■■'■T-yi -：in-i.n.-■= ■.r：■■)ORIGIMALDATA?LEARNINGKATE:0.001-GRADIENTDESCENT.STOP:GRADIENTMOftM<0.05-Errorvs.It?ratioMini-batchdescentrunExpe(orig_data,theta,16,STOP_ITER,thresh=15000,alpha=0.001)? data.- rat-fl.□.00L—Mini^batchde-roait-Step.15Q0DiterationsrbEta/[[-1.0352224 D.0L2425411-15DCC-Ls?tcost.-0,57-Euratioi-LUsaj=iv([[-L0S2224? 0.0L6?Q9T.H0L2d23d]])ORIGINALDATA，LEARNINGRATE:0001?-MINI-BATCH(16)DESCENTSTOP：150DOITERATIONS?ErrorVS.Iteration3Da zooa 4doo 6ana sddq inoon umq i^odo i&aao浮動(dòng)仍然比較大，我們來(lái)嘗試下對(duì)數(shù)據(jù)進(jìn)行標(biāo)準(zhǔn)化將數(shù)據(jù)按其屬性（按列進(jìn)行）減去其均值,然后除以其方差。最后得到的結(jié)果是，對(duì)每個(gè)屬性/每列來(lái)說(shuō)所有數(shù)據(jù)都聚集在0附近，方差值為1fromsklearnimportpreprocessingasppscaled_data=orig_data.copy()scaled_data[:,1:3]=pp.scale(orig_data[:,1:3])runExpe(scaled_data,theta,n,STOP_ITER,thresh=5000,alpha=0.001)

+**5calcddirta一lEartunsrate:D-DOL-GradientdcscEHt一3top:5000itcritiansThrta：[[fl.30f