MachineLearningandData 200

f=x,

3 4–不可微

wTx=f

= v>-1v￡5

y=f w

=fwTx=wTf6y=f

=wT其中f

7 定義代價(jià)函數(shù)為均方誤差ew)= e2n) nh

2Nn

Dw=-Dw=-ew1=-N()Nen?e8 最小均方 ：LeastMean其中e為時(shí)刻new=其中e為時(shí)刻n2 nh=- =-e ew=- =-e

9

基 n+

ew

ek2k?ek

2n nw

w

n)=-

?en

k()

eNe

?

w)=1k2Nn=1k?k

2nLL

w)=12Ln=1k?

2nekek 算法:基本概:

t exp

t=a

eat-e-ateat+e-at

Rumelhart,Hinton&Rumelhart,Hinton&Williams,“Learningrepresentationsbyback-propagatingerrors”,Nature,1986j j

en

1e2n)

2j?

av=

en=

ejn=1j?

n)L Le=1e(n)=1e2n)j j

2Ln=1j?

n

n=

n

n=-

ej

n

?wji)

ej=dj-yjj–如果神經(jīng)元jj

en

1e2n)

n=n)–C:

2j?

N

–對(duì)所有n)誘導(dǎo)局部域：v)

mwy ji vj

n+Dwjie/?wji

en=dn-

n)

n=n)–神經(jīng)元j是輸出節(jié)

1

“可見(jiàn)”誤差的瞬時(shí)能量：e =nyiyj=fjjDwji)e/?wji

n) en)en)ejnyjn)vjn)n)nynvjwjin)nvjyin)

n+=- =/=- =en=即

enenyn)vn)

enf

n)

n)n)nynvnwn)nn+hejjvjyi 神經(jīng)元局部梯度的定義：–對(duì)于輸出層神經(jīng)元

n=en)=-n=-en)en=- ?v

n

n)=

ejnfjv

n y

n=hejjvjyin=djnyi神經(jīng)元局部梯度的定義：–

n=en)=-n=-en)en=-?v

n

n)=

ejnfjv

n y

–n=-en)en)ejyjj

ynvj fjfj –對(duì)于隱藏層神經(jīng)元jdjn=-

en)=-

enyn)=-

envnn)

nv

n)–其中en

n,

=

y=fvm–于m

22

vk=wkjyjn)jen)=

k

nykvk

jjkj?yjjkj

?y

k()

?ekkkjdkjn)?=-ej

=

n=-en)nj =-jvjnjnjvjnj j)

d=ef

n=-en)

en)

f

nen)j j

fj

ej=dkj

ej=dkj

n

dj)=ejnjvjDwjin=hejjvjyin)=hdjyij：ejdkjdj=ejnjj

n=hejvjyin)=hdjyi驗(yàn)證：Dwjihe/?wjien)nwjiDwji=hdjn)yi權(quán)值校 局部梯 jvjnejjvjjjvje

yj=fjvj

=nyin)– – ek=ek=dk-ykdj=jvjnejhn)yi神經(jīng)元j在隱藏層 dj=jvjej

t= exp-

t=aftftjvj=afvjfvj=ayj-yj若jyjojdj=jvjnej=adj-ojojn1-ojdjnf

n)-j

t=a

eat-e-ateat+

at

t=a1ftftdj=jvjnejdjnf

n)

xi,dimmljvljvn=nyln)j

yn=

vn

y0n

n

yLn

n)

ej=dj-ojnfvLn

j?output

n=￠

fjv

j?hidden

n1n+hnyln)

小 機(jī)器學(xué)習(xí)與數(shù)achineLearning&DataScience

ej=dj-yj1

n=n)

en

n) N

e=1

m

=nyin)yj=fjvj

wji?Dwji)/ 稱函數(shù)，如雙曲正切，DropOut(G.E.Hinton,JMLRDropConnect(L.Wanetal.ICML

DropOut(G.E.Hinton,DropConnect(L.Wanetal.784-800x800x10的31.4%1.6%BP1.4%1.6%BP(Random1.2%1.2%RBM(784-BatchWithout[1]SergeyIoffe,ChristianSzegedy,"Batchnormalization:Acceleratingdeepnetworktrainingbyreducinginternalcovariateshift".ICML,2015.

XavierweightStochasticgradientdecentMomentumNesterovacceleratedgradient[1]XavierGlorot&YoshuaBengio,"Understandingthedifficultyoftrainingdeepfeed-forwardneuralnetworks",AI&STATS2010.DwjiaDwjihdji

則是什么？最優(yōu)權(quán)值下，網(wǎng)絡(luò)實(shí)際輸出值是期望響應(yīng)向量的條件期望的 ykj表示網(wǎng)絡(luò)響應(yīng)于xj的第k個(gè)神經(jīng)元的輸出：ykjFkxjTxT 2, M, j , ,..., TFx, 2, M, jN2 N2

R=12Nj

dj-Fxj

,jj如果Fk>Fj)"j?k則把輸入向量x分類(lèi)為 ，其Fk和Fj=1x,x,...,

[1]Richard&Lippmann:NeuralNetworkclassifiersestimateBayesianaposterioriprobabilities,Computation. [1]WebbA.R.andLoweD.:Theoptimalinternalrepresentationofmultilayerclassifiernetworksperformsnonlineardiscriminant ysis,[1]WebbA.R.andLoweD.:Theoptimalinternalrepresentationofmultilayerclassifiernetworksperformsnonlineardiscriminant ysis,NeuralNetworks,vol.5,pp.480-488,1990.–線性判別分析：假設(shè)有2個(gè)類(lèi)別C1 ?在分類(lèi)意義上， Q/PerceptronGaussianGaussianModelofMixtureofExperts混合專(zhuān)家HierarchicalTopicmodel中，LatentDirichletHierarchicalSparse在KernelmachineDeepLearning

SupervisedSupervisedLearningofSemanticClassesforImageAnnotationandRetrieval,PAMI2007Q/法 理論上，一個(gè)隱藏層就足夠作為通用函數(shù)近器地推廣,因?yàn)楦鶕?jù)理論,我們需要控制兩個(gè)因素:經(jīng)驗(yàn)ReLU(x)=max(x,[1]Krizhevsky,A.,Sutskever,I.,andHinton,G.E.Imagenetclassificationwithdeepconvolutionalneuralnetworks.InNIPS,2012. 個(gè) MatthewD.ZeilerandRobFergus,"VisualizingandUnderstandingConvolutional 個(gè) Stride2:相鄰窗口每次滑動(dòng)2個(gè)像素(或單位),i.e.5個(gè)像素 Filtersize7:使用7x7的kernel作用在每個(gè)窗口內(nèi)，輸出一個(gè)響應(yīng)值3x3maxpooling:下采樣(Downsampling)步驟在一個(gè)3x3鄰域內(nèi)的9個(gè)響應(yīng)3x3maxpoolingstride2:相鄰的3x3窗口每次滑動(dòng)2個(gè)像素， 個(gè) 輸入為 110x11055x55:使用3x3maxpoolingwithstride2396通道選用96個(gè)7x7x3的filters對(duì)224x224x3filter給出一個(gè)特征圖(feature個(gè) 得到256個(gè)26x26的featuremaps,然后MaxPooling之后變成13x13x256第6層和第7層為全連接層(FullConnected

[1]LeCun,Yann,etal."Backpropagationappliedtohandwrittenzipcoderecognition."Neuralcomputation,1989.[2]LeCun,Yann,etal."Gradient-basedlearningappliedtorecognition."ProceedingsoftheIEEE(1998):2278-2324.[3]AlexKrizhevsky,IlyaSutskever,andGeoffreyE.Hinton."Imagenetclassificationwithdeepconvolutionalneuralnetworks."NIPS.2012.He,K.,Ren,S.,Sun,J.,&Zhang,X.:DeepResidualLearningforImageRecognition,CVPR2016. xyxyHe,K.,Ren,S.,Sun,J.,&Zhang,X.:DeepResidualLearningforImageRecognition,CVPR[1]Hinton,G.E.,Krizhevsky,A.,Srivastava,N.,Sutskever,I.,&Salakhutdinov,R.:Dropout:asimplewaytopreventneuralnetworksfromoverfitting,JournalofMachineLearningResearch,15,1929-1958,2014.[2]He,K.,Ren,S.,Sun,J.,&Zhang,X.:DeepResidualLearningforImageRecognition,arXiv,2016.[3]SergeyIoffe,ChristianSzegedy,"Batchnormalization:Acceleratingdeepnetworktrainingbyreducinginternalcovariateshift".ICML,2015.[4]Fei-Fei,L.,Karpathy,A.,Leung,T.,Shetty,S.,Sukthankar,R.,&Toderici,G.(2014):Large-Scale NeuralNetworks.IEEECVPR.[5