1、NLP and WordEmbeddingsWord representationdeeplearning.aiWord representationV = a, aaron, , zulu, 1-hot representationWoman (9853)Apple (456)Orange (6257)Man (5391)King (4914)Queen (7157)I want a glass of orange.I want a glass of apple .000010000000100001000000001000000100100000Andrew NgFeaturized re

2、presentation: word embeddingMan (5391)Woman (9853)King (4914)Queen (7157)Apple (456)Orange (6257)0.97-0.950.000.010.950.93-0.010.000.70.690.03-0.020.020.010.950.97I want a glass of orange .I want a glass of apple .Andrew NgVisualizing word embeddingsAndrew Ngvan der Maaten and Hinton., 2008. Visuali

3、zing data using t-SNEmanwomandogkingcatqueenfishgrap applethree foureoneorangetwoNLP and WordEmbeddingsUsing wordembeddingsdeeplearning.aiNamed entity recognition example101000SallyJohnsonisanorangefarmerRobertLinisanapplefarmerAndrew NgTransfer learning and word embeddings1.Learn word embeddings fr

4、om large text corpus. (1-100B words)(Or download pre-trained embedding online.)2.Transfer embedding to new task with smaller training set. (say, 100k words)3.Optional: Continue to finetune the word embeddings with new data.Andrew NgRelation to faceencoding$(&)f($(&)*$()f($()Taigman et. al., 2014. De

5、epFace: Closing the gap to human level performanceAndrew NgNLP and WordEmbeddingsProperties of wordembeddingsdeeplearning.aiAnalogiesMan (5391)Woman (9853)King (4914)Queen (7157)Apple (456)Orange (6257)Gender Royal AgeFood10.010.030.090.970.950.690.0110.020.020.01-0.950.930.700.020.00-0.010.030.950.

6、010.00-0.020.97Mikolov et. al., 2013, Linguistic regularities in continuous space word representationsAndrew NgAnalogies using word vectors()*+ (,-)*+ (/0+1 (?Andrew Ngmandogkingwomancatqueenfishthreefourgrapeapple onetwoorangeCosine similarity345(, (/0+1 ()*+ (,-)*+)Man:Woman as Boy:Girl Ottawa:Can

7、ada as Nairobi:Kenya Big:Bigger as Tall:TallerYen:Japan as Ruble:RussiaAndrew NgNLP and WordEmbeddingsEmbedding matrixdeeplearning.aiEmbedding matrixIn practice, use specialized function to look up an embedding.Andrew NgNLP and WordEmbeddingsLearning wordembeddingsdeeplearning.aiNeural language mode

8、lI4343want9665aglassoforange.1385261636257Iwant aglass oforange*+,+,45+,+,*-./*05-./5044*,1/245,1/25.0.,4*.0.,*.2/35.2/34Bengio et. al., 2003, A neural probabilistic language modelAndrew NgOther context/target pairsI want a glass of orange juice to go along with my cereal.Context: Last 4 words.4 wor

9、ds on left & rightLast 1 wordNearby 1 wordAndrew NgNLP and WordEmbeddingsWord2Vecdeeplearning.aiSkip-gramsI want a glass of orange juice to go along with my cereal.Mikolov et. al., 2013. Efficient estimation of word representations in vector space.Andrew NgModelVocab size = 10,000kAndrew NgProblems

10、with softmax classification(&)%*!#=&()-.,.01-*%,How to sample the context #?Andrew NgNLP and WordEmbeddingsNegative samplingdeeplearning.aiDefining a new learning problemI want a glass of orange juice to go along with my cereal.Mikolov et. al., 2013. Distributed representation of words and phrases a

11、nd their compositionalityAndrew NgModelSoftmax:(&)%*! #=context orange orange orange orange orange(wordjuicetarget?10000-.,. %&, )*01-king book the ofAndrew NgSelecting negative examplescontext orange orange orange orange orangewordjuice king book the oftarget?10000Andrew NgNLP and WordEmbeddingsGlo

12、Ve word vectorsdeeplearning.aiGloVe (globalvectors forword representation)I want a glass of orange juice to go along with my cereal.Pennington et. al., 2014. GloVe: Global vectors for word representationAndrew NgModelAndrew NgA note on the featurization view of word embeddingsMan (5391)Woman (9853)K

13、ing (4914)Queen (7157)Gender Royal AgeFood10.010.030.090.970.950.690.0110.020.020.01-0.950.930.700.026minimize 78,888 78,888 (,-.+ 0 02 log )*+*+*+*:7+:7Andrew NgNLP and WordEmbeddingsSentimentclassificationdeeplearning.aiSentiment classificationproblem!The dessert is excellent.Service was quite slo

14、w.Good for a quick meal, but nothing special.Completely lacking in good taste, good service, and good ambience.Andrew NgSimple sentiment classification modelThe8928dessert2468is4694excellent3180The#$%&$,-$%&$desert#&($,-&($is#(%,-(%excellent#)*$+,-)*$+“Completely lacking in good taste, good service,

15、 and good ambience.”Andrew NgRNN for sentiment classification8:;+-*$6&-%(-&7-)$&-)+,Completelylackingingood.ambienceAndrew Ng:;*+:;:;):;&:;*softmaxNLP and WordEmbeddingsDebiasing wordembeddingsdeeplearning.aiThe problem of bias in word embeddingsMan:Woman as King:QueenMan:Computer_Programmer as Woma

16、n: HomemakerFather:Doctor as Mother:NurseWord embeddings can reflect gender, ethnicity, age, sexual orientation, and other biases of the text used to train the model.Bolukbasi et. al., 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddingsAndrew NgAddressing bias in word e