認知的連接主義模型Chapter2

ConnectionistModelsofCognitionMichaelS.C.ThomasandJamesL.McClellandMichaelS.C.ThomasPosition:ReaderinCognitiveNeuropsychologyPostgraduateTutorCentreforBrainandCognitiveDevelopment,SchoolofPsychology,BirkbeckUniversityofLondon,Myprimaryinterestsareincognitiveandlanguagedevelopment,bothintermsofdevelopmentalprocessesinchildrenandinthefinalcognitivestructurestheyproduceintheadult.JamesL.McClellandProfessor,DepartmentofPsychology

Director,CenterforMind,BrainandComputation

StanfordUniversityJamesL.(Jay)McClelland(bornDecember1,1948)isaProfessorofPsychologyatStanfordUniversity.HeisbestknownforhisworkconcerningParallelDistributedProcessing,applyingconnectionistmodels(orneuralnetworks)toexplaincognitivephenomenasuchasspokenwordrecognitionandvisualwordrecognition.McClellandistoalargeextentresponsibleforthe"connectionistrevolution"ofthe1980s,whichsawalargeincreaseinscientificinterestforconnectionism..----wikipediaOverhiscareer,McClellandhascontributedtoboththeexperimentalandtheoreticalliteraturesinanumberofareas,mostnotablyintheapplicationofconnectionist/paralleldistributedprocessingmodelstoproblemsinperception,cognitivedevelopment,languagelearning,andtheneurobiologyofmemory.Hewasaco-founderwithDavidE.RumelhartoftheParallelDistributedProcessingresearchgroup,andtogetherwithRumelhartheledtheeffortleadingtothepublicationin1986ofthetwo-volumebook,ParallelDistributedProcessing,inwhichtheparalleldistributedprocessingframeworkwaslaidoutandappliedtoawiderangeoftopicsincognitivepsychologyandcognitiveneuroscience.McClellandandRumelhartjointlyreceivedthe1993HowardCrosbyWarrenMedalfromtheSocietyofExperimentalPsychologists,the1996DistinguishedScientificContributionAwardfromtheAmericanPsychologicalAssociation,the2001GrawemeyerPrizeinPsychology,andthe2002IEEENeuralNetworksPioneerAwardforthiswork.McClellandhasservedasSeniorEditorofCognitiveScience,asPresidentoftheCognitiveScienceSociety,andasamemberoftheNationalAdvisoryMentalHealthCouncil,andheiscurrentlypresident-electoftheFedertationoftheBehavioral,Psychological,andCognitiveSciences.HeisamemberoftheNationalAcademyofSciences,andhehasreceivedtheAPSWilliamJamesFellowAwardforlifetimecontributionstothebasicscienceofpsychology.一句話總結：這家伙很牛BackgroundSomeIllustrativeModelsConnectionistInfluencesonCognitiveTheoryConclusionsOutlineBackgroundWhatisConnectionist？AlsoknownasArtificialneuralnetwork(ANN)orParalleldistributedprocessing(PDP)modelshasbeenappliedtoadiverserangeofcognitiveabilities,includingmodelsofmemory,attention,perception,action,language,conceptformation,andreasoningAlthoughmanyofthesemodelsseektocaptureadultfunction,connectionismplacesanemphasisonlearninginternalrepresentations.Histrationcontext萌芽期（20世紀40年代）

1943年，心理學家McCulloch和數學家Pitts建立起了著名的閾值加權和模型，簡稱為M-P模型。發(fā)表于數學生物物理學會刊《BulletinofMethematicalBiophysics》

1949年，心理學家D.O.Hebb提出神經元之間突觸聯系是可變的假說——Hebb學習律。

第一高潮期（1950~1968）

以MarvinMinsky，FrankRosenblatt，BernardWidrow等為代表人物，代表作是單級感知器（Perceptron）。

可用電子線路模擬。

人們樂觀地認為幾乎已經找到了智能的關鍵。許多部門都開始大批地投入此項研究，希望盡快占領制高點。反思期（1969~1982）

M.L.Minsky和S.Papert，《Perceptron》，MITPress，1969年

異或”運算不可表示

第二高潮期（1983~1990）

1982年，J.Hopfield提出Hopfield網絡用Lyapunov函數作為網絡性能判定的能量函數，建立ANN穩(wěn)定性的判別依據闡明了ANN與動力學的關系用非線性動力學的方法來研究ANN的特性指出信息被存放在網絡中神經元的聯接上

1984年，

J.Hopfield設計研制了后來被人們稱為Hopfield網-Tank電路。較好地解決了著名的TSP問題，找到了最佳解的近似解，引起了較大的轟動。

1985年，Hinton、Sejnowsky、Rumelhart等人所在的并行分布處理（PDP）小組的研究者在Hopfield網絡中引入了隨機機制，提出所謂的Boltzmann機。

1986年，并行分布處理小組的Rumelhart等研究者重新獨立地提出多層網絡的學習算法——BP算法，較好地解決了多層網絡的學習問題。（Paker1982和Werbos1974年）

徐雷提出的Ying-Yang機理論模型

甘利俊一(S.Amari)開創(chuàng)和發(fā)展的基于統(tǒng)計流形的方法應用于人工神經網絡的研究,

國內首屆神經網絡大會是1990年12月在北京舉行的。注：以上3頁引自中科院計算所史忠植老師課件：

《神經信息學——平行分布式理論框架》SeeMore…KeypropertiesofConnectionistModels1）

asetofprocessingunits---ui2）

astateofactivationatagiventime---a(t)3）

apatternofconnectivity---wij4）

aruleforpropagatingactivationstatesthroughtthenetwork5）

anactivationruletospecifyhowthenetinputstoagivenunitarecombinedtoproduceitsnewactivationstate---F6）

thealgorithmformodifyingthepatternsofconnectivityasafunctionofexperience

Hebbianlearningruledeltarulebackpropagation7）

arepresentationoftheenvironmentwithrespecttothesysterm.NeuralplausibilityWhetherthese“brain-like”systemsareindeedneurallyplausible?Iftheyarenot,shouldtheyinsteadbeviewedasaclassofstatisticalfunctionapproximators?Andifso,shouldn’ttheabilityofthesemodelstosimulatepatternsofhumanbehaviorbeassessedinthecontextofthelargenumberoffreeparameterstheycontain(e.g.,intheweightmatrix;Green,1998)?Theadvantageofconnectionism,accordingtoitsproponents,isthatitprovidesbettertheoriesofcognition.Manyconnectionistmodelseitherincludepropertiesthatarenotneurallyplausibleoromitotherpropertiesthatneuralsystemsappeartohave.Endeavoringtoshowhowfeaturesofconnectionistsystemsmightinfactberealizedintheneuralmachineryofthebrain.Stressingthecognitivenatureofcurrentconnectionistmodels.Whyconnectionistmodelsshouldbereckonedanymoreplausibleasputativedescriptionsofcognitiveprocessesjustbecausetheyare“brain-like”?ConnectionvssymbolicTwosortsofcriticismTherelationshipbetweenConnectionistModels&BayesianInferenceTherearestronglinksbetweenthecalculationscarriedoutinconnectionistmodelsandkeyelementsofBayesiancalculations(seechapter3)Firstofall,thatunitscanbeviewedasplayingtheroleofprobabilistichypotheses.Second,instochasticneuralnetworks,anetwork’sstateoverallofitsunitscanrepresentaconstellationofhypothesesaboutaninput,and(iftheweightsandthebiasesaresetcorrectly)thattheprobabilityoffindingthenetworkinaparticularstateismonotonicallyrelatedtotheprobabilitythatthestateisthecorrectinterpretationoftheinput.連接主義模型與概率統(tǒng)計模型（如貝葉斯模型）連接主義模型：向底層發(fā)展——〉生物神經網絡的構建。BlueBrain.模擬真實的神經結構，可用于模擬藥物測試等。向高層發(fā)展——〉建模，為認知心理學中的某些現象提供解釋。模型的結構不一定符合生物的神經網絡結構，可以看作是基于統(tǒng)計數據提出的數學模型（概率統(tǒng)計模型）。概率統(tǒng)計模型：本人了解的不多。不過根據miner的說法，差不多隨便什么和認知相關的事都能用貝葉斯模型來解釋，很強大。這部分將由jake在下下次活動時詳細介紹。SomeIllustrativeModelsAnInteractiveActivationModelofContextEffectsinLetterPerception(McClelland&Rumelhart,1981,Rumelhart&McClelland,1982)用于解釋“詞優(yōu)越效應”TheprotocolHe(thescientist)presentedtargetlettersinwords,inunpronounceablenonwords,orontheirown.Thestimuliwerethenfollowedbyapatternmask,afterwhichparticipantswerepresentedwithaforcedchoicebetweentwolettersinagivenposition.Importantly,bothalternativeswereequallyplausible.Thus,theparticipantmightbepresentedwithWOODandaskedwhetherthethirdletterwasOorR.Asexpected,forced-choiceperformancewasmoreaccurateforlettersinwordsthanforlettersinnonwordsorletterspresentedontheirown.識別在詞中的字母時，自上而下（詞—〉字母）與自下而上（形狀—〉字母）相結合，最容易識別threemainassumptionsoftheIAmodel:Perceptualprocessingtakesplaceinasysteminwhichthereareseverallevelsofprocessing,eachofwhichformsarepresentationoftheinputatadifferentlevelofabstraction;visualperceptioninvolvesparallelprocessing,bothofthefourlettersineachwordandofalllevelsofabstractionsimultaneously;(3)perceptionisaninteractiveprocessinwhichconceptuallydrivenanddatadrivenprocessingprovidemultiple,simultaneouslyactingconstraintsthatcombinetodeterminewhatisperceived.Butnotadaptive–connectivitywassetbyhand.(2)theideaofbottom-upexcitationfollowedbycompetitionamongmutuallyexclusivepossibilitiesisastrategyfamiliarinBayesianapproachestocognition.OnLearningthePastTenseofEnglishVerbs

(Rumelhart&McClelland,1986)用于解釋兒童語言學習的U形現象DuringtheacquisitionoftheEnglishpasttense,childrenshowacharacteristicU-shapeddevelopmentalprofileatdifferenttimesforindividualirregularverbs.Initially,theyusethecorrectpasttenseofasmallnumberofhigh-frequencyregularandirregularverbs.Later,theysometimesproduce“overregularized”pasttenseformsforasmallfractionoftheirirregularverbs(e.g.,thinked;Marcusetal.,1992),alongwithother,lessfrequenterrors(Xu&Pinker,1995).Theyarealsoabletoextendthepasttense“rule”tonovelverbs(e.g.,wugwugged).Finally,inolderchildren,performanceapproachesceilingonbothregularandirregularverbs(Berko,1958;Ervin,1964;Kuczaj,1977)能通過學習自發(fā)“掌握”英語單詞過去時態(tài)的變化規(guī)則FindingStructureinTime(Elman,1990)英語語法學習：動詞和主語數的一致Recurrent網絡（前兩個是feedforward網絡），前一時刻的輸入在下一時刻仍有影響。Figure2.5.Trajectoryofinternalactivationstatesasthesimplerecurrentnetwork(SRN)processessentences(Elman,1993).RelatedModels···Cascade-CorrelationandIncrementalNeuralNetworkAlgorithmsMixture-of-Experts-ModelsHybridModelsBayesianGraphicalModelsConnectionistInfluencesonCognitiveTheoryKnowledgevsProcessingKnowledgeencode:active–recent

–howthingsarenowlatent–accumulated–howthingswillbeIfinformationdoesneedtobemovedaroundthesystem,thiswillrequirespecialstructuresandspecialprocesses.(seechapter7&8)Informationwillbeprocessedinthesamesubstratewhereitisstored.–controlsystem

hasnocontent–placeholders

efficiencyLastly,theconnectionistperspectiveonmemoryaltershowweconceiveofdomaingenerality

inprocessingsystems.Knowledgeishardtomoveaboutinconnectionistnetworksbecauseitisencodedintheweights.Cascade-CorrelationandIncrementalNeuralNetworkAlgorithmsCognitiveDevelopmentTheStudyofAcquiredDisordersinCognitiveNeuropsychologythecognitivesystemcomprisesasetofindependentlyfunctioningcomponents.Patternsofselectivecognitiveimpairmentafteracquiredbraindamagecouldthenbeusedtoconstructmodelsofnormalcognitivefunction.TheOriginsofIndividualVariability&DevelopmentalDisordersInadditiontotheirroleinstudyingacquireddisorders,thefactthatmanyconnectionistmodelslearntheircognitiveabilitiesmakesthemanidealframeworkwithinwhichtostudydevelopmentaldisorders,suchasautism,dyslexia,andspecificlanguageimpairmentBespokenetworks–>modelsfittogetherinthelargercognitivesystemComplexityvsSimplification&UnderstandingConnectionismwillbeaffectedbytheincreasingappealtoBayesianprobabilitytheory

inhumanreasoning.ConnectionismwillcontinuetohaveacloserelationtoneuroscienceBehavioralgenetics-buildlinksbetweenbehavior(wherevariabilityismeasured)andthesubstrateonwhichgeneticeffectsact.FutureDirectionsConclusions講完了~~謝謝大家！Feedforwardand