1 signal

observation&reward

RealworldenvironmentAgent historyinfo

simulatorEachtimesteptAgentanaction????Worldupdatesgivenactionat,emitsobservationandAgentreceivesobservationandUseexperiencetoguidefuturedecisions(exploit)signal

observation&reward

RealworldenvironmentAgent historyinfo

simulatorHistory???=(??1,...,????,????,AgentchoosesactionbasedonhistoryisinformationassumedtodeterminewhathappensnextFunctionhistory=(???)Stateisifandonlyif p(????+1|,????)=p(????+1|???,????)Goalselectactionstomaximizetotalexpectedfuturerewardbalancingimmediate&long-termrewardsπdetermineshowtheagentchoosesactionsDeterministicpolicyStochasticpolicyfunctionexpecteddiscountedsumfuturerewardsunderapolicyπ initializeenvPolicymodelinitializeenvPolicymodelinitializepolicyPolicyinferenceinitializepolicyRolloutdataRolloutdataPolicyupdateUpdatepolicyUpdatepolicyHessel,Matteo,etal."Rainbow:Combiningimprovementsindeepreinforcementlearning."——給PPO帶來真正的性能上提升以及將policy約束在trustregion內(nèi)的效果，都不是通過PPO論文中提出的對新的policy和原policy的比值進(jìn)行裁切（clip）帶來的，而是通過code-level的一些技巧帶來的。Engstrom,Logan,etal."Implementationmattersindeeppolicygradients:AcasestudyonPPOandTRPO."Liang,Eric,etal."Rayrllib:Acomposableandscalablereinforcementlearninglibrary."Liang,Eric,etal."Rayrllib:Acomposableandscalablereinforcementlearninglibrary."新算法新算法新架構(gòu) 難以復(fù)用的強化學(xué)習(xí)代碼

可擴展性的強化學(xué)習(xí)框架 TrainingDataMLModelTrainingDataMLModelTrainingsignalθ

observation&reward

RealworldenvironmentAgent historyinfo

simulator面臨的問題面臨的問題新的需求Horgan,Dan,etal."Distributedprioritizedexperiencereplay."可能傳輸大量的數(shù)據(jù)可能傳輸大量的數(shù)據(jù)GPUCPU面臨的問題面臨的問題可能的解決方案 通用的RL算法針對Env開發(fā)支持分布式Star數(shù)目RepoACME+Reverb2.1k/deepmind/acmeELF2k/facebookresearch/ELFRay+RLlib16.4k/ray-project/rayGym24.5k/openai/gymBaselines11.6k/openai/baselinesTorchBeast553/facebookresearch/torchbeastSeedRL617/google-research/seed_rlTianshuo？3.2k/thu-ml/tianshouKeras-RL5.1k/keras-rl/keras-rlRayisafastandsimpleframeworkforbuildingandrunningdistributedapplications./ray-project/ray Rayisafastandsimpleframeworkforbuildingandrunningdistributedapplications.AprocessexecutingtheuserprogramAstatelessprocessthatexecutesremotefunctionsinvokedbyadriverAstatefulprocessthatexecutesDistributedobjectIn-memorydistributedstoragetostoretheinputs/outputs,orstatelesscomputation.ImplementtheobjectstoreviasharedmemoryUseApacheArrowasdataformatsDistributedschedulerSubmittedfirsttolocalschedulerGlobalschedulerconsiderseachloadandconstraintstoschedulingdecisionsGlobalControlAkey-valuestorewithpub-subfunctionalityRLlibisanopen-sourcelibraryforreinforcementlearningthatoffersbothhighscalabilityandaunifiedAPIforavarietyofapplications.RayRayRLlib/ray-project/ray/tree/master/rllib distributedschedulerisanaturalfitforthehierarchicalcontrolmodel,asnestedcomputationcanbeimplementedinRaywithnocentraltaskschedulingbottleneck.Hierarchicalcontrol Actors/Workers RunscriptRemotedecoratorforruninremote InitrayRemotedecoratorforruninremoteInitrayExecutethetrainerandactorinremoteExecutethetrainerandactorinremoteStartthreadforasyncStartthreadforasynctrainingsignal

observation&reward

RealworldenvironmentAgent historyinfo

simulatorPolicyGraphPolicyModelPolicyOptimizerPolicyGraphPolicyModelPolicyOptimizerThepolicyoptimizerisresponsiblefortheperformance-criticaltasksofdistributedsampling,parameterupdates,andmanagingPolicyGraphPolicyModelPolicyOptimizerPseudocodeforfourRLlibpolicyoptimizerstepmethods.Eachstep()operatesalocalpolicygraphandarrayofremoteevaluatorreplicas. Serializationanddeserializationarebottlenecksinparallelanddistributedcomputing,especiallyinmachinelearningapplicationswithlargeobjectsandlargequantitiesofdata.Goalsefficientwithlargenumericaldata(e.g.NumpyandPandasdataframes)AsasPicklePythontypesCompatiblewithsharedmemory(allowingmultipleprocessestousethesamewithoutcopyingit)Deserializationshouldbeextremelylanguageindependent Makingdeserializationfastisimportant.AnobjectmaybeserializedonceandthendeserializedmanytimesAcommonpatternisformanyobjectstobeserializedinparallelandthenaggregatedanddeserializedoneatatimeonasingleworkermakingdeserializationthebottleneckDeserializationisfastandbarelyvisibleUsingonlytheschema,cancomputetheoffsetseachvalueinthedatablobwithoutscanningthroughthedatablob(unlikePickle,thisiswhatenablesfastdeserialization)copyingorotherwiseconvertinglargearraysandothervaluesduringdeserializat