1

importnumpyasimportnumpyasN,D=3,x=np.random.randn(N,y=np.random.randn(N,z=np.random.randn(N,a=x*b=a+c=

===grad_c*np.ones((N,===grad_a*=grad_a*

Python-Python- importimportx,y=y=Python-Python-

靈活高

Python-

Class{void{}voidClass{void{}voidbackward(inputs,{}Class{…REGISTER_LAYER(“Attention”,SGD:??←???SGDwithmomentum:??←???????+?? 前端編程語(yǔ)言和接Python,前端編程語(yǔ)言和接Python,Lua,R,自動(dòng)求導(dǎo)（Auto統(tǒng)一模型表示：計(jì)算流+y圖的優(yōu)化與調(diào)度執(zhí)Batching,Cache,內(nèi)核代碼優(yōu)化與編GPUkernel,autokernel計(jì)計(jì)算硬CPU,GPU,RDMA importnumpyimportnumpyasnpN,D=3,grad_c=grad_b=grad_c*np.ones((N,D))grad_a=grad_b.copy()grad_z=grad_b.copy()grad_x=grad_a*ygrad_y=grad_a*x

======c===x*a+

xyz*xyz*a+bΣc ??(??,????),??(??,????),exp+exp+exp2

+

+ 2) exp+exp2exp+exp2

+

+ xyz*axyz*a+bΣc前端編程語(yǔ)言和接Python,Lua,前端編程語(yǔ)言和接Python,Lua,R,自動(dòng)求導(dǎo)（Auto統(tǒng)一模型表示：計(jì)算流+y圖的優(yōu)化與調(diào)度執(zhí)Batching,Cache,內(nèi)核代碼優(yōu)化與編GPUkernel,autokernel計(jì)計(jì)算硬CPU,GPU,RDMAxw*b+xw*b+yxw*b+yBatchsame-typeoperatorstoleverageGPUmassive++×+M× MMMMM ht-Data-flowgraphofaGRUBatchsame-typeoperatorstoleverageGPUmassive+×+×+M× MMMMM ht-+××+MMht-RWData-flowgraphofaGRUxyz*a+xyz*a+bΣc1xyzxyz*a+bΣcGPU顯式圖劃GPU GPU** Transparenttensor** Transparenttensor**

CPUGPU

a c

計(jì)算硬CPU,GPU,RDMA前端編程語(yǔ)言和接Python,Lua,R,計(jì)算硬CPU,GPU,RDMA前端編程語(yǔ)言和接Python,Lua,R,自動(dòng)求導(dǎo)（Auto統(tǒng)一模型表示：計(jì)算流 b圖的優(yōu)化與調(diào)度執(zhí)Batching,Cache,內(nèi)核代碼優(yōu)化與編GPUkernel,autokernelLayer-StaticLayer-StaticPython-CannotleverageNoprogrammingTensorFlow,CNTK,DeclarativeGraphProgramingwithLargekernel

Moreimportfromtorch.autogradimportVariableN,D=3importfromtorch.autogradimportVariableN,D=3,4x=Variable(torch.randn(N,y=Variable(torch.randn(N,z=Variable(torch.randn(N,foriina=x*b=a+c=c+Layer-StaticLayer-StaticDynamicPython-Imperativeprogramming(Define-by-NographCannotleverageNoprogrammingTensorFlow,CNTK,DeclarativeGraphProgramingwithLargekernel

MoreCompilerCompilerisusedtooptimizegeneralframeworktobemoreefficient,whilekeepingtheexistingflexibility!DeeplearningprovideeasiervariouslibrariesMachineLearningLanguageandPowerfulCompilerCodeoptimization,sparsityoptimization,hardwaretargetingAFull-FeaturedProgrammingLanguageforML:Expressiveandflexible