AGeneralTheoryofEquivariantHomogeneousSpacessonTacoS.CohenQualcommAIResearch?QualcommTechnologiesNetherlandsB.V.tacos@MarioGeigerEPFLAGeneralTheoryofEquivariantHomogeneousSpacessonTacoS.CohenQualcommAIResearch?QualcommTechnologiesNetherlandsB.V.tacos@MarioGeigerEPFLmario.geiger@epfl.chMauriceQUVALabU.ofAmsterdamm.@uva.nlAbstractWepresentageneraltheoryofGroupequivariantConvolutionalNeuralNetworks(G-s)onhomogeneousspacessuchasEuclideanspaceandthesphere.Featuremapsinthesenetworksrepresentfieldsonahomogeneousbasespace,andlayersareequivariantmapsbetweenspacesoffields.ThetheoryenablesasystematicclassificationofallexistingG-sintermsoftheirsymmetrygroup,basespace,andfieldtype.Wealsoanswerafundamentalquestion:whatisthemostgenerallinearmapbetweenfeaturespaces(fields)ofgiventypes?Weshowthatsuchmapscorrespondone-to-onewithgeneralizedconvolutionswithanequivariantkernel,andcharacterizethespaceofsuchkernels.1 IntroductionThroughtheuseofconvolutionlayers,ConvolutionalNeuralNetworks(understandingoflocalityandtranslationalsymmetrythatisinherentinmanylearningproblems.Becauseconvolutionsaretranslationequivariant(ashiftoftheinputleadstoashiftoftheoutput),convolutionlayerspreservethetranslationsymmetry.Thisisimportant,becauseitmeansthatfurtherlayersofthenetworkcanalsoexploitthesymmetry.Motivatedbythesuccessofs,manyresearchershaveworkedongeneralizations,leadingtoagrowingbodyofworkonGroupequivariants(G-s)forsignalsonEuclideanspaceandthesphere[1–7]aswellasgraphs[8,9].Withtheproliferationofequivariantnetworklayers,ithasbecomedifficulttoseetherelationsbetweenthevariousapproaches.Furthermore,whenfacedwithanewmodality(diffusiontensorMRI,say),itmaynotbeimmediatelyobvioushowtocreateanequivariantnetworkforit,orwhetheragivenkindofequivariantlayeristhemostgeneralone.InthispaperwepresentageneraltheoryofhomogeneousG-s.Featurespacesaremledasspacesoffieldsonahomogeneousspace.TheyarecharacterizedbyagroupofsymmetriesG,asubgroupH GthattogetherwithGdeterminesahomogeneousspaceB'G/H,andarepresentation?ofHthatdeterminesthetypeoffield(vector,tensor,etc.).Relatedworkisclassifiedby(G,H,).ThemaintheoremssaythatequariantlinearmapsbetweenfieldserBcanbewithanequivariantkernel,andthatthespaceofequivariantkernelscanberealizedinthreeequivalentways.Wewillassumesomefamiliaritywithgroups,cosets,quotients,representationsandrelatednotions(seeAppendixA).Thispaperdoesnotcontaintrulynewmathematics(inthesensethataprofessionalmathematicianwithexpertiseintherelevantsubjectswouldnotbesurprisedbyourresults),butinsteadprovidesanewformalismforthestudyofequivariantconvolutionalnetworks.Thisformalismturnsouttobearemarkablygoodfitfordescribingreal-worldG-s.Moreover,bydescribingG-sinalanguageusedthroughoutmodernphysicsandmathematics(fields,fiberbundles,etc.),itbecomespossibletoapplyknowledgegainedovermanydecadesinthosedomainstomachinelearning.*QualcommAIResearchisaninitiativeofQualcommTechnologies,Inc.33rdConferenceonNeuralInformationProcessingSystems(NeurIPS2019),Vancouver,Canada.1.1OverviewoftheTheoryThispaperhastwomainparts.First,inSec.2,weintroduceamathematicalmfor1.1OverviewoftheTheoryThispaperhastwomainparts.First,inSec.2,weintroduceamathematicalmforconvolutionalfeaturespaces.Thebasicideaisthatfeaturemapsrepresentfieldsoverahomogeneousspace.Asitturnsout,definingthenotionofafieldisquiteabitofwork.Soinordertomotivatetheintroductionofeachoftherequiredconcepts,wewillinthissectionprovideanoverviewoftherelevantconceptsandtheirrelations,usingtheexampleofaSphericalwithvectorfieldfeaturemaps.Thesecondpartofthispaper(Section3)isaboutmapsbetweenthefeaturespaces.Werequirethesetobeequivariant,andfocusinparticularonthelinearlayers.Themaintheorems(3.1–3.4)showthatlinearequivariantmapsbetweenthefeaturespacesareinone-to-onecorrespondencewithequivariantconvolutionkernels(i.e.convolutionisallyouneed),andthatthespaceofequivariantkernelscanberealizedasaspaceofmatrix-valuedfunctionsonagroup,cosetspace,ordoublecosetspace,subjecttolinearconstraints.Inordertospecifyaconvolutionalfeaturespace,weneedtospecifytwothings:ahomogeneousspaceBoverwhichthefieldisdefined,andthetypeoffield(e.g.vectorfield,tensorfield,etc.).AhomogeneousspaceforagroupGisaspaceBwhereforanytwox,y2Bthereisatransformationg2Gthatrelatesthemviagx=y.HereweconsidertheexampleofavectorfieldonthesphereB=S2withsymmetrygroupG=SO(3),thegroupof3Drotations.ThesphereisahomogeneousspaceforSO(3)becausewecanmapanypointonthespheretoanyotherviaarotation.Formally,afieldisdefinedasasectionofavectorbundleassociatedtoaprincipalbundle.Inordertounderstandwhatthismeans,wemustfirstknowwhatafiberbundleis(Sec.2.1),andunderstandhowthegroupGcanbeviewedasaprincipalbundle(Sec.2.2).Briefly,afiberbundleformalizestheideaofparameterizingasetofidenticalspacescalledfibersbyanotherspacecalledthebasespace.ThefirstwayinwhichfiberbundlesplayaroleinthetheoryisthattheactionofGonBallowsustothinkofGasa“bundleofgroups”orprincipalbundle.Roughlyspeaking,thisworksasfollows:ifwefixanorigino2B,wecanconsiderthestabilizersubgroupH Goftransformationsthatleaveounchanged:H={g2G|go=o}.Forexample,onthespherethestabilizerisSO(2),thegroupofrotationsaroundtheaxisthrougho(e.g.thenorthpole).AswewillseeinSection2.2,thisallowsustoviewGasabundlewithbaseFigure1:SO(3)asaprincipalSO(2)bundleoverS2.spaceB'G/HandafiberH.ThisisshownforthesphereinFig.1(cartoon).Inthiscase,wecanthinkofSO(3)asabundleofcircles(H=SO(2))overthesphere,whichitselfisthequotientS2'SO(3)/SO(2).Todefinetheassociatedbundle(Sec.2.3)wetaketheprincipalbundleGandreplacethefiberHbyavectorspaceVonwhichHactslinearlyviaagroupepesentation.ThisyieldsaectorundlewiththesamebasespaceBandanewfiberV.Forexample,thetangentbundleofS2(Fig.2)isobtainedbyreplacingthecircularSO(2)fibersinFig.1by2Dplanes.UndertheactionofH=SO(2),atangentvectoratthenorthpoleisrotated(eventhoughthenorthpoleitselfisfixedbySO(2)),sowelet?(h)bea2?2rotation2:TangentbundleofS2.matrix.Inageneralconvolutionalfeaturespacewithnchannels,Vwouldbeann-dimensionalvectorspace.Finally,fieldsaredefinedassectionsofthisbundle,i.e.anassignmenttoeachpointxofanelementinthefiberoverx(seeFig.3).Havingdefinedthefeaturespace,weneedtospecifyhowittransforms(e.g.sayhowavectorfieldonS2isrotated).Thenaturalwaytotrans-forma?-fieldisviatheinducedrepresentation?=IndG?ofG(Section2.4),whichcombinesHtheactionofGonthebasespaceBandtheac-tionof?onthefiberVtoproduceanactiononsectionsoftheassociatedbundle(SeeFigure3).Finally,havingdefinedthefeaturespacesandtheirtransformationlaws,wecanstudyequiv-ariantlinearmapsbetweenthem(Section3).InSec.4–6wecoverimplementationaspects,re-latedwork,andconcreteexamples,respectively.Figure3:mapsscalarfieldstovectorfields,andisequivarianttotheinducedrepresentation?i=ndSO(3)?i.SO(2)2FeatureSpacesFiberBundlesIntuitively,afiberbundleisaparameterizationofasetofisomorphicspaces(thefibers)byanotherspace(thebase).FeatureSpacesFiberBundlesIntuitively,afiberbundleisaparameterizationofasetofisomorphicspaces(thefibers)byanotherspace(thebase).Forexample,wecanthinkofafeaturespaceinaclassicalasasetofvectorspacesVx'Rn(nbeingthenumberofchannels),oneperpositionxintheplane[2].Thisisanexampleofatrivialbundle,becauseitissimplytheCartesianproductoftheplaneandRn.Generalfiberbundlesareonlylocallytrivial,meaningthattheylocallylooklikeaproductwhilehavingadifferentglobaltopologicalstructure.Thesimplestexampleofanon-trivialbundleistheMobiusstrip,whichlocallylookslikeaproductofthecircle(thebase)withaligment(thefiber),butisgloballydistinctfromacylinder(seeFig.4).Amorepracticallyrelevantexampleisgivenbythetangentbundleofthesphere(Fig.2),whichhasasbasespaceS2andfibersthatlooklikeR2,butisFigure4:CylinderandM?biusstriptopologicallydistinctfromS2?R2asabundle.Formally,abundleconsistsoftopologicalspacesE(totalspace),B(basespace),F(canonicalandaprojectionmapp:E!B,satisfyingalocaltrivialitycondition.Basically,thisconditionsaysthatlocally,thebundlelookslikeaproductU?FofapieceU?Bofthebasespace,andFthecanonicalfiber.Formally,theconditionisthatforeverya2E,thereisanopenneighbourhoodU?Bofp(a)andahomeomorphism':p1(U)!U?Fsothatthe'proj1mapp1(U) U?F!Uagreeswithp:p1(U)!U(whereproj(u,f)=u).Thehomeomorphim'issaidtolocallytrvializetheundleabethetrvializingneighbourhoodU.1Consideringthatforanyx2Uthepreimageproj11(x)isF,and'isahomeomorphism,weseethatthepreimageFx=p1(x)forx2BisalsohomeomorphictoF.Thus,wecallFxthefiberoverx,andseethatallfibersaomeomorphic.Knowingthis,wecandenoteabundlebyitsprojectionmapp:E!B,leavingthecanonicalfiberFimplicit.Variousmorerefinednotionsoffiberbundleexist,eachcorrespondingtoadifferentkindoffiber.Inthispaperwewillworkwithprincipalbundles(bundlesofgroups)andvectorbundles(bundlesofvectorspaces).Asectionsofafiberbundleisanassignmenttoeachx2Bofanelements(x)2Fx.Formally,itisamaps:B!Ethatsatisfiesps=idB.Ifthebundleistrivial,asectionisequivalenttoafunctionf:B!F,butforanon-trivialbundlewecannotcontinuouslyalignallthefiberssimultaneously,andsowemustkeepeachs(x)initsownfiberFx.Nevertheless,onatrivializingneighbourhoodU?B,wecandescribethesectionasafunctionU:U!F,bysetting'(s(x))=(x,U(x)).2.2GasaPrincipalH-BundleRecall(Sec.1.1)thatwitheveryfeaturespaceofaG-isassociatedahomogeneousspacethesphere,projectivespace,hyperbolicspace,Grassmann&Stiefelmanifolds,etc.),andrecallfurtherthatsuchaspacehasastabilizersubgroupH={g2G|go=o}(thisgroupbeingindependentoforiginouptoisomorphism).AsdiscussedinAppendixA,thecosetsgHofH(e.g.thecirclesinFig.1)partitionG,andthesetofcosets,denotedG/H(e.g.thesphereinFig.1),canbeidentifiedwithB(uptoachoiceoforigin).ItisthispartitioningofGintocosetsthatinducesaspecialkindofbundlestructureonG.Theprojectionmapthatdefinesthebundlestructuresendsanelementg2GtothecosetgHitbelongsto.Thus,itisamapp:G!G/H,andwehaveabundlewithtotalspaceG,basespaceG/HandcanonicalfiberH.Intuitively,thisallowsustothinkofGasabasespaceG/HwithacopyofHattachedateachpointx2G/H.ThecopiesofHaregluedtogetherinapotentiallytwistedmanner.ThisbundleiscalledaprincipalH-bundle,becausewehaveatransitiveandfixed-point groupactionG?H!Gthatpreservesthefibers.Thi ionisgivenbyrightmultiplication,g7!gh,whichpreservesfibersbecausep(gh)=ghH=gH=p(g).Thatis,byright-multiplyinganelementg2Gbyh2H,wegetanelementghthatisingeneraldifferentfromgbutisstillwithinthesamecoset(i.e.fiber).Thattheactionistransitiveand oncosetsfollowsimmediatelyfromthegroupaxioms.3Onecanthinkofaprincipalbundleasabundleofgeneralizedframesorgaugesrelativetowhichgeometricalqutiescanbeexpressednumerically.Underthisinterpretationthefiberatxisaspaceofgeneralizedframes,andtheactionbyHisachangeofframe.Forinstance,eachpointontheOnecanthinkofaprincipalbundleasabundleofgeneralizedframesorgaugesrelativetowhichgeometricalqutiescanbeexpressednumerically.Underthisinterpretationthefiberatxisaspaceofgeneralizedframes,andtheactionbyHisachangeofframe.Forinstance,eachpointonthecirclesinFig.1canbeidentifiedwitharight-handedorthogonalframe,andtheactionofSO(2)correspondstoarotationofthisframe.ThegroupHmayalsoincludeinternalsymmetries,suchascolorspacerotations,whichdonotrelateinanywaytothespatialdimensionsofB.InordertonumericallyrepresentafieldonsomeneighbourhoodU?G/H,weneedtochooseaframeforeachx2Uinacontinuousmanner.Thisisformalizedasasectionoftheprincipalbundle.Recallthatasectionofp:G!G/Hisamaps:G/H!Gthatsatisfiesps=idG/H.SincepprojectsgtoitscosetgH,thesectionchoosesarepresentatives(gH)2gHforeachcosetgH.Non-trivialprincipalbundlesdonothavecontinuousglobalsections,butwecanalwaysusealocalsectiononU?G/H,andrepresentafieldonoverlap localpatchescoveringG/H.AsidefromtherightactionofH,whichturnsGintoaprincipalH-bundle,wealsohavealeftactionofGonitself,aswellasanactionofGonthebasespaceG/H.Ingeneral,theactionofGonG/HdoesnotagreewiththeactiononG,inthatgs(x)6=s(gx),becausetheactiononincludesatwistofthefiber.Thistwistisdescribedbythefunctionh:G/H?G!Hdefinedbygs(x)=s(gx)h(x,g)(wheneverboths(x)ands(gx)aredefined).Thisfunctionwillbeusedinvariouscalculationsbelow.Wenotefortheinterestederthathsatisfiesthecocycleconditionh(x,g1g2)=h(g2x,g1)h(x,g2).2.3TheAssociatedVectorBundleFeaturespacesaredefinedasspacesofsectionsoftheassociatedvectorbundle,whichwewillnowdefine.Inphysics,asectionofanassociatedbundleissimplycalledafield.pTodefinetheassociatedvectorbundle,westartwiththeprincipalH-bundleG G/H,andessentiallyreplacethefibers(cosets)byectorspacesV.ThespaceV' arriesagroupRnrepresentation?ofHthatdescribesthetransformationbehaviourofthefeaturevectorsinVunderachangeofframe.Thesefeaturescouldforinstancetransformasascalar,avector,atensor,orsomeothergeometricalqu ty[2,6,8].Figure3showsanexampleofavectorfield(?(h)beinga2?2rotationmatrixinthiscase)andascalarfield(?(h)=1).ThefirststepinconstructingtheassociatedvectorbundleistotaketheproductG?V.Inthecontextofrepresentationlearning,wecanthinkofanelement(g,v)ofG?Vasafeaturevectorv2Vandanassociatedposevariableg Gthatdescribeshowthefeaturedetectorwassteeredtoobtainv.Forinstance,inaSpherical[10]onewouldrotateafilterbankbyg2SO(3)andmatchitwiththeinputtoobtainv.Ifweapplyatransformationh2Htogandsimultaneouslyapplyitsinversetov,wegetanequivalentelement(gh,?(h1)v).InaSpherical ,thiswouldcorrespondtoachangeinorientationofthefiltersbyh2SO(2).Soinordertocreatetheassociatedbundle,wetakethequotientoftheproductG?Vbythisaction:A=G??V=(G?V)/H.Inotherwords,theelementsofAareorbits,definedas[g,v]={(gh,?(h1)v)|h2H}.TheprojectionpA:A!G/HisdefinedaspA([g,v])=gH.Onemaycheckthatthisiswelldefined,i.e.independentoftheorbitrepresentativegof[g,v]=[gh,?(h1)v].Thus,theassociatedbundlehasbaseG/HandfiberV,meaningthatlocallyitlooksnotethattheassociatedbundleconstructionworksforanyprincipalH-bundle,nogjustp:G!G/H,whichsuggestsadirectionforfurthergeneralization[11].Afield(“stackoffeaturemaps”)isasectionoftheassociatedbundle,meaningthatitisamaps:G/H!Asuchthat??s=idG/H.Wewillrefertothespaceofsectionsoftheassociatedtwowaystoencodeasection:asfunctionsf:G!Vsubjecttoaconstraint,andaslocalfunctionsfromU?G/HtoV.Wewillnowdefineboth.2.3.1SectionsasMackeyFunctionsTheconstructionoftheassociatedbundleasaproductG?Vsubjecttoanequivalencerelationsuggestsawaytodescribesectionsconcretely:asectioncanberepresentedbyafunctionf:G!Vsubjecttotheequivarianceconditionf(gh)=?(h1)f(g).(1)4SuchfunctionsarecalledMackeyfunctions.TheyprovidearedundantencodingofasectionofA,byencodingthevalueofthesectionrelativetoanychoiceofframe/sectionoftheprincipalbundlesimultaneously,withtheequivarianceconstraintensuringconsistency.AlinearcombinationofMackeyfunctionsisaMackeyfunction,sotheyformavectorspace,whichwewillrefertoasIG.Mackeyfunctionsareeasytoworkwithbecausetheyallowaconcreteandglobaldescriptionofafield,buttheirredundancymakesthemunsuitableforcomputerimplementation.2.3.2LocalSectionsasFunctionsonG/HTheassociatedbundlehasbaseG/HandfiberV,solocally,wecandescribeasectionasanunconstrainedfunctionf:U!VSuchfunctionsarecalledMackeyfunctions.TheyprovidearedundantencodingofasectionofA,byencodingthevalueofthesectionrelativetoanychoiceofframe/sectionoftheprincipalbundlesimultaneously,withtheequivarianceconstraintensuringconsistency.AlinearcombinationofMackeyfunctionsisaMackeyfunction,sotheyformavectorspace,whichwewillrefertoasIG.Mackeyfunctionsareeasytoworkwithbecausetheyallowaconcreteandglobaldescriptionofafield,buttheirredundancymakesthemunsuitableforcomputerimplementation.2.3.2LocalSectionsasFunctionsonG/HTheassociatedbundlehasbaseG/HandfiberV,solocally,wecandescribeasectionasanunconstrainedfunctionf:U!VwhereU?G/Hisatrivializingneighbourhood(seeSec.2.1).WerefertothespaceofsuchsectionsasIC.Givenalocalsectionf2IC,wecanencodeitasaMackeyfunctionthroughthefollowingliftingisomorphism?:IC!IG:[?f](g)=?(h(g)1)f(gH),[?1f0](x)=f0(s(x)),(2)whe(g)=h(H,g)=s(gH)1g2Hands(x)2Gisacosetrepresentativeforx2G/H.Thismapisanalogoustotheliftingdefinedby1]forcalarfields(i.e.?(h)=I),andcanbedefinedmoregenerallyforanyprincipal/associatedbundle[13].2.4TheInducedRepresentationGTheinducedrepresentation?=Ind?describestheactionofGonfields.InI,itisdefinedas:GH[?G(g)f](k)=f(g1k).(3)InIC,wecandefinetheinducedrepresentation?ConalocalneighbourhoodUas[C(g)f](x)=?(h(g1,x)1)f(g1x).(4)Herewehaveassumedthathisdefinedat(g1,x).Ifitisnot,onewouldneedtochangetoadifferentsectionofG!G/H. Onemayverify,usingthecompositionlawforh(Sec. 2.2),thatEq. 4doesdefinearepresentationofG. Moreover,onemayverifythat?G(g)?=??C(g),i.e.theydefineisomorphicrepresentations.WecaninterpretEq.4asfollows.Totransformafield,wemovethefiberatg1xtox,andweapplyatrans-formationtothefiberitselfusing?.ThisisvisualizedinFig.5foraplanarvectorfield.Someotherexam-plesincludeanRGBimage?(h)=I3),afieldofwindf(g1x)?(g)f(g1x)f(x)directionsonearth(?(h)a2?2rotationmatrix),adiffu-Figure5:Therotationofaplanarvectorfieldintwosteps:movingeachvectortoitsnewpositionwithoutchangingitsori-entation,andthenrotatingthevectors.siontensorMRIimage(?(h)arepresentationofSO(3)actingon2-tensors),aregularG- onZ3[14,15](?aregularrepresentationofH).3 EquivariantMapsandConvolutionsEachfeaturespaceinaG-isdefinedasthespaceofsectionsofsomeassociatedvectorbundle,definedbyachoiceofbaseG/Handrepresentation?ofHthatdescribeshowthefiberstransform.AlayerinaG- isamapbetweenthesefeaturespacesthatisequivarianttotheinducedrepresentationingonthem.Inthissectionwewillshowthatequivariantlinearmapscanalwaysbewrittenasaconvolution-likeoperationusinganequivariantkernel.WewillfirstderivethisresultfortheinducedrepresentationrealizedinthespaceIGofMackeyfunctions,andthenconverttheresulttolocalsectionsoftheassociatedvectorbundleinSection3.2.WewillassumethatGislocallycompactandunimodular.Consideradjacentfeaturespacesi=1,2witharepresentation(i,i)ofHi .Let?i=ndG?iHibetherepresentationactingonI.AboundedlinearoperatorI!Icanbewrittenasi12GGG[·f](g)=Z (g,g0)f(g0)dg0,(5)G5usingatwo-argumentlinearoperator-valuedkernel :G?G!Hom(V1,V2),wheom(V1,V2)denotesthespaceoflinearmapsV1!V2.Choosingbases,wegetamatrix-valuedkernel.Weareinterestedinthespaceofequivariantlinearmapsbetweeninducedrepresentations,definedasH=usingatwo-argumentlinearoperator-valuedkernel :G?G!Hom(V1,V2),wheom(V1,V2)denotesthespaceoflinearmapsV1!V2.Choosingbases,wegetamatrix-valuedkernel.Weareinterestedinthespaceofequivariantlinearmapsbetweeninducedrepresentations,definedasH=HomG(I1,I2)={2Hom(I1,I2)|?1(g)=2(g),8g2G.InorderforEq.5todefineanequivariantmap2H,thekernel mustsatisfyaconstraint.By(partially)resolvingthisconstraint,wewillshowthatEq.5canalwaysbewrittenasacross-correlation1Theorem3.1.(convolutionisallyouneed)Anequivariantmap2Hcanalwaysbewrittenasaintegral.Proof.Sinceweareonlyinterestedinequivariantmaps,wegetaconstrainton.Forallu,g2G:[·[?1(u)f]](g)(g,g0)f(u1g0)dg0=[?2(u)[·f]](g)ZZ= (u1g,g0)f(g0)dg0,,,,GGZZ(6)(g,ug0)f(g0)dg0(g,ug0)(ug,ug0)= (u1g,g0)f(g0)dg0GG= (u1g,g0)= (g,g0)wecandefinethetwo-argumentkernel(·,·)intermsofaone-agumenternel:(g1g0)?(e,g1g0)=(ge,gg1g0)=(g,g0).Theapplicationof tofthusreducestoacross-correlation:[·f](g)=Z (g,g0)f(g0)dg0=Z (g1g0)f(g0)dg0=[?f](g).(7)GG3.1TheSpaceofEquivariantKernelsTheconstraintEq.6impliesaconstraintontheone-argumentkernel.Thespaceofadmissiblekernelsisinone-to-onecorrespondencewiththespaceofequivariantmaps.Herewegivethreedifferentcharacterizationsofthisspaceofkernels.edproofscanbefoundinAppendixB.Theorem3.2.Hisisomorphictothespaceofbi-equivariantkernelsonG,definedas:KG={:G!Hom(1,2)|(h2gh1)=?2(h2)(g)1(h1),8g2G,h12H1,h22H2}.(8)Proof.Itiseasilyverified(seesupp.mat.)thatrightequivariancefollowsfromthefactthatf12IGisaMackeyfunction,andleftequivariancefollowsfromtherequirementthat ?f2IGshouldbea2Macyfunction.TheisomorphismisgenbyG:KG!Hdefinedas[G]f= ?f.Theanalogousresultforthetwoargumentkernelisthat (gh2,g0h1)shouldbeequalto?2(h21)(g,g0)?1(h1)forg,g02G,h12H1,h22H2.Thishasthefollowinginterestingin-terpretation: isasectionofacertainassociatedbundle.Wedefinearight-actionofH1?H2onG?Gbysetting(g,g0)·(1,2)=(gh1,g0h2)andarepresentation?12ofH1?H2onHom(V1,2)bysetting?12(h1,2)=2(h2)1(h1)for2Hom(V1,V2).Thentheconstrainton(·,·)canbewittenas((g,g0)·(h1,2))=?12((h1,2)1)((g,g0)).erecognizethisastheconditionofbeingaMacyfunction(Eq.1)fortheundle(G?G)??12Hom(V1,2).Thereisanotheranotherwaytocharacterizethespaceofequivariantkernels:Theorem3.3.Hisisomorphictothespaceofleft-equivariantkernelsonG/H1,definedas:KC={:G/1!Hom(V1,2)|(h2x)=2(h2)(x)?1(h1(x,h2)1),(9)8h22H2,x2G/H1}1Asinmostoftheliterature,wewillnotbepreciseaboutdistinguishingconvolutionandcorrelation.6Proof.usingthedecompositiong=s(gH1)h1(g)(seeAppendixA),wecandefine(g)=(s(gH1)h1(g))=(s(g1))?1(h1(g))?(g1)1(h1(g)),(10)Thisdefinestheliftingisomorphismforernels,Proof.usingthedecompositiong=s(gH1)h1(g)(seeAppendixA),wecandefine(g)=(s(gH1)h1(g))=(s(g1))?1(h1(g))?(g1)1(h1(g)),(10)Thisdefinestheliftingisomorphismforernels,?K:KC!KG.Itiseasytoerifythatwhendefinedinthisway, satisfiesrightH1-equivariance.WestillhavetheleftH2-equivarianceconstraintfromEq.8,whichtranslatestoasfollows(insupp.mat.).Forg2G,h22H2andx2G/H1,(h2g)=?2(h2)(g),(2x)=2(h2)(x)?1(h1(x,h2)1).s(11)(x)H2Theorem3.4.HisisomorphictothespaceofH-equivariantkernelsonH\G/H:121x1K ={ˉ:H\G/H!Hom(V,V)|ˉ(x)=?(h)ˉ(x)?(h) ,D211 221(12)(x)H18x2H2\G/H1,h2H2 },Where:H\G/H!Gisachoiceofdoublecosetrepresentatives,and?isarepresentationofx211(x)H2thestabilizerH={h2H|h(x)H=(x)H} H1,definedas1211?x(h)=?1(h1((x)H1,h))=1((x)1h(x)),(13)1Proof.Insupplementarymaterial.Forexamples,seeSection6.3.2LocalSectionsonG/HWehaveseenthatanequivariantmapbetweenspacesofMackeyfunctionscanalwaysberealizedasacross-correlationonG,andwehavestudiedthepropertiesofthekernel,whichcanbeencodedasakernelonGorG/H1orH2\G/H1,subjecttotheappropriateconstraints.WhenimplementingaG- touseaMackeyfunctiononG,soweneedtounderstandwhatitmeansforfieldsrealizedbylocalfunctionsf:U!VforU?G/H1.Thisisdonebysandwichingthecross-correlation?:I!Iwiththeliftingisomorphisms?:I!I.1 2i iiG GC G[?1[?[?1f]]](x)=Z(s2(x)1s1(y))f(y)dyG(14)=Z(s2(x)1y)?1(h1(s2(x)1s1(y)))f(y)dyG/H1Whichwerefertoasthe?1-twistedcross-correlationonG/H1.Wenotethatforsemidirectproductgroups,the?1factordisappearsandweareleftwithastandardcross-correlationonG/H1withkernel2KC.Wenotethesimilarityofthisexpressiontogaugeequivariantconvolutionasdefinedin[11].3.3EquivariantNonlinearitiesThenetworkasawholeisequivarifallofitslayersareequivariant.Soourtheorywouldnotbecompletewithoutadiscussionofequivariantnonlinearitiesandotherkindsoflayers.InaregularG- [1],?istheregularrepresentationofH,whichmeansthatitcanberealizedbypermutationmatrices.Sincepermutationsandpointwisenonlinearitiescommute,anysuchnonlinearitycanbeused.orotherkindsofrepresentations,specialequariantnonlinearitiesmustbeused.Somechoicesincludenormnonlinearities[3]forunitaryrepresentations,tensorproductnonlinearities[8],orgatednonlinearitieswhereascalarfieldisnormalizedbyasigmoidandthenmultipliedbyanotherfield[6].Otherconstructions,suchasbatchnormandResNets,canalsobeequivariant[1,2].Acomp ensiveoverviewandcomparisonoverequivariantnonlinearitiescanbefoundin[7].74 ImplementationSeveraldifferentapproachestoimplementinggroupequivariantshavebeenproposedintheliterature.Theimplementationstherebydependonthespecificchoice4 ImplementationSeveraldifferentapproachestoimplementinggroupequivariantshavebeenproposedintheliterature.TheimplementationstherebydependonthespecificchoiceofsymmetrygroupG,thehomogeneousspaceG/H,itsdiscretizationandtherepresentation?.Inaycae,sincetheequivarianceconstraintsonconvolutionkernelsarelinear,thespaceofH-equivariantkernelsisalinearsubspaceoft