LauraSánchezGarcíaJulioAntonioSotoVicenteIEUniversity(C4__466671-AdvancedArtificialIntelligence)IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall20241/841Intro2DenoisingDiffusionProbabilisticModels3Advancementsandimprovements4Largediffusionmodels5BeyondimagegenerationIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall20242/84IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall20243/84A(certainlynotcomplete)list:?LatentVariablemodels(incl.VAEs)?Autoregressivemodels(incl.GPT-styleLanguageModels)?GANs?Flow-basedmodels(incl.NormalizingFlows)?Energy-BasedModels(incl.Score-basedmodels)?Diffusionmodels(kindofmixofallpreviouspoints)?CombinationsIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall20244/84ImageImagegenerationSource:Hoelal.[2020]”AphotoofaCorgidogridingabikeinTimesSquare.Itiswearingsunglassesandabeachhat.”Source:Sahariaetal.[2022]IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall20245/84Source:Brooksetal.[2024]IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall20246/84IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall20247/84Outline?Theforwardprocess?TheNice?property?Thereverseprocess?Lossfunction?Trainingalgorithm?Themodel?SamplingalgorithmIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall20248/84IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall20249/84”Creatingnoisefromdataiseasy;creatingdatafromnoiseisgenerativemodeling.”Songetal.,2020IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202410/84”Creatingnoisefromdataiseasy;creatingdatafromnoiseisgenerativemodeling.”Songetal.,2020DDPMsprogressivelygenerateimagesoutofnoiseSource:Hoetal.[2020]IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202410/84Inorderforamodeltolearnhowtodothat,weneedaprocesstogeneratesuitabletrainingdataintheformof(noise,image)pairsIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202411/84Inorderforamodeltolearnhowtodothat,weneedaprocesstogeneratesuitabletrainingdataintheformof(noise,image)pairsWewillrefertotheprocessofcreatingthetrainingdataastheforwarddiffusionprocess,whichwillprogressivelymakeanimagenoisierIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202411/84Inorderforamodeltolearnhowtodothat,weneedaprocesstogeneratesuitabletrainingdataintheformof(noise,image)pairsWewillrefertotheprocessofcreatingthetrainingdataastheforwarddiffusionprocess,whichwillprogressivelymakeanimagenoisierIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202411/84Ourmodelwilllearnhowtorevertthatnoisingprocess,throughwhatitisknownasthereversediffusionprocess→progressivelydenoisinganoisyimageIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202412/84Ourmodelwilllearnhowtorevertthatnoisingprocess,throughwhatitisknownasthereversediffusionprocess→progressivelydenoisinganoisyimageIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202412/84Ourmodelwilllearnhowtorevertthatnoisingprocess,throughwhatitisknownasthereversediffusionprocess→progressivelydenoisinganoisyimageOncetrained,themodelshouldhavethereforelearnedhowtodenoiseimagesSowecangeneratesomepurelyrandomnoise,runitthroughourmodelandgetbackanimagegeneratedoutofpurenoise!IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202412/84Moreformally,DDPMsworkthroughmanystepstwhichare0,1,…,T?x0istheoriginalimage?q(xt∣xt?1)istheforwarddiffusionprocess?pθ(xt?1∣xt)willbethereversediffusionprocess(learnedbyourmodelwithweightsθ)IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202413/84Moreformally,DDPMsworkthroughmanystepstwhichare0,1,…,T?x0istheoriginalimage?q(xt∣xt?1)istheforwarddiffusionprocess?pθ(xt?1∣xt)willbethereversediffusionprocess(learnedbyourmodelwithweightsθ)DuringforwarddiffusionweaddGaussian(Normal)noisetotheimageineveryt,producingnoisyimagesx1,x2,…xTAstbecomeshigher,theimagebecomesmoreandmorenoisyIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202413/84q(xt∣xt?1)?√1?βt?xt?1+N(0,βtI)IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202414/84q(xt∣xt?1)?√1?βt?xt?1+N(0,βtI)Takeanimageatsomepointt?1GenerateGaussiannoisefromanisotropicmultivariateNormalofsizext,Scalext?1valuesby√1?βt(sodatascaledoesnotgrowasweaddnoise)AddthenoisethethescaledimageIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202414/84q(xt∣xt?1)?√1?βt?xt?1+N(0,βtI)Takeanimageatsomepointt?1GenerateGaussiannoisefromanisotropicmultivariateNormalofsizext,Scalext?1valuesby√1?βt(sodatascaledoesnotgrowasweaddnoise)AddthenoisethethescaledimageItcanbedirectlycomputedasq(xt∣xt?1)?N(xt;√1?βtxt?1,βtI)IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202414/84q(xt∣xt?1)?√1?βt?xt?1+N(0,βtI)Takeanimageatsomepointt?1GenerateGaussiannoisefromanisotropicmultivariateNormalofsizext,Scalext?1valuesby√1?βt(sodatascaledoesnotgrowasweaddnoise)AddthenoisethethescaledimageItcanbedirectlycomputedasq(xt∣xt?1)?N(xt;√1?βtxt?1,βtI)noiseisaddedineachstept11Inthepaperitismadetogrowlinearlyfromβ1=10?4toβT=0.02forT=1000IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202414/84Thefullforwardprocessistherefore:Tq(x1:T|x0)污Πq(xt|xt—1)t=1ForalargeT,thefinalimageisbasicallyonlynoise(alloriginalimageinfoisessentiallylost),soitbecomesroughlyxT~N(0,I)Demohere!IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202415/84Tricktogetanyxtfromx0withouthavingtocomputetheintermediatesteps.Παs.WecanusethereparametrizationtrickfortheNormaldistributiontoget:q(xt|x0)=N(xt;√α- Inthepaperthisisdescribedas”Anotableproperty”.IbelievethatthefirsttocallthisasanicepropertywasWeng[2021].WewillcallittheNice?propertyIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202416/84Tricktogetanyxtfromx0withouthavingtocomputetheintermediatesteps.Παs.WecanusethereparametrizationtrickfortheNormaldistributiontoget:DetailsinAppendixA!q(xt|x0)=N(xt;√α-DetailsinAppendixA!?Easier,fastercomputation?Anyimagestatextcomesfroma(Normal)probabilitydistribution,drasticallysimplyfingderivationsDemohere!Inthepaperthisisdescribedas”Anotableproperty”.IbelievethatthefirsttocallthisasanicepropertywasWeng[2021].WewillcallittheNice?propertyIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202416/84WewilltrainamodelpθtolearntoperformthereverseprocessStartingfromp(xT)=N(0,I),itwilltrytorecreatetheimage!IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202417/84WewilltrainamodelpθtolearntoperformthereverseprocessStartingfromp(xT)=N(0,I),itwilltrytorecreatetheimage!pθ(xt1|xt)污N(xt1;μθ(xt,t),Σθ(xt,t))WillbeaneuralnetworkpredictionWillbesettoavalueσIbasedonβtIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202417/84WewilltrainamodelpθtolearntoperformthereverseprocessStartingfromp(xT)=N(0,I),itwilltrytorecreatetheimage!pθ(xt1|xt)污N(xt1;μθ(xt,t),Σθ(xt,t))WillbeaneuralnetworkpredictionWillbesettoavalueσIbasedonβtAndTpθ(x0:T)污p(xT)Πpθ(xt—1|xt)t=1IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202417/84SummarySummaryIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202418/84SummarySummaryTheforwardprocessposterioristheground-truthreversediffusionprocessthatthemodelwilllearntoapproximate!IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202418/84JustlikeinVAEs,thelossfunctionisbasedontheEvidenceLowerBound(ELBO):ELBO=Eq(x1∶T∣x0)[logIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202419/84JustlikeinVAEs,thelossfunctionisbasedontheEvidenceLowerBound(ELBO):ELBO=Eq(x1∶T∣x0)[logWhichbecomes:?T?EqIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202419/84JustlikeinVAEs,thelossfunctionisbasedontheEvidenceLowerBound(ELBO):ELBO=Eq(x1∶T∣x0)[logWhichbecomes:?T?EqIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202419/84JustlikeinVAEs,thelossfunctionisbasedontheEvidenceLowerBound(ELBO):ELBO=Eq(x1∶T∣x0)[logWhichbecomes:?T?EqDetailsDetailsinAppendixB!IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202419/84JustlikeinVAEs,thelossfunctionisbasedontheEvidenceLowerBound(ELBO):ELBO=Eq(x1∶T∣x0)[logWhichbecomes:?T?Eq?LT→priormatchingterm.Hasnolearnableparameters,soDetailsinAppendixB!?LtDetailsinAppendixB!?L0→reconstructionterm.Onlylearninghowtogofromx1tox0,soauthorsendedupignoringit(simplerandbetterresults)IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202419/84LossthereforefocusesonLt?1:Where:?q(xt?1∣xt,x0)istheforwardprocessposterior(i.e.whatwouldbetheprefect,ground-truthreverseprocess)conditionedonx0?pθ(xt?1∣xt)willbeourlearnedreverseprocessasseeninslide17 IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202420/84LossthereforefocusesonLt?1:Where:?q(xt?1∣xt,x0)istheforwardprocessposterior(i.e.whatwouldbetheprefect,ground-truthreverseprocess)conditionedonx0xt?1∣xt)willbeourlearnedreverseprocessasseeninslide17Theforwardprocessposterioristractableandcanbecomputedas:?q(xt?1∣xt,x0)=N(xt?1;t(xt,x0),βtI)WhereIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202420/84LossthereforefocusesonLt?1:Where:?q(xt?1∣xt,x0)istheforwardprocessposterior(i.e.whatwouldbetheprefect,ground-truthreverseprocess)conditionedonx0xt?1∣xt)willbeourlearnedreverseprocessasseeninslide17Theforwardprocessposterioristractableandcanbecomputedas:?q(xt?1∣xt,x0)=N(xt?1;t(xt,x0),βtI)WhereDetailsinAppendixC!DetailsinAppendixC!IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202420/84LossthereforefocusesonLt?1:Where:?q(xt?1∣xt,x0)istheforwardprocessposterior(i.e.whatwouldbetheprefect,ground-truthreverseprocess)conditionedonx0xt?1∣xt)willbeourlearnedreverseprocessasseeninslide17Theforwardprocessposterioristractableandcanbecomputedas:?q(xt?1∣xt,x0)=N(xt?1;t(xt,x0),βtI)WhereDetailsinAppendixC!DetailsinAppendixC!IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202420/84LossisthereforetheKLdivergencebetweentwoNormals:theforwardprocessposteriorq(xt?1∣xt,x0)andthereverseprocessthatourmodelwilllearnIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202421/84LossisthereforetheKLdivergencebetweentwoNormals:theforwardprocessposteriorq(xt?1∣xt,x0)andthereverseprocessthatourmodelwilllearnSincebothareNormaldistributions,thisKLdivergenceis: Eqt(xt,x0)?μθ(xt,t)∥]Forwardprocessposteriormean(slide20) Frommodel’sprediction(slide17)IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202421/84LossisthereforetheKLdivergencebetweentwoNormals:theforwardprocessposteriorq(xt?1∣xt,x0)andthereverseprocessthatourmodelwilllearnpxt?1∣xt)SincebothareNormaldistributions,thisKLdivergenceis: ]Forwardprocessposteriormean(slide20) Frommodel’sprediction(slide17)However:authorsdecideinsteadtopredictthenoiseaddedduringtheforwardprocess.Reformulating: Addednoiseinforwardpass ModelpredictingthenoiseusingxtandtasfeaturesIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202421/84LossisthereforetheKLdivergencebetweentwoNormals:theforwardprocessposteriorq(xt?1∣xt,x0)andthereverseprocessthatourmodelwilllearnpxt?1∣xt)SincebothareNormaldistributions,thisKLdivergenceis: ]Forwardprocessposteriormean(slide20) Frommodel’sprediction(slide17)However:authorsdecideinsteadtopredictthenoiseaddedduringtheforwardprocess.Reformulating: process.Reformulating:Addednoiseinforwardpass ModelpredictingthenoiseusingxtandtasfeaturesDetailsinAppendixD!IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202421/84LossisthereforetheKLdivergencebetweentwoNormals:theforwardprocessposteriorq(xt?1∣xt,x0)andthereverseprocessthatourmodelwilllearnpxt?1∣xt)SincebothareNormaldistributions,thisKLdivergenceis: ]Forwardprocessposteriormean(slide20) Frommodel’sprediction(slide17)However:authorsdecideinsteadtopredictthenoiseaddedduringtheforwardprocess.Reformulating: Addednoiseinforwardpass ModelpredictingthenoiseusingxtandtasfeaturesDetailsinAppendixD!IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202421/84Source:Hoetal.[2020]IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202422/84Source:Hoetal.[2020]Where√tx0+√1+t∈isjustxtcomputedthroughtheNice?property!IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202422/84ProposedmodelisaU-Netarchitecture(Ronnebergeretal.[2015])thatincludesself-attentionblocks IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202423/84ProposedmodelisaU-Netarchitecture(Ronnebergeretal.[2015])thatincludesself-attentionblocksTheyalsoincludeGroupNorm(WuandHe[2018])inResNetandself-attentionblockstisaddedoneveryResNetblockthroughpositionalencoding(Vaswanietal.[2017])IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202423/84Oncethemodelistrained,wecangeneratenewimagesby:Source:Hoetal.[2020] IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202424/84Oncethemodelistrained,wecangeneratenewimagesby:Source:Hoetal.[2020]Step4justappliesthereparametrizationtricktothelearnedreverseprocesspθ(xt?1∣xt)=N(xt?1;μθ(xt,t),Σθ(xt,t))IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202424/84Samplingisaniterativeprocess:weprogressivelyremovepredictednoise IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202425/84Samplingisaniterativeprocess:weprogressivelyremovepredictednoiseYoumaywonder:Ifatanysinglestepwearepredictingthefulladdednoise∈,whydon’tweremoveitcompletelyinasinglestep?IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202425/84Samplingisaniterativeprocess:weprogressivelyremovepredictednoiseYoumaywonder:Ifatanysinglestepwearepredictingthefulladdednoise∈,whydon’tweremoveitcompletelyinasinglestep?Answer:DetailsDetailsinAppendixE!IEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202425/84AdvancementsandimprovementsIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202426/84AdvancementsAdvancementsandimprovementsOutline?Variance/noiseschedulers?Learningthereverseprocessvariance?Fastersampling:DDIMs?Conditionalgeneration?ClassifierGuidance?Classifier-FreeGuidance?Conditioningonimages?ControlNet?ConditioningontextIEUniversity-L.Sánchez,J.A.SotoDiffusionmodelsFall202427/84Variance/noiseVariance/noiseschedulersNicholandDhariwal[2021]propot=beingtinywhentiscloseto0(theysetittos=0.008)ComparisonbetweenschedulerinDDPMandNichol&Dhariwal’scosineschedulerproposal.Source:Nichol&Dhariwal[2021]IEUniversity-L.Sánchez,J.A.Soto