Denoising Diffusion Probabilistic Models Jonathan Ho Ajay Jain Pieter Abbeel UC Berkeley UC Berkeley UC Berkeley jonathanho@berkeley.edu ajayj@berkeley.edu
--- title: "Denoising Diffusion Probabilistic Models Jonathan Ho Ajay Jain Pieter Abbeel UC Berkeley UC Berkeley UC Berkeley jonathanho@berkeley.edu ajayj@berkeley.edu" aliases: - "Diffusion" - "arXiv:2006.11239" source: "https://arxiv.org/abs/2006.11239" arxiv: "2006.11239" created: 2026-07-16 type: paper-translation status: extraction-complete_translation-pending tags: - paper - ml - deep-learning - vision
Denoising Diffusion Probabilistic Models Jonathan Ho Ajay Jain Pieter Abbeel UC Berkeley UC Berkeley UC Berkeley jonathanho@berkeley.edu ajayj@berkeley.edu
原文全文
Page 1
<a id="S0001"></a> Source: p.1 S0001
Denoising Diffusion Probabilistic Models Jonathan Ho Ajay Jain Pieter Abbeel UC Berkeley UC Berkeley UC Berkeley jonathanho@berkeley.edu ajayj@berkeley.edu pabbeel@cs.berkeley.edu Abstract We present high quality image synthesis results using diffusion probabilistic models, a class of latent variable models inspired by considerations from nonequilibrium thermodynamics.
<a id="S0002"></a> Source: p.1 S0002
Our best results are obtained by training on a weighted variational bound designed according to a novel connection between diffusion probabilistic models and denoising score matching with Langevin dynamics, and our models naturally admit a progressive lossy decompression scheme that can be interpreted as a generalization of autoregressive decoding.
<a id="S0003"></a> Source: p.1 S0003
On the unconditional CIFAR10 dataset, we obtain an Inception score of 9.46 and a state-of-the-art FID score of 3.17.
<a id="S0004"></a> Source: p.1 S0004
On 256x256 LSUN, we obtain sample quality similar to ProgressiveGAN.
<a id="S0005"></a> Source: p.1 S0005
Our implementation is available at https://github.com/hojonathanho/diffusion. 1 Introduction Deep generative models of all kinds have recently exhibited high quality samples in a wide variety of data modalities.
<a id="S0006"></a> Source: p.1 S0006
Generative adversarial networks (GANs), autoregressive models, flows, and variational autoencoders (VAEs) have synthesized striking image and audio samples [14, 27, 3, 58, 38, 25, 10, 32, 44, 57, 26, 33, 45], and there have been remarkable advances in energy-based modeling and score matching that have produced images comparable to those of GANs [11, 55].
<a id="S0007"></a> Source: p.1 S0007
Figure 1: Generated samples on CelebA-HQ 256 × 256 (left) and unconditional CIFAR10 (right) 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada. 0202 ceD 61 ]GL.sc[ 2v93211.6002:viXra
Page 2
<a id="S0008"></a> Source: p.2 S0008
x T ! · · · ! x t ! <latexit sha1_base64="XVzP503G8Ma8Lkwk3KKGZcZJbZ0=">AAACEnicbVC7SgNBFJ2Nrxhfq5Y2g0FICsNuFEwZsLGMYB6QLMvsZDYZMvtg5q4Y1nyDjb9iY6GIrZWdf+Mk2SImHrhwOOde7r3HiwVXYFk/Rm5tfWNzK79d2Nnd2z8wD49aKkokZU0aiUh2PKKY4CFrAgfBOrFkJPAEa3uj66nfvmdS8Si8g3HMnIAMQu5zSkBLrlmO3R4MGZBSLyAw9Pz0YeKmcG5P8CNekKDsmkWrYs2AV4mdkSLK0HDN714/oknAQqCCKNW1rRiclEjgVLBJoZcoFhM6IgPW1TQkAVNOOntpgs+00sd+JHWFgGfq4kRKAqXGgac7p0eqZW8q/ud1E/BrTsrDOAEW0vkiPxEYIjzNB/e5ZBTEWBNCJde3YjokklDQKRZ0CPbyy6ukVa3YF5Xq7WWxXsviyKMTdIpKyEZXqI5uUAM1EUVP6AW9oXfj2Xg1PozPeWvOyGaO0R8YX7+bCp4F</latexit> p ✓ ( xt 1 | ! xt) xt 1 ! · · · ! x 0 <latexit sha1_base64="l4LvSgM7PR7I/kkuy5soikK4gpU=">AAAEoXictVLditNAFE7XqGv92a5eejOYLexKLU0VFKRQ9EYvhCrb3YUklOlk2g6dnzBzYrcb8zK+lU/gazhJK6atuiB4YODM+T/n+8YJZwY6nW+1vRvuzVu39+/U7967/+CgcfjwzKhUEzokiit9McaGcibpEBhwepFoisWY0/Px/G3hP/9MtWFKnsIyoZHAU8kmjGCwplHjeygwzAjThNM4Kz/jSXaZj05zFHIlp5pNZ4C1VgsUkliB2TX/oQLYCpe/4rJwZhJM6NPMJyLPt9IM0SwBA0tOUaVGBs/8/J8mWVRH6eSjhtdpd0pBu4q/VjxnLYPR4d7XMFYkFVQC4diYwO8kEGVYA7P183qYGmr3meMpDawqsaAmykpEctS0lhhNlLZPAiqt1YwMC2OWYmwjiynNtq8w/s4XpDB5FWVMJilQSVaNJilHoFABL4qZpgT40irYntTOisgMa0zAkqC+0QbY/MquIfCcYssbsBH1UNIFUUJgGVePGfhR1qyj1YETXAaH/SqAnp836/lGftUfdNcFiqbBT8L2jouQdvE9iVAoVUyDWONFa5XVYlJSjezEPT+BlmCSiVQgw65or2vBaE0Y5z1e4D/VeBmhstwJyo5C0YeZ53vdo/z19lhVjly71+K6xRb/ZbO/rbLCS8HMwmVZ7W9zeFc567b95+3uxxde/82a3/vOY+eJc+z4zkun77xzBs7QIbUPNVP7Ustdz33vDtxPq9C92jrnkbMhbvAD81mObw==</latexit> >tixetal/<=o5kGOXA8nPw+jg58xpF0iV+df+Fc65c4dlh4JDBJAKNtMMqLdIwOTQ4OGxReUn+Vexdrd1nq3qVS+6LLjfIyWlZGbSFISWRvZGKYWSswTsoY2QALRDSIPOKiWMkUNDNJSYwXREzdn3/JuZ26ZIbTsDaJdcrTIoY7pfugK8WqgEoZY1pF5zHZCTdCBnKa1eMCLjYam2lpm20EGWlCCmmX67utOGhwg6SkA5NreffakjAkeu4VB6c6taxut/HlsiBHuYFIW6x56m+V2ogxYxCcIRDpY8bxlkmVNg98zc3csTaInUxVkk2wxMNanAnmGbCVvkkkodEMt8ShFs7Lb/92Z3lupC3Y/9WYN1tejn2nlShmbWG9OOwBXPte0up3wv3p0vXiMYxi/l0UvKJW2kmbYXayKV7BsfBjVM6xiuKdXJPRA2ljWNd+V7Xt2JExMwSNBVbciH+BAAA>"=g/Sbcz2lT7cvIVd5PuyPY0nrFy7"=46esab_1ahs tixetal< q <latexit sha1_base64="eAZ87UuTmAQoJ4u19RGH5tA+bCI=">AAACC3icbVC7TgJBFJ31ifhatbSZQEywkOyiiZQkNpaYyCMBspkdZmHC7MOZu0ay0tv4KzYWGmPrD9j5N87CFgieZJIz59ybe+9xI8EVWNaPsbK6tr6xmdvKb+/s7u2bB4dNFcaSsgYNRSjbLlFM8IA1gINg7Ugy4ruCtdzRVeq37plUPAxuYRyxnk8GAfc4JaAlxyzclbo+gaHrJQ8TB/AjnvsmcGZPTh2zaJWtKfAysTNSRBnqjvnd7Yc09lkAVBClOrYVQS8hEjgVbJLvxopFhI7IgHU0DYjPVC+Z3jLBJ1rpYy+U+gWAp+p8R0J8pca+qyvTRdWil4r/eZ0YvGov4UEUAwvobJAXCwwhToPBfS4ZBTHWhFDJ9a6YDokkFHR8eR2CvXjyMmlWyvZ5uXJzUaxVszhy6BgVUAnZ6BLV0DWqowai6Am9oDf0bjwbr8aH8TkrXTGyniP0B8bXL+1hmu8=</latexit> (xt| xt
<a id="S0009"></a> Source: p.2 S0009
1) Figure 2: The directed graphical model considered in this work.
<a id="S0010"></a> Source: p.2 S0010
This paper presents progress in diffusion probabilistic models [53]. A diffusion probabilistic model (which we will call a “diffusion model” for brevity) is a parameterized Markov chain trained using variational inference to produce samples matching the data after finite time.
<a id="S0011"></a> Source: p.2 S0011
Transitions of this chain are learned to reverse a diffusion process, which is a Markov chain that gradually adds noise to the data in the opposite direction of sampling until signal is destroyed.
<a id="S0012"></a> Source: p.2 S0012
When the diffusion consists of small amounts of Gaussian noise, it is sufficient to set the sampling chain transitions to conditional Gaussians too, allowing for a particularly simple neural network parameterization.
<a id="S0013"></a> Source: p.2 S0013
Diffusion models are straightforward to define and efficient to train, but to the best of our knowledge, there has been no demonstration that they are capable of generating high quality samples.
<a id="S0014"></a> Source: p.2 S0014
We show that diffusion models actually are capable of generating high quality samples, sometimes better than the published results on other types of generative models (Section 4).
<a id="S0015"></a> Source: p.2 S0015
In addition, we show that a certain parameterization of diffusion models reveals an equivalence with denoising score matching over multiple noise levels during training and with annealed Langevin dynamics during sampling (Section 3.2) [55, 61].
<a id="S0016"></a> Source: p.2 S0016
We obtained our best sample quality results using this parameterization (Section 4.2), so we consider this equivalence to be one of our primary contributions.
<a id="S0017"></a> Source: p.2 S0017
Despite their sample quality, our models do not have competitive log likelihoods compared to other likelihood-based models (our models do, however, have log likelihoods better than the large estimates annealed importance sampling has been reported to produce for energy based models and score matching [11, 55]).
<a id="S0018"></a> Source: p.2 S0018
We find that the majority of our models’ lossless codelengths are consumed to describe imperceptible image details (Section 4.3).
<a id="S0019"></a> Source: p.2 S0019
We present a more refined analysis of this phenomenon in the language of lossy compression, and we show that the sampling procedure of diffusion models is a type of progressive decoding that resembles autoregressive decoding along a bit ordering that vastly generalizes what is normally possible with autoregressive models. 2 Background (cid:82) Diffusion models [53] are latent variable models of the form p (x ) := p (x ) dx , where θ 0 θ 0:T 1:T x , . . . , x are latents of the same dimensionality as the data x q(x ).
<a id="S0020"></a> Source: p.2 S0020
The joint distribution 1 T 0 0 ∼ p (x ) is called the reverse process, and it is defined as a Markov chain with learned Gaussian θ 0:T transitions starting at p(x ) = (x ; 0, I): T T N T (cid:89) p (x ) := p(x ) p (x x ), p (x x ) := (x ; µ (x , t), Σ (x , t)) (1) θ 0:T T θ t − 1 | t θ t − 1 | t N t − 1 θ t θ t t=1 What distinguishes diffusion models from other types of latent variable models is that the approximate posterior q(x x ), called the forward process or diffusion process, is fixed to a Markov chain that 1:T 0 | gradually adds Gaussian noise to the data according to a variance schedule β , . . . , β : 1 T T (cid:89) (cid:112) q(x x ) := q(x x ), q(x x ) := (x ; 1 β x , β I) (2) 1:T 0 t t 1 t t 1 t t t 1 t | | − | − N − − t=1 Training is performed by optimizing the usual variational bound on negative log likelihood: (cid:20) (cid:21) (cid:20) (cid:21) E [ log p θ (x 0 )] E q log p θ (x 0:T ) = E q log p(x T ) (cid:88) log p θ (x t − 1 | x t ) =: L (3) − ≤ − q(x x ) − − q(x x ) 1:T 0 t t 1 | t 1 | − ≥ The forward process variances β can be learned by reparameterization [33] or held constant as t hyperparameters, and expressiveness of the reverse process is ensured in part by the choice of Gaussian conditionals in p (x x ), because both processes have the same functional form when θ t 1 t β are small [53]. A notable p−ro | perty of the forward process is that it admits sampling x at an t t arbitrary timestep t in closed form: using the notation α := 1 β and α¯ := (cid:81)t α , we have t − t t s=1 s q(x x ) = (x ; √α¯ x , (1 α¯ )I) (4) t 0 t t 0 t | N − 2
Page 3
<a id="S0021"></a> Source: p.3 S0021
Efficient training is therefore possible by optimizing random terms of L with stochastic gradient descent.
<a id="S0022"></a> Source: p.3 S0022
Further improvements come from variance reduction by rewriting L (3) as: (cid:20) (cid:21) (cid:88) E D (q(x x ) p(x )) + D (q(x x , x ) p (x x )) log p (x x ) (5) q KL T 0 T KL t 1 t 0 θ t 1 t θ 0 1 (cid:124) | (cid:123)(cid:122) (cid:107) (cid:125) (cid:124) − | (cid:123)(cid:122) (cid:107) − | (cid:125) − (cid:124) (cid:123)(cid:122) | (cid:125) t>1 LT Lt 1 L0 − (See Appendix A for details.
<a id="S0023"></a> Source: p.3 S0023
The labels on the terms are used in Section 3.) Equation (5) uses KL divergence to directly compare p (x x ) against forward process posteriors, which are tractable θ t 1 t when conditioned on x : − | 0 q(x x , x ) = (x ; µ˜ (x , x ), β˜ I), (6) t − 1 | t 0 N t − 1 t t 0 t √α¯ β √α (1 α¯ ) 1 α¯ where µ˜ t (x t , x 0 ) := 1 t − α¯ 1 t x 0 + t 1 − α¯ t − 1 x t and β˜ t := 1 − α¯ t − 1 β t (7) t t t − − − Consequently, all KL divergences in Eq. (5) are comparisons between Gaussians, so they can be calculated in a Rao-Blackwellized fashion with closed form expressions instead of high variance Monte Carlo estimates. 3 Diffusion models and denoising autoencoders Diffusion models might appear to be a restricted class of latent variable models, but they allow a large number of degrees of freedom in implementation.
<a id="S0024"></a> Source: p.3 S0024
One must choose the variances β of the t forward process and the model architecture and Gaussian distribution parameterization of the reverse process.
<a id="S0025"></a> Source: p.3 S0025
To guide our choices, we establish a new explicit connection between diffusion models and denoising score matching (Section 3.2) that leads to a simplified, weighted variational bound objective for diffusion models (Section 3.4).
<a id="S0026"></a> Source: p.3 S0026
Ultimately, our model design is justified by simplicity and empirical results (Section 4).
<a id="S0027"></a> Source: p.3 S0027
Our discussion is categorized by the terms of Eq. (5). 3.1 Forward process and L T We ignore the fact that the forward process variances β are learnable by reparameterization and t instead fix them to constants (see Section 4 for details).
<a id="S0028"></a> Source: p.3 S0028
Thus, in our implementation, the approximate posterior q has no learnable parameters, so L is a constant during training and can be ignored. T 3.2 Reverse process and L 1:T 1 − Now we discuss our choices in p (x x ) = (x ; µ (x , t), Σ (x , t)) for 1 < t T .
<a id="S0029"></a> Source: p.3 S0029
First, we set Σ (x , t) = σ2I to untrain θ ed t t−im 1 | e d t epen N dent t −co 1 nst θ ants t .
<a id="S0030"></a> Source: p.3 S0030
Expe θ rim t entally, both σ2≤ = β and θ t t t t σ se t 2 co = nd β˜ t is = op 1 ti − 1 m− α¯ a α t ¯− l t 1 fo β r t x had d s e i t m er i m la i r n r is e t s i u c l a t l s l . y T se h t e t fi o rs o t n c e h p o o ic in e t. is T o h p e ti s m e a a l re fo t r h x e 0 tw ∼ o N ex ( t 0 re , m I) e , c a h n o d ic th e e s 0 corresponding to upper and lower bounds on reverse process entropy for data with coordinatewise unit variance [53].
<a id="S0031"></a> Source: p.3 S0031
Second, to represent the mean µ (x , t), we propose a specific parameterization motivated by the θ t following analysis of L .
<a id="S0032"></a> Source: p.3 S0032
With p (x x ) = (x ; µ (x , t), σ2I), we can write: t θ t − 1 | t N t − 1 θ t t (cid:20) (cid:21) 1 L = E µ˜ (x , x ) µ (x , t) 2 + C (8) t − 1 q 2σ t 2 (cid:107) t t 0 − θ t (cid:107) where C is a constant that does not depend on θ.
<a id="S0033"></a> Source: p.3 S0033
So, we see that the most straightforward parameterization of µ is a model that predicts µ˜ , the forward process posterior mean.
<a id="S0034"></a> Source: p.3 S0034
However, we can expand θ t Eq. (8) further by reparameterizing Eq. (4) as x (x , (cid:15)) = √α¯ x + √1 α¯ (cid:15) for (cid:15) (0, I) and t 0 t 0 t − ∼ N applying the forward process posterior formula (7): (cid:34) (cid:13) (cid:18) (cid:19) (cid:13)2 (cid:35) L t − 1 − C = E x0,(cid:15) 2σ 1 t 2 (cid:13) (cid:13) (cid:13) µ˜ t x t (x 0 , (cid:15)), √ 1 α¯ t (x t (x 0 , (cid:15)) − √1 − α¯ t (cid:15)) − µ θ (x t (x 0 , (cid:15)), t) (cid:13) (cid:13) (cid:13) (9) (cid:34) (cid:13) (cid:18) (cid:19) (cid:13)2 (cid:35) = E x0,(cid:15) 2σ 1 2 (cid:13) (cid:13) (cid:13) √ 1 α x t (x 0 , (cid:15)) − √1 β t α¯ (cid:15) − µ θ (x t (x 0 , (cid:15)), t) (cid:13) (cid:13) (cid:13) (10) t t − t 3
Page 4
<a id="S0035"></a> Source: p.4 S0035
Algorithm 1 Training Algorithm 2 Sampling 1: repeat 1: x ∼ N (0, I) T 2: x ∼ q(x ) 0 0 2: for t = T, . . . , 1 do 3: t ∼ Uniform({1, . . . , T }) 3: z ∼ N (0, I) if t > 1, else z = 0 4: (cid:15) ∼ N (0, I) (cid:16) (cid:17) 5: Take ∇ g θ ra (cid:13) (cid:13) d (cid:15) ie − nt (cid:15) d θ e ( s √ ce α¯ n t t x s 0 te + p o √ n 1 − α¯ t (cid:15), t) (cid:13) (cid:13) 2 4 5 : : en x d t− f 1 or = √1 αt x t − √1 1 − − α α¯ t t (cid:15) θ (x t , t) + σ t z 6: until converged 6: return x 0 (cid:16) (cid:17) Equation (10) reveals that µ must predict 1 x βt (cid:15) given x .
<a id="S0036"></a> Source: p.4 S0036
Since x is available as input to the model, we may c θ hoose the param √ e α t t eriza t ti − on √1 − α¯t t t (cid:18) (cid:19) (cid:18) (cid:19) 1 1 β µ (x , t) = µ˜ x , (x √1 α¯ (cid:15) (x )) = x t (cid:15) (x , t) (11) θ t t t √α¯ t − − t θ t √α t − √1 α¯ θ t t t t − where (cid:15) is a function approximator intended to predict (cid:15) from x .
<a id="S0037"></a> Source: p.4 S0037
To sample x p (x x ) is θ t t 1 θ t 1 t (cid:16) (cid:17) − ∼ − | to compute x = 1 x βt (cid:15) (x , t) + σ z, where z (0, I).
<a id="S0038"></a> Source: p.4 S0038
The complete sampling procedure, A t l − g 1 orith √ m αt 2, re t se − m √ bl 1 e−s α¯ L t an θ gev t in dynam t ics with (cid:15) ∼ a N s a learned gradient of the data θ density.
<a id="S0039"></a> Source: p.4 S0039
Furthermore, with the parameterization (11), Eq. (10) simplifies to: E x0,(cid:15) (cid:20) 2σ2α β (1 t 2 α¯ ) (cid:13) (cid:13)(cid:15) − (cid:15) θ (√α¯ t x 0 + √1 − α¯ t (cid:15), t) (cid:13) (cid:13) 2 (cid:21) (12) t t − t which resembles denoising score matching over multiple noise scales indexed by t [55].
<a id="S0040"></a> Source: p.4 S0040
As Eq. (12) is equal to (one term of) the variational bound for the Langevin-like reverse process (11), we see that optimizing an objective resembling denoising score matching is equivalent to using variational inference to fit the finite-time marginal of a sampling chain resembling Langevin dynamics.
<a id="S0041"></a> Source: p.4 S0041
To summarize, we can train the reverse process mean function approximator µ to predict µ˜ , or by θ t modifying its parameterization, we can train it to predict (cid:15). (There is also the possibility of predicting x , but we found this to lead to worse sample quality early in our experiments.) We have shown that 0 the (cid:15)-prediction parameterization both resembles Langevin dynamics and simplifies the diffusion model’s variational bound to an objective that resembles denoising score matching.
<a id="S0042"></a> Source: p.4 S0042
Nonetheless, it is just another parameterization of p (x x ), so we verify its effectiveness in Section 4 in an θ t 1 t ablation where we compare predicting (cid:15) aga−in | st predicting µ˜ . t 3.3 Data scaling, reverse process decoder, and L 0 We assume that image data consists of integers in 0, 1, . . . , 255 scaled linearly to [ 1, 1].
<a id="S0043"></a> Source: p.4 S0043
This { } − ensures that the neural network reverse process operates on consistently scaled inputs starting from the standard normal prior p(x ).
<a id="S0044"></a> Source: p.4 S0044
To obtain discrete log likelihoods, we set the last term of the reverse T process to an independent discrete decoder derived from the Gaussian (x ; µ (x , 1), σ2I): N 0 θ 1 1 (cid:89) D (cid:90) δ+(xi 0 ) p (x x ) = (x; µi (x , 1), σ2) dx θ 0 | 1 N θ 1 1 i=1 δ − (xi 0 ) (13) (cid:26) (cid:26) if x = 1 if x = 1 δ (x) = ∞ δ (x) = −∞ − + x + 1 if x < 1 − x 1 if x > 1 255 − 255 − where D is the data dimensionality and the i superscript indicates extraction of one coordinate. (It would be straightforward to instead incorporate a more powerful decoder like a conditional autoregressive model, but we leave that to future work.) Similar to the discretized continuous distributions used in VAE decoders and autoregressive models [34, 52], our choice here ensures that the variational bound is a lossless codelength of discrete data, without need of adding noise to the data or incorporating the Jacobian of the scaling operation into the log likelihood.
<a id="S0045"></a> Source: p.4 S0045
At the end of sampling, we display µ (x , 1) noiselessly. θ 1 3.4 Simplified training objective With the reverse process and decoder defined above, the variational bound, consisting of terms derived from Eqs. (12) and (13), is clearly differentiable with respect to θ and is ready to be employed for 4
Page 5
<a id="S0046"></a> Source: p.5 S0046
Model IS FID NLL Test (Train) Table 2: Unconditional CIFAR10 reverse Conditional process parameterization and training objec- EBM [11] 8.30 37.9 tive ablation.
<a id="S0047"></a> Source: p.5 S0047
Blank entries were unstable to JEM [17] 8.76 38.4 train and generated poor samples with out-of- BigGAN [3] 9.22 14.73 range scores.
<a id="S0048"></a> Source: p.5 S0048
StyleGAN2 + ADA (v1) [29] 10.06 2.67 Objective IS FID Unconditional µ˜ prediction (baseline) Diffusion (original) [53] 5.40 Gated PixelCNN [59] 4.60 65.93 3.0 ≤ 3 (2.90) L, learned diagonal Σ 7.28 0.10 23.69 Sparse Transformer [7] 2.80 L, fixed isotropic Σ 8.06 ± 0.09 13.22 PixelIQN [43] 5.29 49.46 µ˜ µ˜ 2 ± – – EBM [11] 6.78 38.2 (cid:107) − θ(cid:107) NCSNv2 [56] 31.75 (cid:15) prediction (ours) NCSN [55] 8.87 0.12 25.32 SNGAN [39] 8.22 ± 0.05 21.7 L, learned diagonal Σ – – S S N ty G le A G N A - N D 2 D + L A S D [4 A ] (v1) [29] 9 9 . . 7 0 4 9 ± ± 0 0 . . 1 0 0 5 1 3 5 . . 2 4 6 2 (cid:107) L ˜(cid:15) , fi − xe (cid:15) d θ i (cid:107) so 2 tr ( o L p s ic im Σ ple) 9 7 . . 4 6 6 7 ± ± 0 0. . 1 1 3 1 1 3 3 . . 1 5 7 1 Ours (L, fixed isotropic Σ) 7.67 ± 0.13 13.51 3.70 (3.69) Ours (Lsimple) 9.46 ± ± 0.11 3.17 ≤ ≤ 3.75 (3.72) training.
<a id="S0049"></a> Source: p.5 S0049
However, we found it beneficial to sample quality (and simpler to implement) to train on the following variant of the variational bound: L simple (θ) := E t,x0,(cid:15) (cid:104)(cid:13) (cid:13)(cid:15) − (cid:15) θ (√α¯ t x 0 + √1 − α¯ t (cid:15), t) (cid:13) (cid:13) 2 (cid:105) (14) where t is uniform between 1 and T .
<a id="S0050"></a> Source: p.5 S0050
The t = 1 case corresponds to L with the integral in the 0 discrete decoder definition (13) approximated by the Gaussian probability density function times the bin width, ignoring σ2 and edge effects.
<a id="S0051"></a> Source: p.5 S0051
The t > 1 cases correspond to an unweighted version of 1 Eq. (12), analogous to the loss weighting used by the NCSN denoising score matching model [55]. (L does not appear because the forward process variances β are fixed.) Algorithm 1 displays the T t complete training procedure with this simplified objective.
<a id="S0052"></a> Source: p.5 S0052
Since our simplified objective (14) discards the weighting in Eq. (12), it is a weighted variational bound that emphasizes different aspects of reconstruction compared to the standard variational bound [18, 22].
<a id="S0053"></a> Source: p.5 S0053
In particular, our diffusion process setup in Section 4 causes the simplified objective to down-weight loss terms corresponding to small t.
<a id="S0054"></a> Source: p.5 S0054
These terms train the network to denoise data with very small amounts of noise, so it is beneficial to down-weight them so that the network can focus on more difficult denoising tasks at larger t terms.
<a id="S0055"></a> Source: p.5 S0055
We will see in our experiments that this reweighting leads to better sample quality. 4 Experiments We set T = 1000 for all experiments so that the number of neural network evaluations needed during sampling matches previous work [53, 55].
<a id="S0056"></a> Source: p.5 S0056
We set the forward process variances to constants increasing linearly from β = 10 4 to β = 0.02.
<a id="S0057"></a> Source: p.5 S0057
These constants were chosen to be small 1 − T relative to data scaled to [ 1, 1], ensuring that reverse and forward processes have approximately − the same functional form while keeping the signal-to-noise ratio at x as small as possible (L = T T D (q(x x ) (0, I)) 10 5 bits per dimension in our experiments).
<a id="S0058"></a> Source: p.5 S0058
KL T 0 − | (cid:107) N ≈ To represent the reverse process, we use a U-Net backbone similar to an unmasked PixelCNN++ [52, 48] with group normalization throughout [66].
<a id="S0059"></a> Source: p.5 S0059
Parameters are shared across time, which is specified to the network using the Transformer sinusoidal position embedding [60].
<a id="S0060"></a> Source: p.5 S0060
We use self-attention at the 16 16 feature map resolution [63, 60].
<a id="S0061"></a> Source: p.5 S0061
Details are in Appendix B. × 4.1 Sample quality Table 1 shows Inception scores, FID scores, and negative log likelihoods (lossless codelengths) on CIFAR10.
<a id="S0062"></a> Source: p.5 S0062
With our FID score of 3.17, our unconditional model achieves better sample quality than most models in the literature, including class conditional models.
<a id="S0063"></a> Source: p.5 S0063
Our FID score is computed with respect to the training set, as is standard practice; when we compute it with respect to the test set, the score is 5.24, which is still better than many of the training set FID scores in the literature. 5
Page 6
<a id="S0064"></a> Source: p.6 S0064
FID=7.89 Figure 4: LSUN Bedroom samples.
<a id="S0065"></a> Source: p.6 S0065
FID=4.90 Algorithm 3 Sending x Algorithm 4 Receiving 0 1: Send x T ∼ q(x T |x 0 ) using p(x T ) 1: Receive x T using p(x T ) 2: for t = T − 1, . . . , 2, 1 do 2: for t = T − 1, . . . , 1, 0 do 3: Send x t ∼ q(x t |x t+1 , x 0 ) using p θ (x t |x t+1 ) 3: Receive x t using p θ (x t |x t+1 ) 4: end for 4: end for 5: Send x 0 using p θ (x 0 |x 1 ) 5: return x 0 We find that training our models on the true variational bound yields better codelengths than training on the simplified objective, as expected, but the latter yields the best sample quality.
<a id="S0066"></a> Source: p.6 S0066
See Fig. 1 for CIFAR10 and CelebA-HQ 256 256 samples, Fig. 3 and Fig. 4 for LSUN 256 256 samples [71], × × and Appendix D for more. 4.2 Reverse process parameterization and training objective ablation In Table 2, we show the sample quality effects of reverse process parameterizations and training objectives (Section 3.2).
<a id="S0067"></a> Source: p.6 S0067
We find that the baseline option of predicting µ˜ works well only when trained on the true variational bound instead of unweighted mean squared error, a simplified objective akin to Eq. (14).
<a id="S0068"></a> Source: p.6 S0068
We also see that learning reverse process variances (by incorporating a parameterized diagonal Σ (x ) into the variational bound) leads to unstable training and poorer sample quality θ t compared to fixed variances.
<a id="S0069"></a> Source: p.6 S0069
Predicting (cid:15), as we proposed, performs approximately as well as predicting µ˜ when trained on the variational bound with fixed variances, but much better when trained with our simplified objective. 4.3 Progressive coding Table 1 also shows the codelengths of our CIFAR10 models.
<a id="S0070"></a> Source: p.6 S0070
The gap between train and test is at most 0.03 bits per dimension, which is comparable to the gaps reported with other likelihood-based models and indicates that our diffusion model is not overfitting (see Appendix D for nearest neighbor visualizations).
<a id="S0071"></a> Source: p.6 S0071
Still, while our lossless codelengths are better than the large estimates reported for energy based models and score matching using annealed importance sampling [11], they are not competitive with other types of likelihood-based generative models [7].
<a id="S0072"></a> Source: p.6 S0072
Since our samples are nonetheless of high quality, we conclude that diffusion models have an inductive bias that makes them excellent lossy compressors.
<a id="S0073"></a> Source: p.6 S0073
Treating the variational bound terms L + + L 1 T · · · as rate and L as distortion, our CIFAR10 model with the highest quality samples has a rate of 1.78 0 bits/dim and a distortion of 1.97 bits/dim, which amounts to a root mean squared error of 0.95 on a scale from 0 to 255.
<a id="S0074"></a> Source: p.6 S0074
More than half of the lossless codelength describes imperceptible distortions.
<a id="S0075"></a> Source: p.6 S0075
Progressive lossy compression We can probe further into the rate-distortion behavior of our model by introducing a progressive lossy code that mirrors the form of Eq. (5): see Algorithms 3 and 4, which assume access to a procedure, such as minimal random coding [19, 20], that can transmit a sample x q(x) using approximately D (q(x) p(x)) bits on average for any distributions p and KL ∼ (cid:107) q, for which only p is available to the receiver beforehand.
<a id="S0076"></a> Source: p.6 S0076
When applied to x q(x ), Algorithms 3 0 0 ∼ and 4 transmit x , . . . , x in sequence using a total expected codelength equal to Eq. (5).
Page 7
<a id="S0077"></a> Source: p.7 S0077
at any time t, has the partial information x fully available and can progressively estimate: t (cid:0) (cid:1) x xˆ = x √1 α¯ (cid:15) (x ) /√α¯ (15) 0 0 t t θ t t ≈ − − due to Eq. (4). (A stochastic reconstruction x p (x x ) is also valid, but we do not consider 0 θ 0 t ∼ | it here because it makes distortion more difficult to evaluate.) Figure 5 shows the resulting ratedistortion plot on the CIFAR10 test set.
<a id="S0078"></a> Source: p.7 S0078
At each time t, the distortion is calculated as the root mean (cid:112) squared error x xˆ 2/D, and the rate is calculated as the cumulative number of bits received 0 0 (cid:107) − (cid:107) so far at time t.
<a id="S0079"></a> Source: p.7 S0079
The distortion decreases steeply in the low-rate region of the rate-distortion plot, indicating that the majority of the bits are indeed allocated to imperceptible distortions. 80 60 40 20 0 0 200 400 600 800 1,000 Reverse process steps (T − t) )ESMR(noitrotsiD 1.5 1 0.5 0 0 200 400 600 800 1,000 Reverse process steps (T − t) )mid/stib(etaR 80 60 40 20 0 0 0.5 1 1.5 Rate (bits/dim) )ESMR(noitrotsiD Figure 5: Unconditional CIFAR10 test set rate-distortion vs. time.
<a id="S0080"></a> Source: p.7 S0080
Distortion is measured in root mean squared error on a [0, 255] scale.
<a id="S0081"></a> Source: p.7 S0081
Progressive generation We also run a progressive unconditional generation process given by progressive decompression from random bits.
<a id="S0082"></a> Source: p.7 S0082
In other words, we predict the result of the reverse process, xˆ , while sampling from the reverse process using Algorithm 2.
<a id="S0083"></a> Source: p.7 S0083
Figures 6 and 10 show the 0 resulting sample quality of xˆ over the course of the reverse process.
<a id="S0084"></a> Source: p.7 S0084
Large scale image features 0 appear first and details appear last.
<a id="S0085"></a> Source: p.7 S0085
Figure 7 shows stochastic predictions x p (x x ) with x 0 θ 0 t t ∼ | frozen for various t.
<a id="S0086"></a> Source: p.7 S0086
When t is small, all but fine details are preserved, and when t is large, only large scale features are preserved.
<a id="S0087"></a> Source: p.7 S0087
Perhaps these are hints of conceptual compression [18].
<a id="S0088"></a> Source: p.7 S0088
Figure 6: Unconditional CIFAR10 progressive generation (xˆ over time, from left to right).
<a id="S0089"></a> Source: p.7 S0089
Extended samples 0 and sample quality metrics over time in the appendix (Figs. 10 and 14).
<a id="S0090"></a> Source: p.7 S0090
Figure 7: When conditioned on the same latent, CelebA-HQ 256 × 256 samples share high-level attributes.
<a id="S0091"></a> Source: p.7 S0091
Bottom-right quadrants are x , and other quadrants are samples from p (x |x ). t θ 0 t Connection to autoregressive decoding Note that the variational bound (5) can be rewritten as: (cid:34) (cid:35) (cid:88) L = D (q(x ) p(x )) + E D (q(x x ) p (x x )) + H(x ) (16) KL T T q KL t 1 t θ t 1 t 0 (cid:107) − | (cid:107) − | t 1 ≥ (See Appendix A for a derivation.) Now consider setting the diffusion process length T to the dimensionality of the data, defining the forward process so that q(x x ) places all probability mass t 0 on x with the first t coordinates masked out (i.e. q(x x ) masks | out the tth coordinate), setting 0 t t 1 p(x ) to place all mass on a blank image, and, for th | e s−ake of argument, taking p (x x ) to T θ t 1 t − | 7
Page 8
<a id="S0092"></a> Source: p.8 S0092
Figure 8: Interpolations of CelebA-HQ 256x256 images with 500 timesteps of diffusion. be a fully expressive conditional distribution.
<a id="S0093"></a> Source: p.8 S0093
With these choices, D (q(x ) p(x )) = 0, and KL T T (cid:107) minimizing D (q(x x ) p (x x )) trains p to copy coordinates t + 1, . . . , T unchanged KL t 1 t θ t 1 t θ and to predict the tth co−ord | inat (cid:107) e given −t + | 1, . . . , T .
<a id="S0094"></a> Source: p.8 S0094
Thus, training p with this particular diffusion is θ training an autoregressive model.
<a id="S0095"></a> Source: p.8 S0095
We can therefore interpret the Gaussian diffusion model (2) as a kind of autoregressive model with a generalized bit ordering that cannot be expressed by reordering data coordinates.
<a id="S0096"></a> Source: p.8 S0096
Prior work has shown that such reorderings introduce inductive biases that have an impact on sample quality [38], so we speculate that the Gaussian diffusion serves a similar purpose, perhaps to greater effect since Gaussian noise might be more natural to add to images compared to masking noise.
<a id="S0097"></a> Source: p.8 S0097
Moreover, the Gaussian diffusion length is not restricted to equal the data dimension; for instance, we use T = 1000, which is less than the dimension of the 32 32 3 or 256 256 3 images in our experiments. × × × × Gaussian diffusions can be made shorter for fast sampling or longer for model expressiveness. 4.4 Interpolation We can interpolate source images x , x q(x ) in latent space using q as a stochastic encoder, 0 (cid:48)0 ∼ 0 x , x q(x x ), then decoding the linearly interpolated latent x¯ = (1 λ)x + λx into image t (cid:48)t ∼ t | 0 t − 0 (cid:48)0 space by the reverse process, x¯ p(x x¯ ).
<a id="S0098"></a> Source: p.8 S0098
In effect, we use the reverse process to remove 0 0 t ∼ | artifacts from linearly interpolating corrupted versions of the source images, as depicted in Fig. 8 (left).
<a id="S0099"></a> Source: p.8 S0099
We fixed the noise for different values of λ so x and x remain the same.
<a id="S0100"></a> Source: p.8 S0100
Fig. 8 (right) t (cid:48)t shows interpolations and reconstructions of original CelebA-HQ 256 256 images (t = 500).
<a id="S0101"></a> Source: p.8 S0101
The × reverse process produces high-quality reconstructions, and plausible interpolations that smoothly vary attributes such as pose, skin tone, hairstyle, expression and background, but not eyewear.
<a id="S0102"></a> Source: p.8 S0102
Larger t results in coarser and more varied interpolations, with novel samples at t = 1000 (Appendix Fig. 9). 5 Related Work While diffusion models might resemble flows [9, 46, 10, 32, 5, 16, 23] and VAEs [33, 47, 37], diffusion models are designed so that q has no parameters and the top-level latent x has nearly zero T mutual information with the data x .
<a id="S0103"></a> Source: p.8 S0103
Our (cid:15)-prediction reverse process parameterization establishes a 0 connection between diffusion models and denoising score matching over multiple noise levels with annealed Langevin dynamics for sampling [55, 56].
<a id="S0104"></a> Source: p.8 S0104
Diffusion models, however, admit straightforward log likelihood evaluation, and the training procedure explicitly trains the Langevin dynamics sampler using variational inference (see Appendix C for details).
<a id="S0105"></a> Source: p.8 S0105
The connection also has the reverse implication that a certain weighted form of denoising score matching is the same as variational inference to train a Langevin-like sampler.
<a id="S0106"></a> Source: p.8 S0106
Other methods for learning transition operators of Markov chains include infusion training [2], variational walkback [15], generative stochastic networks [1], and others [50, 54, 36, 42, 35, 65].
<a id="S0107"></a> Source: p.8 S0107
By the known connection between score matching and energy-based modeling, our work could have implications for other recent work on energy-based models [67–69, 12, 70, 13, 11, 41, 17, 8].
<a id="S0108"></a> Source: p.8 S0108
Our rate-distortion curves are computed over time in one evaluation of the variational bound, reminiscent of how rate-distortion curves can be computed over distortion penalties in one run of annealed importance sampling [24].
<a id="S0109"></a> Source: p.8 S0109
Our progressive decoding argument can be seen in convolutional DRAW and related models [18, 40] and may also lead to more general designs for subscale orderings or sampling strategies for autoregressive models [38, 64]. 8
Page 9
<a id="S0110"></a> Source: p.9 S0110
6 Conclusion We have presented high quality image samples using diffusion models, and we have found connections among diffusion models and variational inference for training Markov chains, denoising score matching and annealed Langevin dynamics (and energy-based models by extension), autoregressive models, and progressive lossy compression.
<a id="S0111"></a> Source: p.9 S0111
Since diffusion models seem to have excellent inductive biases for image data, we look forward to investigating their utility in other data modalities and as components in other types of generative models and machine learning systems.
<a id="S0112"></a> Source: p.9 S0112
Broader Impact Our work on diffusion models takes on a similar scope as existing work on other types of deep generative models, such as efforts to improve the sample quality of GANs, flows, autoregressive models, and so forth.
<a id="S0113"></a> Source: p.9 S0113
Our paper represents progress in making diffusion models a generally useful tool in this family of techniques, so it may serve to amplify any impacts that generative models have had (and will have) on the broader world.
<a id="S0114"></a> Source: p.9 S0114
Unfortunately, there are numerous well-known malicious uses of generative models.
<a id="S0115"></a> Source: p.9 S0115
Sample generation techniques can be employed to produce fake images and videos of high profile figures for political purposes.
<a id="S0116"></a> Source: p.9 S0116
While fake images were manually created long before software tools were available, generative models such as ours make the process easier.
<a id="S0117"></a> Source: p.9 S0117
Fortunately, CNN-generated images currently have subtle flaws that allow detection [62], but improvements in generative models may make this more difficult.
<a id="S0118"></a> Source: p.9 S0118
Generative models also reflect the biases in the datasets on which they are trained.
<a id="S0119"></a> Source: p.9 S0119
As many large datasets are collected from the internet by automated systems, it can be difficult to remove these biases, especially when the images are unlabeled.
<a id="S0120"></a> Source: p.9 S0120
If samples from generative models trained on these datasets proliferate throughout the internet, then these biases will only be reinforced further.
<a id="S0121"></a> Source: p.9 S0121
On the other hand, diffusion models may be useful for data compression, which, as data becomes higher resolution and as global internet traffic increases, might be crucial to ensure accessibility of the internet to wide audiences.
<a id="S0122"></a> Source: p.9 S0122
Our work might contribute to representation learning on unlabeled raw data for a large range of downstream tasks, from image classification to reinforcement learning, and diffusion models might also become viable for creative uses in art, photography, and music.
<a id="S0123"></a> Source: p.9 S0123
Acknowledgments and Disclosure of Funding This work was supported by ONR PECASE and the NSF Graduate Research Fellowship under grant number DGE-1752814.
<a id="S0124"></a> Source: p.9 S0124
Google’s TensorFlow Research Cloud (TFRC) provided Cloud TPUs.
<a id="S0125"></a> Source: p.9 S0125
References [1] Guillaume Alain, Yoshua Bengio, Li Yao, Jason Yosinski, Eric Thibodeau-Laufer, Saizheng Zhang, and Pascal Vincent.
<a id="S0126"></a> Source: p.9 S0126
Information and Inference: A Journal of the IMA, 5(2):210–249, 2016. [2] Florian Bordes, Sina Honari, and Pascal Vincent.
<a id="S0127"></a> Source: p.9 S0127
Learning to generate samples from noise through infusion training.
<a id="S0128"></a> Source: p.9 S0128
In International Conference on Learning Representations, 2017. [3] Andrew Brock, Jeff Donahue, and Karen Simonyan.
<a id="S0129"></a> Source: p.9 S0129
Large scale GAN training for high fidelity natural image synthesis.
<a id="S0130"></a> Source: p.9 S0130
In International Conference on Learning Representations, 2019. [4] Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio.
<a id="S0131"></a> Source: p.9 S0131
Your GAN is secretly an energy-based model and you should use discriminator driven latent sampling. arXiv preprint arXiv:2003.06060, 2020. [5] Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud.
<a id="S0132"></a> Source: p.9 S0132
In Advances in Neural Information Processing Systems, pages 6571–6583, 2018. [6] Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel.
<a id="S0133"></a> Source: p.9 S0133
PixelSNAIL: An improved autoregressive generative model.
<a id="S0134"></a> Source: p.9 S0134
In International Conference on Machine Learning, pages 863–871, 2018. [7] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever.
<a id="S0135"></a> Source: p.9 S0135
Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019. 9
Page 10
<a id="S0136"></a> Source: p.10 S0136
[8] Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc’Aurelio Ranzato.
<a id="S0137"></a> Source: p.10 S0137
Residual energy-based models for text generation. arXiv preprint arXiv:2004.11714, 2020. [9] Laurent Dinh, David Krueger, and Yoshua Bengio.
<a id="S0138"></a> Source: p.10 S0138
NICE: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516, 2014. [10] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio.
<a id="S0139"></a> Source: p.10 S0139
Density estimation using Real NVP. arXiv preprint arXiv:1605.08803, 2016. [11] Yilun Du and Igor Mordatch.
<a id="S0140"></a> Source: p.10 S0140
Implicit generation and modeling with energy based models.
<a id="S0141"></a> Source: p.10 S0141
In Advances in Neural Information Processing Systems, pages 3603–3613, 2019. [12] Ruiqi Gao, Yang Lu, Junpei Zhou, Song-Chun Zhu, and Ying Nian Wu.
<a id="S0142"></a> Source: p.10 S0142
Learning generative ConvNets via multi-grid modeling and sampling.
<a id="S0143"></a> Source: p.10 S0143
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9155–9164, 2018. [13] Ruiqi Gao, Erik Nijkamp, Diederik P Kingma, Zhen Xu, Andrew M Dai, and Ying Nian Wu.
<a id="S0144"></a> Source: p.10 S0144
Flow contrastive estimation of energy-based models.
<a id="S0145"></a> Source: p.10 S0145
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7518–7528, 2020. [14] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.
<a id="S0146"></a> Source: p.10 S0146
In Advances in Neural Information Processing Systems, pages 2672–2680, 2014. [15] Anirudh Goyal, Nan Rosemary Ke, Surya Ganguli, and Yoshua Bengio.
<a id="S0147"></a> Source: p.10 S0147
Variational walkback: Learning a transition operator as a stochastic recurrent net.
<a id="S0148"></a> Source: p.10 S0148
In Advances in Neural Information Processing Systems, pages 4392–4402, 2017. [16] Will Grathwohl, Ricky T. Q.
<a id="S0149"></a> Source: p.10 S0149
Chen, Jesse Bettencourt, and David Duvenaud.
<a id="S0150"></a> Source: p.10 S0150
FFJORD: Free-form continuous dynamics for scalable reversible generative models.
<a id="S0151"></a> Source: p.10 S0151
In International Conference on Learning Representations, 2019. [17] Will Grathwohl, Kuan-Chieh Wang, Joern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky.
<a id="S0152"></a> Source: p.10 S0152
Your classifier is secretly an energy based model and you should treat it like one.
<a id="S0153"></a> Source: p.10 S0153
In International Conference on Learning Representations, 2020. [18] Karol Gregor, Frederic Besse, Danilo Jimenez Rezende, Ivo Danihelka, and Daan Wierstra.
<a id="S0154"></a> Source: p.10 S0154
In Advances In Neural Information Processing Systems, pages 3549–3557, 2016. [19] Prahladh Harsha, Rahul Jain, David McAllester, and Jaikumar Radhakrishnan.
<a id="S0155"></a> Source: p.10 S0155
The communication complexity of correlation.
<a id="S0156"></a> Source: p.10 S0156
In Twenty-Second Annual IEEE Conference on Computational Complexity (CCC’07), pages 10–23.
<a id="S0157"></a> Source: p.10 S0157
IEEE, 2007. [20] Marton Havasi, Robert Peharz, and José Miguel Hernández-Lobato.
<a id="S0158"></a> Source: p.10 S0158
Minimal random code learning: Getting bits back from compressed model parameters.
<a id="S0159"></a> Source: p.10 S0159
In International Conference on Learning Representations, 2019. [21] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
<a id="S0160"></a> Source: p.10 S0160
GANs trained by a two time-scale update rule converge to a local Nash equilibrium.
<a id="S0161"></a> Source: p.10 S0161
In Advances in Neural Information Processing Systems, pages 6626–6637, 2017. [22] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with a constrained variational framework.
<a id="S0162"></a> Source: p.10 S0162
In International Conference on Learning Representations, 2017. [23] Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel.
<a id="S0163"></a> Source: p.10 S0163
Flow++: Improving flow-based generative models with variational dequantization and architecture design.
<a id="S0164"></a> Source: p.10 S0164
In International Conference on Machine Learning, 2019. [24] Sicong Huang, Alireza Makhzani, Yanshuai Cao, and Roger Grosse.
<a id="S0165"></a> Source: p.10 S0165
Evaluating lossy compression rates of deep generative models.
<a id="S0166"></a> Source: p.10 S0166
In International Conference on Machine Learning, 2020. [25] Nal Kalchbrenner, Aaron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu.
<a id="S0167"></a> Source: p.10 S0167
In International Conference on Machine Learning, pages 1771–1779, 2017. [26] Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron van den Oord, Sander Dieleman, and Koray Kavukcuoglu.
<a id="S0168"></a> Source: p.10 S0168
In International Conference on Machine Learning, pages 2410–2419, 2018. [27] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.
<a id="S0169"></a> Source: p.10 S0169
Progressive growing of GANs for improved quality, stability, and variation.
<a id="S0170"></a> Source: p.10 S0170
In International Conference on Learning Representations, 2018. [28] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks.
<a id="S0171"></a> Source: p.10 S0171
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10
Page 11
<a id="S0172"></a> Source: p.11 S0172
4401–4410, 2019. [29] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila.
<a id="S0173"></a> Source: p.11 S0173
Training generative adversarial networks with limited data. arXiv preprint arXiv:2006.06676v1, 2020. [30] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.
<a id="S0174"></a> Source: p.11 S0174
Analyzing and improving the image quality of StyleGAN.
<a id="S0175"></a> Source: p.11 S0175
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110–8119, 2020. [31] Diederik P Kingma and Jimmy Ba.
<a id="S0176"></a> Source: p.11 S0176
Adam: A method for stochastic optimization.
<a id="S0177"></a> Source: p.11 S0177
In International Conference on Learning Representations, 2015. [32] Diederik P Kingma and Prafulla Dhariwal.
<a id="S0178"></a> Source: p.11 S0178
Glow: Generative flow with invertible 1x1 convolutions.
<a id="S0179"></a> Source: p.11 S0179
In Advances in Neural Information Processing Systems, pages 10215–10224, 2018. [33] Diederik P Kingma and Max Welling.
<a id="S0180"></a> Source: p.11 S0180
Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114, 2013. [34] Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling.
<a id="S0181"></a> Source: p.11 S0181
Improved variational inference with inverse autoregressive flow.
<a id="S0182"></a> Source: p.11 S0182
In Advances in Neural Information Processing Systems, pages 4743–4751, 2016. [35] John Lawson, George Tucker, Bo Dai, and Rajesh Ranganath.
<a id="S0183"></a> Source: p.11 S0183
Energy-inspired models: Learning with sampler-induced distributions.
<a id="S0184"></a> Source: p.11 S0184
In Advances in Neural Information Processing Systems, pages 8501–8513, 2019. [36] Daniel Levy, Matt D.
<a id="S0185"></a> Source: p.11 S0185
Generalizing Hamiltonian Monte Carlo with neural networks.
<a id="S0186"></a> Source: p.11 S0186
In International Conference on Learning Representations, 2018. [37] Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther.
<a id="S0187"></a> Source: p.11 S0187
BIVA: A very deep hierarchy of latent variables for generative modeling.
<a id="S0188"></a> Source: p.11 S0188
In Advances in Neural Information Processing Systems, pages 6548–6558, 2019. [38] Jacob Menick and Nal Kalchbrenner.
<a id="S0189"></a> Source: p.11 S0189
Generating high fidelity images with subscale pixel networks and multidimensional upscaling.
<a id="S0190"></a> Source: p.11 S0190
In International Conference on Learning Representations, 2019. [39] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.
<a id="S0191"></a> Source: p.11 S0191
Spectral normalization for generative adversarial networks.
<a id="S0192"></a> Source: p.11 S0192
In International Conference on Learning Representations, 2018. [40] Alex Nichol.
<a id="S0193"></a> Source: p.11 S0193
VQ-DRAW: A sequential discrete VAE. arXiv preprint arXiv:2003.01599, 2020. [41] Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu.
<a id="S0194"></a> Source: p.11 S0194
On the anatomy of MCMC-based maximum likelihood learning of energy-based models. arXiv preprint arXiv:1903.12370, 2019. [42] Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu.
<a id="S0195"></a> Source: p.11 S0195
Learning non-convergent non-persistent short-run MCMC toward energy-based model.
<a id="S0196"></a> Source: p.11 S0196
In Advances in Neural Information Processing Systems, pages 5233–5243, 2019. [43] Georg Ostrovski, Will Dabney, and Remi Munos.
<a id="S0197"></a> Source: p.11 S0197
Autoregressive quantile networks for generative modeling.
<a id="S0198"></a> Source: p.11 S0198
In International Conference on Machine Learning, pages 3936–3945, 2018. [44] Ryan Prenger, Rafael Valle, and Bryan Catanzaro.
<a id="S0199"></a> Source: p.11 S0199
WaveGlow: A flow-based generative network for speech synthesis.
<a id="S0200"></a> Source: p.11 S0200
In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3617–3621.
<a id="S0201"></a> Source: p.11 S0201
IEEE, 2019. [45] Ali Razavi, Aaron van den Oord, and Oriol Vinyals.
<a id="S0202"></a> Source: p.11 S0202
Generating diverse high-fidelity images with VQ- VAE-2.
<a id="S0203"></a> Source: p.11 S0203
In Advances in Neural Information Processing Systems, pages 14837–14847, 2019. [46] Danilo Rezende and Shakir Mohamed.
<a id="S0204"></a> Source: p.11 S0204
Variational inference with normalizing flows.
<a id="S0205"></a> Source: p.11 S0205
In International Conference on Machine Learning, pages 1530–1538, 2015. [47] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra.
<a id="S0206"></a> Source: p.11 S0206
Stochastic backpropagation and approximate inference in deep generative models.
<a id="S0207"></a> Source: p.11 S0207
In International Conference on Machine Learning, pages 1278–1286, 2014. [48] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation.
<a id="S0208"></a> Source: p.11 S0208
In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 234–241.
<a id="S0209"></a> Source: p.11 S0209
Springer, 2015. [49] Tim Salimans and Durk P Kingma.
<a id="S0210"></a> Source: p.11 S0210
Weight normalization: A simple reparameterization to accelerate training of deep neural networks.
<a id="S0211"></a> Source: p.11 S0211
In Advances in Neural Information Processing Systems, pages 901–909, 2016. [50] Tim Salimans, Diederik Kingma, and Max Welling.
<a id="S0212"></a> Source: p.11 S0212
Markov Chain Monte Carlo and variational inference: Bridging the gap.
<a id="S0213"></a> Source: p.11 S0213
In International Conference on Machine Learning, pages 1218–1226, 2015. 11
Page 12
<a id="S0214"></a> Source: p.12 S0214
[51] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.
<a id="S0215"></a> Source: p.12 S0215
In Advances in Neural Information Processing Systems, pages 2234–2242, 2016. [52] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma.
<a id="S0216"></a> Source: p.12 S0216
PixelCNN++: Improving the PixelCNN with discretized logistic mixture likelihood and other modifications.
<a id="S0217"></a> Source: p.12 S0217
In International Conference on Learning Representations, 2017. [53] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.
<a id="S0218"></a> Source: p.12 S0218
Deep unsupervised learning using nonequilibrium thermodynamics.
<a id="S0219"></a> Source: p.12 S0219
In International Conference on Machine Learning, pages 2256–2265, 2015. [54] Jiaming Song, Shengjia Zhao, and Stefano Ermon. A-NICE-MC: Adversarial training for MCMC.
<a id="S0220"></a> Source: p.12 S0220
In Advances in Neural Information Processing Systems, pages 5140–5150, 2017. [55] Yang Song and Stefano Ermon.
<a id="S0221"></a> Source: p.12 S0221
Generative modeling by estimating gradients of the data distribution.
<a id="S0222"></a> Source: p.12 S0222
In Advances in Neural Information Processing Systems, pages 11895–11907, 2019. [56] Yang Song and Stefano Ermon.
<a id="S0223"></a> Source: p.12 S0223
Improved techniques for training score-based generative models. arXiv preprint arXiv:2006.09011, 2020. [57] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu.
<a id="S0224"></a> Source: p.12 S0224
WaveNet: A generative model for raw audio. arXiv preprint arXiv:1609.03499, 2016. [58] Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu.
<a id="S0225"></a> Source: p.12 S0225
International Conference on Machine Learning, 2016. [59] Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu.
<a id="S0226"></a> Source: p.12 S0226
Conditional image generation with PixelCNN decoders.
<a id="S0227"></a> Source: p.12 S0227
In Advances in Neural Information Processing Systems, pages 4790–4798, 2016. [60] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
<a id="S0228"></a> Source: p.12 S0228
In Advances in Neural Information Processing Systems, pages 5998–6008, 2017. [61] Pascal Vincent. A connection between score matching and denoising autoencoders.
<a id="S0229"></a> Source: p.12 S0229
Neural Computation, 23(7):1661–1674, 2011. [62] Sheng-Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros.
<a id="S0230"></a> Source: p.12 S0230
Cnn-generated images are surprisingly easy to spot...for now.
<a id="S0231"></a> Source: p.12 S0231
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020. [63] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.
<a id="S0232"></a> Source: p.12 S0232
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7794–7803, 2018. [64] Auke J Wiggers and Emiel Hoogeboom.
<a id="S0233"></a> Source: p.12 S0233
Predictive sampling with forecasting autoregressive models. arXiv preprint arXiv:2002.09928, 2020. [65] Hao Wu, Jonas Köhler, and Frank Noé.
<a id="S0234"></a> Source: p.12 S0234
Stochastic normalizing flows. arXiv preprint arXiv:2002.06707, 2020. [66] Yuxin Wu and Kaiming He.
<a id="S0235"></a> Source: p.12 S0235
In Proceedings of the European Conference on Computer Vision (ECCV), pages 3–19, 2018. [67] Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet.
<a id="S0236"></a> Source: p.12 S0236
In International Conference on Machine Learning, pages 2635–2644, 2016. [68] Jianwen Xie, Song-Chun Zhu, and Ying Nian Wu.
<a id="S0237"></a> Source: p.12 S0237
Synthesizing dynamic patterns by spatial-temporal generative convnet.
<a id="S0238"></a> Source: p.12 S0238
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7093–7101, 2017. [69] Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, and Ying Nian Wu.
<a id="S0239"></a> Source: p.12 S0239
Learning descriptor networks for 3d shape synthesis and analysis.
<a id="S0240"></a> Source: p.12 S0240
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8629–8638, 2018. [70] Jianwen Xie, Song-Chun Zhu, and Ying Nian Wu.
<a id="S0241"></a> Source: p.12 S0241
Learning energy-based spatial-temporal generative convnets for dynamic patterns.
<a id="S0242"></a> Source: p.12 S0242
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019. [71] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao.
<a id="S0243"></a> Source: p.12 S0243
LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015. [72] Sergey Zagoruyko and Nikos Komodakis.
<a id="S0244"></a> Source: p.12 S0244
Wide residual networks. arXiv preprint arXiv:1605.07146, 2016. 12
Page 13
<a id="S0245"></a> Source: p.13 S0245
Extra information LSUN FID scores for LSUN datasets are included in Table 3.
<a id="S0246"></a> Source: p.13 S0246
Scores marked with are reported ∗ by StyleGAN2 as baselines, and other scores are reported by their respective authors.
<a id="S0247"></a> Source: p.13 S0247
Table 3: FID scores for LSUN 256 256 datasets × Model LSUN Bedroom LSUN Church LSUN Cat ProgressiveGAN [27] 8.34 6.42 37.52 StyleGAN [28] 2.65 4.21 8.53 ∗ ∗ StyleGAN2 [30] - 3.86 6.93 Ours (L ) 6.36 7.89 19.75 simple Ours (L , large) 4.90 - simple Progressive compression Our lossy compression argument in Section 4.3 is only a proof of concept, because Algorithms 3 and 4 depend on a procedure such as minimal random coding [20], which is not tractable for high dimensional data.
<a id="S0248"></a> Source: p.13 S0248
These algorithms serve as a compression interpretation of the variational bound (5) of Sohl-Dickstein et al. [53], not yet as a practical compression system.
<a id="S0249"></a> Source: p.13 S0249
Table 4: Unconditional CIFAR10 test set rate-distortion values (accompanies Fig. 5) Reverse process time (T t + 1) Rate (bits/dim) Distortion (RMSE [0, 255]) − 1000 1.77581 0.95136 900 0.11994 12.02277 800 0.05415 18.47482 700 0.02866 24.43656 600 0.01507 30.80948 500 0.00716 38.03236 400 0.00282 46.12765 300 0.00081 54.18826 200 0.00013 60.97170 100 0.00000 67.60125 A Extended derivations Below is a derivation of Eq. (5), the reduced variance variational bound for diffusion models.
<a id="S0250"></a> Source: p.13 S0250
This material is from Sohl-Dickstein et al. [53]; we include it here only for completeness. (cid:20) (cid:21) p (x ) L = E log θ 0:T (17) q − q(x x ) 1:T 0 | = E q log p(x T ) (cid:88) log p θ (x t − 1 | x t ) (18) − − q(x x ) t t 1 t 1 | − ≥ (cid:34) (cid:35) = E q log p(x T ) (cid:88) log p θ (x t − 1 | x t ) log p θ (x 0 | x 1 ) (19) − − q(x x ) − q(x x ) t t 1 1 0 t>1 | − | (cid:34) (cid:35) = E q log p(x T ) (cid:88) log p θ (x t − 1 | x t ) q(x t − 1 | x 0 ) log p θ (x 0 | x 1 ) (20) − − q(x x , x ) · q(x x ) − q(x x ) t 1 t 0 t 0 1 0 t>1 − | | | (cid:34) (cid:35) = E q log p(x T ) (cid:88) log p θ (x t − 1 | x t ) log p θ (x 0 x 1 ) (21) − q(x x ) − q(x x , x ) − | T 0 t 1 t 0 | t>1 − | 13
Page 14
<a id="S0251"></a> Source: p.14 S0251
(cid:34) (cid:35) (cid:88) = E D (q(x x ) p(x )) + D (q(x x , x ) p (x x )) log p (x x ) q KL T 0 T KL t 1 t 0 θ t 1 t θ 0 1 | (cid:107) − | (cid:107) − | − | t>1 (22) The following is an alternate version of L.
<a id="S0252"></a> Source: p.14 S0252
It is not tractable to estimate, but it is useful for our discussion in Section 4.3. L = E q log p(x T ) (cid:88) log p θ (x t − 1 | x t ) (23) − − q(x x ) t t 1 t 1 | − ≥ = E q log p(x T ) (cid:88) log p θ (x t − 1 | x t ) q(x t − 1 ) (24) − − q(x x ) · q(x ) t 1 t t t 1 − | ≥ = E q log p(x T ) (cid:88) log p θ (x t − 1 | x t ) log q(x 0 ) (25) − q(x ) − q(x x ) − T t 1 t t 1 − | ≥ (cid:88) = D KL (q(x T ) p(x T )) + E q D KL (q(x t 1 x t ) p θ (x t 1 x t )) + H(x 0 ) (26) (cid:107) − | (cid:107) − | t 1 ≥ B Experimental details Our neural network architecture follows the backbone of PixelCNN++ [52], which is a U-Net [48] based on a Wide ResNet [72].
<a id="S0253"></a> Source: p.14 S0253
We replaced weight normalization [49] with group normalization [66] to make the implementation simpler.
<a id="S0254"></a> Source: p.14 S0254
Our 32 32 models use four feature map resolutions (32 32 × × to 4 4), and our 256 256 models use six.
<a id="S0255"></a> Source: p.14 S0255
All models have two convolutional residual blocks × × per resolution level and self-attention blocks at the 16 16 resolution between the convolutional × blocks [6].
<a id="S0256"></a> Source: p.14 S0256
Diffusion time t is specified by adding the Transformer sinusoidal position embedding [60] into each residual block.
<a id="S0257"></a> Source: p.14 S0257
Our CIFAR10 model has 35.7 million parameters, and our LSUN and CelebA-HQ models have 114 million parameters.
<a id="S0258"></a> Source: p.14 S0258
We also trained a larger variant of the LSUN Bedroom model with approximately 256 million parameters by increasing filter count.
<a id="S0259"></a> Source: p.14 S0259
We used TPU v3-8 (similar to 8 V100 GPUs) for all experiments.
<a id="S0260"></a> Source: p.14 S0260
Our CIFAR model trains at 21 steps per second at batch size 128 (10.6 hours to train to completion at 800k steps), and sampling a batch of 256 images takes 17 seconds.
<a id="S0261"></a> Source: p.14 S0261
Our CelebA-HQ/LSUN (2562) models train at 2.2 steps per second at batch size 64, and sampling a batch of 128 images takes 300 seconds.
<a id="S0262"></a> Source: p.14 S0262
We trained on CelebA-HQ for 0.5M steps, LSUN Bedroom for 2.4M steps, LSUN Cat for 1.8M steps, and LSUN Church for 1.2M steps.
<a id="S0263"></a> Source: p.14 S0263
The larger LSUN Bedroom model was trained for 1.15M steps.
<a id="S0264"></a> Source: p.14 S0264
Apart from an initial choice of hyperparameters early on to make network size fit within memory constraints, we performed the majority of our hyperparameter search to optimize for CIFAR10 sample quality, then transferred the resulting settings over to the other datasets: • We chose the β schedule from a set of constant, linear, and quadratic schedules, all t constrained so that L 0.
<a id="S0265"></a> Source: p.14 S0265
We set T = 1000 without a sweep, and we chose a linear T schedule from β = 10 ≈4 to β = 0.02. 1 − T • We set the dropout rate on CIFAR10 to 0.1 by sweeping over the values 0.1, 0.2, 0.3, 0.4 . { } Without dropout on CIFAR10, we obtained poorer samples reminiscent of the overfitting artifacts in an unregularized PixelCNN++ [52].
<a id="S0266"></a> Source: p.14 S0266
We set dropout rate on the other datasets to zero without sweeping. • We used random horizontal flips during training for CIFAR10; we tried training both with and without flips, and found flips to improve sample quality slightly.
<a id="S0267"></a> Source: p.14 S0267
We also used random horizontal flips for all other datasets except LSUN Bedroom. • We tried Adam [31] and RMSProp early on in our experimentation process and chose the former.
<a id="S0268"></a> Source: p.14 S0268
We left the hyperparameters to their standard values.
<a id="S0269"></a> Source: p.14 S0269
We set the learning rate to 2 10 4 without any sweeping, and we lowered it to 2 10 5 for the 256 256 images, − − × × × which seemed unstable to train with the larger learning rate. 14
Page 15
<a id="S0270"></a> Source: p.15 S0270
• We set the batch size to 128 for CIFAR10 and 64 for larger images.
<a id="S0271"></a> Source: p.15 S0271
We did not sweep over these values. • We used EMA on model parameters with a decay factor of 0.9999.
<a id="S0272"></a> Source: p.15 S0272
Final experiments were trained once and evaluated throughout training for sample quality.
<a id="S0273"></a> Source: p.15 S0273
Sample quality scores and log likelihood are reported on the minimum FID value over the course of training.
<a id="S0274"></a> Source: p.15 S0274
On CIFAR10, we calculated Inception and FID scores on 50000 samples using the original code from the OpenAI [51] and TTUR [21] repositories, respectively.
<a id="S0275"></a> Source: p.15 S0275
On LSUN, we calculated FID scores on 50000 samples using code from the StyleGAN2 [30] repository.
<a id="S0276"></a> Source: p.15 S0276
CIFAR10 and CelebA-HQ were loaded as provided by TensorFlow Datasets (https://www.tensorflow.org/datasets), and LSUN was prepared using code from StyleGAN.
<a id="S0277"></a> Source: p.15 S0277
Dataset splits (or lack thereof) are standard from the papers that introduced their usage in a generative modeling context.
<a id="S0278"></a> Source: p.15 S0278
All details can be found in the source code release. C Discussion on related work Our model architecture, forward process definition, and prior differ from NCSN [55, 56] in subtle but important ways that improve sample quality, and, notably, we directly train our sampler as a latent variable model rather than adding it after training post-hoc.
<a id="S0279"></a> Source: p.15 S0279
We use a U-Net with self-attention; NCSN uses a RefineNet with dilated convolutions.
<a id="S0280"></a> Source: p.15 S0280
We condition all layers on t by adding in the Transformer sinusoidal position embedding, rather than only in normalization layers (NCSNv1) or only at the output (v2). 2.
<a id="S0281"></a> Source: p.15 S0281
Diffusion models scale down the data with each forward process step (by a √1 β factor) t − so that variance does not grow when adding noise, thus providing consistently scaled inputs to the neural net reverse process.
<a id="S0282"></a> Source: p.15 S0282
Unlike NCSN, our forward process destroys signal (D (q(x x ) (0, I)) 0), ensur- KL T 0 | (cid:107) N ≈ ing a close match between the prior and aggregate posterior of x .
<a id="S0283"></a> Source: p.15 S0283
Also unlike NCSN, our T β are very small, which ensures that the forward process is reversible by a Markov chain t with conditional Gaussians.
<a id="S0284"></a> Source: p.15 S0284
Both of these factors prevent distribution shift when sampling. 4.
<a id="S0285"></a> Source: p.15 S0285
Our Langevin-like sampler has coefficients (learning rate, noise scale, etc.) derived rigorously from β in the forward process.
<a id="S0286"></a> Source: p.15 S0286
Thus, our training procedure directly trains our t sampler to match the data distribution after T steps: it trains the sampler as a latent variable model using variational inference.
<a id="S0287"></a> Source: p.15 S0287
In contrast, NCSN’s sampler coefficients are set by hand post-hoc, and their training procedure is not guaranteed to directly optimize a quality metric of their sampler. D Samples Additional samples Figure 11, 13, 16, 17, 18, and 19 show uncurated samples from the diffusion models trained on CelebA-HQ, CIFAR10 and LSUN datasets.
<a id="S0288"></a> Source: p.15 S0288
Latent structure and reverse process stochasticity During sampling, both the prior x T ∼ (0, I) and Langevin dynamics are stochastic.
<a id="S0289"></a> Source: p.15 S0289
To understand the significance of the second source N of noise, we sampled multiple images conditioned on the same intermediate latent for the CelebA 256 256 dataset.
<a id="S0290"></a> Source: p.15 S0290
Figure 7 shows multiple draws from the reverse process x p (x x ) that 0 θ 0 t × ∼ | share the latent x for t 1000, 750, 500, 250 .
<a id="S0291"></a> Source: p.15 S0291
To accomplish this, we run a single reverse chain t ∈ { } from an initial draw from the prior.
<a id="S0292"></a> Source: p.15 S0292
At the intermediate timesteps, the chain is split to sample multiple images.
<a id="S0293"></a> Source: p.15 S0293
When the chain is split after the prior draw at x , the samples differ significantly. T =1000 However, when the chain is split after more steps, samples share high-level attributes like gender, hair color, eyewear, saturation, pose and facial expression.
<a id="S0294"></a> Source: p.15 S0294
This indicates that intermediate latents like x encode these attributes, despite their imperceptibility. 750 Coarse-to-fine interpolation Figure 9 shows interpolations between a pair of source CelebA 256 256 images as we vary the number of diffusion steps prior to latent space interpolation. × Increasing the number of diffusion steps destroys more structure in the source images, which the 15
Page 16
<a id="S0295"></a> Source: p.16 S0295
model completes during the reverse process.
<a id="S0296"></a> Source: p.16 S0296
This allows us to interpolate at both fine granularities and coarse granularities.
<a id="S0297"></a> Source: p.16 S0297
In the limiting case of 0 diffusion steps, the interpolation mixes source images in pixel space.
<a id="S0298"></a> Source: p.16 S0298
On the other hand, after 1000 diffusion steps, source information is lost and interpolations are novel samples.
<a id="S0299"></a> Source: p.16 S0299
Source Rec. λ=0.1 λ=0.2 λ=0.3 λ=0.4 λ=0.5 λ=0.6 λ=0.7 λ=0.8 λ=0.9 Rec.
<a id="S0300"></a> Source: p.16 S0300
Source 1000 steps 875 steps 750 steps 625 steps 500 steps 375 steps 250 steps 125 steps 0 steps Figure 9: Coarse-to-fine interpolations that vary the number of diffusion steps prior to latent mixing. 10 8 6 4 2 0 200 400 600 800 1,000 Reverse process steps (T − t) erocSnoitpecnI 300 200 100 0 0 200 400 600 800 1,000 Reverse process steps (T − t) DIF Figure 10: Unconditional CIFAR10 progressive sampling quality over time 16
Page 17
<a id="S0301"></a> Source: p.17 S0301
Figure 11: CelebA-HQ 256 256 generated samples × 17
Page 18
<a id="S0302"></a> Source: p.18 S0302
(a) Pixel space nearest neighbors (b) Inception feature space nearest neighbors Figure 12: CelebA-HQ 256 256 nearest neighbors, computed on a 100 100 crop surrounding the × × faces.
<a id="S0303"></a> Source: p.18 S0303
Generated samples are in the leftmost column, and training set nearest neighbors are in the remaining columns. 18
Page 19
<a id="S0304"></a> Source: p.19 S0304
Figure 13: Unconditional CIFAR10 generated samples 19
Page 20
<a id="S0305"></a> Source: p.20 S0305
Figure 14: Unconditional CIFAR10 progressive generation 20
Page 21
<a id="S0306"></a> Source: p.21 S0306
(a) Pixel space nearest neighbors (b) Inception feature space nearest neighbors Figure 15: Unconditional CIFAR10 nearest neighbors.
<a id="S0307"></a> Source: p.21 S0307
Generated samples are in the leftmost column, and training set nearest neighbors are in the remaining columns. 21
Page 22
<a id="S0308"></a> Source: p.22 S0308
Figure 16: LSUN Church generated samples.
Page 23
<a id="S0309"></a> Source: p.23 S0309
Figure 17: LSUN Bedroom generated samples, large model.
Page 24
<a id="S0310"></a> Source: p.24 S0310
Figure 18: LSUN Bedroom generated samples, small model.