Modèles génératifs pour l’image

B. GALERNE, A. LECLAIRE

Image processingModelling

Prè-requis

Les bases d’un M1 de mathématiques appliquées en probabilités, statistiques et optimisation.

Avoir suivi le cours « Introduction to Probabilistic Graphical Models and Deep Generative Models » est conseillé mais non obligatoire.

Objectif du cours

The goal of this course is to present and study generative models that can be used for various image generation tasks. The first part of the course will focus on models relying on an adversarial framework, namely Generative Adversarial Networks (GAN) and Wasserstein Generative Adversarial Networks (WGANs) with a particular focus on the dual formulations related to optimal transport. Applications of these models for conditional image generation and imaging inverse problems will be discussed.

The remainder of the course will be dedicated to the study of a new contender in generative modeling called diffusion models or Score-Based Generative Models. We will study the mathematical aspects (Time reversal of stochastic processes, …) and practical aspects of these algorithms. In particular we will introduce the core concepts of practical implementations like DDPM (Denoising Diffusion Probabilistic Models) and DDIM (Denoising Implicit Diffusion Models) and their application to imaging inverse problems and text-to-image synthesis. We will also see how score-based denoisers can be used to address plug-and-play image restoration, with the benefits of both explicit prior and convergence guarantees.

Classes will be given in French but slides will be written in English.

Site web : https://generativemodelingmva.github.io/

Organisation des séances

9 séances de 3h

Mode de validation

1 Devoir maison obligatoire à mi-cours
1 Projet avec étude d’un article à la fin du cours

Références

Generative Adversarial Networks (Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio, 2014)

Auto-Encoding Variational Bayes (Kingma and Welling, 2014)

An Introduction to Variational Autoencoders (Kingma and Welling, 2019)

Wasserstein GAN (Martin Arjovsky, Soumith Chintala, Léon Bottou, 2017)

A style-based generator architecture for generative adversarial networks (Karras, T., Laine, S., and Aila, T., 2019)

SinGAN: Learning a Generative Model from a Single Natural Image (Tamar Rott Shaham, Tali Dekel, Tomer Michaeli, 2019)

Image-To-Image Translation With Conditional Adversarial Networks (Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros, 2019)

Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network (Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, Wenzhe Shi, 2017)

Generative Modeling by Estimating Gradients of the Data Distribution (Yang Song, Stefano Ermon, 2019)

Score-Based Generative Modeling through Stochastic Differential Equations (Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, Ben Poole, 2021)

Denoising Diffusion Probabilistic Models (Jonathan Ho, Ajay Jain, Pieter Abbeel, 2020)

Denoising Diffusion Implicit Models (Jiaming Song, Chenlin Meng, Stefano Ermon, 2020)

Score-based Generative Modeling in Latent Space (Arash Vahdat, Karsten Kreis, Jan Kautz, 2021)

The Little Engine That Could: Regularization by Denoising (RED) (Yaniv Romano, Michael Elad, and Peyman Milanfar, 2017)

Gradient Step Denoiser for convergent Plug-and-Play (Samuel Hurault, Arthur Leclaire, and Nicolas Papadakis, 2022)

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, Mohammad Norouzi, 2022)

High-Resolution Image Synthesis with Latent Diffusion Models (Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer, 2022)