Deep Learning in Practice

G. CHARPIAT

Deep LearningLearning

2nd semestre

Prè-requis

The « Introduction to Deep Learning » course by Vincent Lepetit (taking place during the 1st semester)

Notions in machine learning, Bayesian statistics, analysis, differential calculus.

Basic knowledge of Python and PyTorch.

Objectif du cours

Despite impressive mediatized results, deep learning methods are still poorly understood, neural networks are often difficult to train, and the results are black-boxes missing explanations, which is problematic given the societal impact of machine learning today (used as assistance in medicine, hiring process, bank loans…). Besides, real world problems usually do not fit the standard assumptions or frameworks of the most famous academic work (e.g., data quantity and quality). This course aims at providing insights and tools to address these practical aspects, based on mathematical concepts.

We will first emphasize the gap between practice and classical theory (e.g., number of parameters), and reconcile them thanks to recent theoretical advances.

After a review of recent architectures, we will study visualization techniques, and check that undesired biases present in the dataset (e.g., sensitivity to gender when matching CVs to job offers) are not reproduced.

We will then investigate practical issues when training neural networks, in particular data quantity (small or big data), application to reinforcement learning and physical problems, and automatic hyper-parameter tuning.

Presentation : here

Organisation des séances

The course will comprise lectures as well as practical and theoretical exercises (in PyTorch), which will be evaluated.

The practical sessions will be run on personal laptops (no GPU needed).

Announcements are made through a dedicated mailing-list; questions and discussions take place on an online forum.

All details are available on the course webpage: https://www.lri.fr/~gcharpia/deeppractice/ or when subscribing to the mailing-list.

Mode de validation

The evaluation of the course is based on 6 exercises.

Références

. Related conferences:
    – machine learning: NIPS / NeurIPS, ICML, ICLR
    – computer vision: CVPR, ICCV, ECCVRelated books:
    – « Deep Learning » by I. Goodfellow, Y. Bengio and A. Courville

 . Examples of references:
    – « Do Deep Nets Really Need to be Deep? »; L. J. Ba, R. Caruana; NIPS 2014
    – « Graph Attention Networks »; P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Lio, Y. Bengio; ICLR 2018
    – « Women also Snowboard: Overcoming Bias in Captioning Models »; L. A. Hendricks, K. Burns, K. Saenko, T Darrell, A Rohrbach; FAT-ML 2018
    – « Scaling description of generalization with number of parameters in deep learning »; M. Geiger, A. Jacot, S. Spigler, F. Gabriel, L. Sagun, S. d’Ascoli, G. Biroli, C. Hongler, M. Wyart; arXiv 2019
    – « Deep Sets »; M.Zaheer, S.Kottur, S. Ravanbakhsh, B. Poczos, R. Salakhutdinov, A. Smola; NIPS 2017
    – « DGM: A deep learning algorithm for solving partial differential equations »; J. Sirignano and K. Spiliopoulos; Journal of Computational Physics 2018