Prè-requis
Python with numpy proficiency is required. Some familiarity with scipy, opencv, scikit-image, pandas, and one of the major neural network frameworks (pytorch, tensorflow, jax) is recommended. Jupyter will be used for interactive teaching and for evaluation.
Objectif du cours
Art history and cultural heritage science are undergoing rapid advances due to the simultaneous advent of new and inexpensive imaging modalities, fast computation, and ubiquitous high-speed internet. We can easily collect gigabytes or terabytes of multimodal imaging data about an artwork such as a painting, offering the ability to dramatically improve our understanding of who made it, how it was made, its state of conservation, and its relation to other artworks. Additionally, cultural heritage institutions are digitizing millions of works
and making them freely available, presenting opportunities for researchers to investigate and visualize central art-historical questions such as attribution, style, and iconography in new ways.
In all cases, extracting the full value from these data requires the extensive use of sophisticated computational tools. The aims of this course are two-fold: first, to familiarize students with the rich and little-known world of cultural heritage imaging and its many
fascinating and challenging problems; and second, to provide an introduction to a selection of the most important recent developments in computer vision and deep learning to solve these problems.
Organisation des séances
Four 3-hours lectures
A considerable portion of each session will involve hands-on work with real data from across the instructor’s 19 years of experience in the field of cultural heritage imaging. Course readings will include important papers from the computer vision and deep learning fields
(KeyNet, MAGSAC++, Neural Style Transfer, U-Net, CLIP, DINOv2, etc.)
Mode de validation
Evaluation will occur via projects submitted via code and documented and demonstrated via Jupyter notebooks.
Références
Additional reading :
Tummers, A. (2011). The eye of the connoisseur: Authenticating paintings by Rembrandt and his contemporaries. J. Paul Getty Museum.
Thèmes abordés
1. Overview of technical and chemical imaging; relation to materiality
a. high-resolution visible photography (VIS)
b. infrared reflectography (IRR)
c. reflectance imaging spectroscopy (RIS)
d. radiography
e. macro x-ray fluorescence (MA-XRF)
f. structured-light scanning
2. Cross-modal image registration
a. Deep networks for feature detection, description, and matching: KeyNet; AffNet; HardNet
b. Robust parametric estimation of geometric transforms: homographies; lens distortion; MAGSAC++
c. Learning nonlinear filters for precise patchwise offset estimation: scale space theory; structure tensor; similarity metric design; strategies for parallel tile-stack cross-correlation
3. Modeling conditional expectation in multimodal imaging of oil paintings
a. Density estimation for low-dimensional data (R, G, B, IRR, e.g.); surprisal (examples from Hieronymus Bosch)
b. CNNs to predict color from chemistry (VIS from MA-XRF); detection of restoration (examples from Rembrandt)
4. Direct microstructural analysis of paintings
a. Crack type classification; craquelure segmentation and analysis from high-resolution visible photography (examples from Van Eyck)
b. Canvas analysis from painting radiography; reuniting paintings cut from the same roll (examples from Poussin, Van Gogh, Rembrandt, Frans Hals)
5. Text and image embeddings with pre-trained networks
a. Joint image-text embedding with CLIP, including applications:
i. Cultural heritage image search engine
ii. Semantic directions in the embedding space
iii. High-resolution microstructural image comparison
b. Self-supervised embeddings with DINOv2, including applications:
i. Semantic correspondences between artworks via comparison of local patch embeddings
ii. Visualization via PCA
6. Image-to-image translation with CNNs
a. U-Net architecture
b. VGG16, Gram matrix, principles of style transfer
c. Case study: the Magic Eraser (ink removal to detect watermarks in works on paper)
d. Case study: Reconstructing the missing parts of Rembrandt’s Night Watch
Robert G. ERDMANN
(Univ. Amsterdam)