Course Syllabus 2026-2027 :

1. Introduction to NLP and Attention Mechanism

• Foundational Text Representations: Covers the progression from classic bag-of-words and n-grams to continuous vector spaces, including Word2Vec, GloVe, and subword tokenization schemes.
• Sequence Modeling & Attention: Explores recurrent architectures (RNNs, LSTMs, GRUs), sequence-to-sequence formulation, and the mathematical mechanics of additive and multiplicative attention (Bahdanau and Luong attention).
• Graph-of-Words (GoW) Framework: Integrates non-Euclidean text representation, replacing bag-of-words assumptions with sliding-window graph models that encode word co-occurrence, syntactic closeness, and phrase salience. Graph Degeneracy for Text Mining: Applies graph-theoretic metrics—such as $k$-core decomposition and shortest-path graph kernels—to keyword extraction, document degeneracy analysis, and text similarity.

2. Deep Learning for NLP

• Deep Discriminative Architectures: Examines 1D convolutional neural networks (CNNs), bidirectional encoders, and Hierarchical Attention Networks (HAN) designed for sentence and document classification.
• Transfer Learning & Pretraining: Details the shift from task-specific feature engineering to self-supervised pretraining (masked language modeling, sequence-to-sequence autoencoding).
• French & Multilingual Language Resources: Incorporates research from the DaSciM group on regional language adaptation, highlighting French sequence-to-sequence pretrained models such as BARThez.
• Efficient NLP Evaluation: Explores model compression and learned evaluation metrics, specifically integrating FrugalScore for cost-effective, high-correlation evaluation of generated text.

3. LLMs

• The Transformer Engine: Details scaled dot-product attention, multi-head self-attention mechanisms, and positional encoding variants (sinusoidal, Rotary Position Embeddings / RoPE, ALiBi).
• Autoregressive Pretraining & Scaling: Analyzes pretraining dynamics, causal masking, compute-optimal scaling laws (Chinchilla regimes), and modern tokenization behavior at scale.
• Downstream Adaptation & Parameter Efficiency: Covers instruction fine-tuning and parameter-efficient fine-tuning (PEFT) methods, with an emphasis on Low-Rank Adaptation (LoRA) and prefix tuning.
• Sovereign & Multilingual LLMs: Connects foundational models to regional sovereignty and evaluation, featuring native-sourced multitask benchmark design such as GreekMMLU to evaluate non-dominant languages.

4. LLM Advanced Topics

• Alignment & Post-Training: Studies preference optimization, including Reinforcement Learning from Human/AI Feedback (RLHF), Proximal Policy Optimization (PPO), and Direct Preference Optimization (DPO).
• Retrieval-Augmented Generation (RAG): Explores external knowledge retrieval, dense passage retrieval, context grounding, and methods for mitigating factual hallucination.
• Quantization & Efficient Inference: Examines low-bit weight and activation quantization (AWQ, GPTQ, bitsandbytes), KV-cache optimization, and pruning for resource-constrained deployment.
• Graph-Augmented LLMs: Explores structured knowledge retrieval frameworks, linking LLM reasoning capabilities with underlying knowledge graphs and structured relational databases.