CENG524 Advanced Paradigms in NLP

Information

Course Goal / Objective

The primary objective of this course is to explore state-of-the-art (SotA) architectures and paradigms in Natural Language Processing that extend beyond standard Transformer models. The course aims to develop students' ability to critically analyze recent top-tier research papers, understand fundamental architectural shifts in the LLM ecosystem, and apply these advanced concepts to formulate and solve complex research problems in generative AI.

Course Content

This research-oriented course focuses on the latest advancements and structural shifts in modern NLP. Key topics include overcoming the quadratic bottlenecks of standard Transformers through alternative architectures like State Space Models (e.g., Mamba); dynamic compute allocation strategies such as Mixture of Experts (MoE) and Mixture of Depths (MoD); and the shift towards inference-time compute and "System 2" logical reasoning. Additionally, the course covers the mathematical foundations of Parameter-Efficient Fine-Tuning (PEFT), the evolution of Large Action Models (LAMs) for autonomous GUI/OS interactions, and natively multimodal (Omni) architectures. The curriculum is heavily driven by literature review, paper discussions, and advanced research projects.

Course Precondition

none

Resources

- Scaling Laws for Neural Language Models (Kaplan et al., 2020) - Training Compute-Optimal Large Language Models [Chinchilla] (Hoffmann et al., 2022) - Attention Is All You Need (Vaswani et al., 2017) - FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (Dao et al., 2022) - LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021) - QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., 2023) - Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al., 2023) - Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020) - Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023) - From Local to Global: A Graph RAG Approach to Query-Focused Summarization (Edge et al., 2024) - A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (Ji et al., 2023) - SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (Manakul et al., 2023) - Mamba: Linear-Time Sequence Modeling with Selective State Spaces (Gu & Dao, 2023) - Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al., 2022) - Let's Verify Step by Step (Lightman et al., 2023) - Mixtral of Experts (Jiang et al., 2024) - Mixture-of-Depths: Dynamically allocating compute in transformer-based language models (Raposo et al., 2024) - OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (Xie et al., 2024)

Notes

- AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling (Zhan et al., 2024)

Course Learning Outcomes

Relation with Program Learning Outcome

Week Plan

Student Workload - ECTS