How To AI (Almost) Anything

[chengjun]

MAS.S60 How2AI
Schedule
** Exact topics and schedule subject to change, based on student interests and course discussions. **

Date Topics Readings
2/4 Week 1 Introduction [slides] [video]
Course syllabus and requirements
Introduction to AI and AI research
Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
Multimodal Machine Learning: A Survey and Taxonomy
Representation Learning: A Review and New Perspectives

2/6 Week 1 Introduction to AI Research [slides] [video]
Introduction to AI and AI research
Generating ideas, reading and writing papers, AI experimentation

2/11 Week 2 Foundation 1: Data, structure, information [slides] [video]
Common data modalities
Data collection strategies
Training objectives and generalization
Machine learning: Trends, Perspectives, and Prospects
Representation Learning: A Review and New Perspectives
Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges

2/14 Week 2 Foundation 2: Practical AI tools [slides] [video]
Getting started with PyTorch
Huggingface packages
Debugging machine learning models
A Recipe for Training Neural Networks
Fine-tuning a Code LLM on Custom Code on a single GPU
MAS.S60 Pytorch Introduction

2/18 Week 3 No class, shifted President's day
2/20 Week 3 Project proposal presentations

2/25 Week 4 Foundation 3: Model architectures [slides] [video]
Structure and invariances
Temporal sequence models
Spatial convolution models
Models for sets and graphs
Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Attention Is All You Need
Neural Machine Translation by Jointly Learning to Align and Translate
Deep Sets
Graph Attention Networks

2/25 Week 4 Discussion 1: Learning and generalization
Learning the Bitter Lesson
Unifying Grokking and Double Descent
Generalization in Neural Networks
Textbooks are all you Need
A Conceptual Pipeline for Machine Learning

3/4 Week 5 Multimodal 1: Connections and alignment [slides] [video]
Heterogeneity, connections, and interactions
Multimodal technical challenges
Alignment and transformers
Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
What Makes for Good Views for Contrastive Learning?
Characterization and classification of semantic image-text relations
When and why vision-language models behave like bags-of-words, and what to do about it?

3/6 Week 5 Discussion 2: Modern AI architectures
Scaling Laws for Generative Mixed-Modal Models
Not All Tokens Are What You Need for Pretraining
PaLI: A Jointly-Scaled Multilingual Language-Image Model
The Evolution of Multimodal Model Architectures
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
A ConvNet for the 2020s
Inductive Representation Learning on Large Graphs
Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs

3/11 Week 6 Multimodal 2: Interactions and fusion [slides] [video]
Cross-modal interactions
Multimodal fusion
Ten Myths of Multimodal Interaction
Multimodal interaction: A review
Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework
Does my multimodal model learn cross-modal interactions? It’s harder to tell than you might think!

3/13 Week 6 Discussion 3: Multimodal alignment
The Platonic Representation Hypothesis
What Makes for Good Views for Contrastive Learning?
Understanding the Emergence of Multimodal Representation Alignment
Does equivariance matter at scale?
Learning Transferable Visual Models From Natural Language Supervision?
Emerging Properties in Self-Supervised Vision Transformers
Foundations & trends in multimodal machine learning - Principles, challenges, and open questions

3/18 Week 7 Multimodal 3: Cross-modal transfer [slides] [video]
Cross-modal learning via fusion
Cross-modal learning via alignment
Cross-modal learning via translation
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
DreamLLM: Synergistic Multimodal Comprehension and Creation
PaLM-E: An Embodied Multimodal Language Model

3/20 Week 7 Discussion 4: Multimodal interactions
Multimodal interaction: A review
Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework
Does my multimodal model learn cross-modal interactions? It’s harder to tell than you might think!
Kosmos-2: Grounding Multimodal Large Language Models to the World
Chameleon: Mixed-modal early-fusion foundation models
MM1: Methods, Analysis and Insights from Multimodal LLM Pre-training
MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts

3/25 Week 8 No class, spring break

4/1 Week 9 Large models 1: Large foundation models [slides] [video]
Pre-training data
Self-supervised learning
Fine-tuning, instructing, alignment
Training Compute-Optimal Large Language Models
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
LoRA: Low-Rank Adaptation of Large Language Models
A Visual Guide to Mixture of Experts (MoE)
A Visual Guide to Quantization
Improved Baselines with Visual Instruction Tuning

4/3 Week 9 Project midterm presentations

4/8 Week 10 No class, member's week

4/15 Week 11 Large models 2: Large multimodal models [slides] [video]
Multimodal pre-training
Adapting large language models to multimodal
Multimodal LLMs with generation
Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework
Multimodal Transformer for Unaligned Multimodal Language Sequences
Masked Autoencoders Are Scalable Vision Learners
Scaling Laws for Native Multimodal Models Scaling Laws for Native Multimodal Models
Transfer between Modalities with MetaQueries

4/17 Week 11 Discussion 5: Large language models
LoRA: Low-Rank Adaptation of Large Language Models
Gated Linear Attention Transformers with Hardware-Efficient Training
Unintended Impacts of LLM Alignment on Global Representation
A Visual Guide to Quantization
Scaling Instruction-Finetuned Language Models

4/22 Week 12 Large models 3: Modern generative models [slides] [video]
Diffusion models
Controllable generation
Flow Matching
Scalable Diffusion Models with Transformers
Flow Matching in Latent Space
Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
Movie Gen: A Cast of Media Foundation Models

4/24 Week 12 Discussion 6: Large multimodal models
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention
Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference
ModaVerse: Efficiently Transforming Modalities with LLMs
Spider: Any-to-Many Multimodal LLM
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models
How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites
NExT-GPT: Any-to-Any Multimodal LLM
Learning to rebalance multi-modal optimization by adaptively masking subnetworks

4/29 Week 13 No class, CHI week
5/1 Week 13 Discussion 7: Generative AI
Large Language Diffusion Models
Compositional Generative Modeling: A Single Model is Not All You Need
Flow Matching for Generative Modeling
Flow Matching Guide and Code
FlowMotion: Target-Predictive Conditional Flow Matching for Jitter-Reduced Text-to-Motion Generation
MusFlow: Multimodal Music Generation via Conditional Flow Matching
Unraveling the Connections Between Flow Matching and Diffusion Probabilistic Models
Exploring Diffusion and Flow Matching Under Generator Matching

5/6 Week 14 Interaction 1: Interactive agents and reasoning [slides] [video]
Reinforcement learning
Multi-step reasoning
Deep reinforcement learning from human preferences
Deepseek-r1: Incentivizing reasoning capability in LLMs via reinforcement learning
Faulty reward functions in the wild
Direct preference optimization: Your language model is secretly a reward model

5/8 Week 14 Project final presentations

5/13 Week 15 Interaction 2: Human AI interaction [slides] [video]
Interaction mediums
Human in the loop learning
Safety and reliability
Interactive Sketchpad: A Multimodal Tutoring System for Collaborative, Visual Problem-Solving
VideoWebArena: Evaluating Multimodal Agents on Video Understanding Web Tasks
OpenVLA: An Open-Source Vision-Language-Action Model
ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs
Guidelines for Human-AI Interaction