Adversarial Attacks on Maximum Entropy IRL

This repository contains the source code for a research project on adversarial attacks against Maximum Entropy Inverse Reinforcement Learning (MaxEnt IRL), implemented using the imitation library.

---

📖 Table of Contents

• Project Overview
• Directory Structure
• Setup and Installation
• Usage: Running Experiments
• Code Explanation
• Notes

---

🚀 Project Overview

This project investigates the robustness of Inverse Reinforcement Learning (IRL) to adversarial data. The core experiment involves:

1. Expert Policy: Solve for the optimal policy in a tabular environment (CliffWorld or FourRooms).
2. Adversarial Policy: Find a “malicious” policy that minimizes the expected reward while keeping its feature expectations (state visitation frequencies) within an ε-ball of the expert’s.
3. Data Poisoning: Create mixed datasets with varying ratios of expert and adversarial trajectories.
4. Evaluation: Run MaxEnt IRL on these poisoned datasets and measure how the learned reward and downstream policy performance degrade.

---

📁 Directory Structure

The repository is organized as follows:

├── configs/         # Configuration files for experiments (e.g., hyperparameters).
├── data/
│   ├── raw/         # Raw, immutable data (e.g., expert trajectories).
│   └── processed/   # Processed or cleaned data.
├── docs/            # Project documentation.
├── logs/            # Experiment logs.
├── models/          # Saved model checkpoints.
├── notebooks/       # Jupyter notebooks for analysis and visualization.
├── outputs/         # Generated plots, figures, and results.
├── scripts/         # Main scripts for running training and evaluation.
├── src/
│   └── maxent_irl_adv/ # Main source code for the project.
├── .gitignore       # Specifies files for git to ignore.
└── README.md        # This file.

⚙️ Setup and Installation

1. Clone the repository:

git clone git@github.com:CL2-UWaterloo/adversarial-attacks-irl.git
cd adversarial-attacks-irl

2. Create and activate a Python virtual environment:

# Create a virtual environment named 'venv'
python3 -m venv venv

# Activate the environment (on Linux/macOS)
source venv/bin/activate

3. Install the required packages:

# Upgrade pip and install all dependencies
pip install --upgrade pip
pip install -r requirements.txt

---

▶️ Usage: Running Experiments

Update the script names/flags to match your repo if they differ.

---

📌 Notes

• Built on top of the imitation library (MaxEnt IRL and helpers).
• Results can vary with random seeds, environment configs, and the adversary’s ε.
• If you use this code or ideas in academic work, please cite the project and the imitation library.