Universal Sparse Autoencoders

Harrish Thasarathan, Julian Forsyth, Thomas Fel, Matthew Kowal, Konstantinos G. Derpanis

Universal Sparse Autoencoders (USAEs) create a universal, interpretable concept space that reveals what multiple vision models learn in common about the visual world.

• Paper (Arxiv)
• Project Page
• Interactive Demo: Concept Explorer

Getting Started

See setup instructions for Overcomplete. Python 3.8 or later is required.

Run uni_demo.py to set up and train a USAE: python -m uni_demo

Features

• USAE training framework
• Evaluation Notebook to validate metrics of trained USAE

To replicate the model used for our paper, see config.yaml file for hyperparameters. The training script assumes activations are in specified form (see uni_demo.py for details). Our dataset/loader is customized for Imagenet and assumes activations have been cached to .npz form.

Data Pipeline Details

During training, each batch is expected to be in the form:

• Model Activations X and labels Y in tuple ( X{Dict: (models s_1...s_k)}, Y) where each value from model key s_i is in form N(batch size) x d_i(activation dimension)

The Activation Dataset/Loader is customized for training on Imagenet:

• requires model activations to be cached
• assumes activation directory follows format of Imagenet directory
• recommended to store all models' activations for each image in one npz file (hence 'combined_npz' param)
• see models.py for model activation details

To cache model activations from Imagenet, see scripts in the caching_acts folder.

Recreating Paper Results

Download the checkpoints from the following link:

• ICML USAE Model Checkpoints
• Place each *.pth file in the checkpoints folder
• Modify config path in example_visualization.py to checkpoints/config.yaml and launch the script

Citation

If you use this work, please cite:

@inproceedings{
thasarathan2025universal,
title={Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment},
author={Harrish Thasarathan and Julian Forsyth and Thomas Fel and Matthew Kowal and Konstantinos G. Derpanis},
booktitle={Forty-second International Conference on Machine Learning},
year={2025},
url={https://openreview.net/forum?id=UoaxRN88oR}
}

Acknowledgements

This project was built on top of Thomas Fel's Overcomplete library.