README

NewComputeBench is a project to develop a benchmark suite for the new compute paradigm (Spiking neural networks, Optical computation, In-Memory computation, etc). The project is divided into three main components:

• Model Training
• Model Behavior-Level Simulation
• Hardware-Performance Simulation

🔖 For tutorials and examples, please refer to this site.

Model Training

LLMs

We adopt Llama-3 architecture and aim to support the following features:

• Pretraining
• Generation (inference)
• Parameter-efficient fine-tuning
• 🚧 TODO 🐌 LowPriority: Supervised-fine-tuning
• Evaluation

PreTraining

The LLM pretraining is built on top of torchtitan.

• Model architecture: Llama3
• Model configs: 60M, 200M, 400M, 1.1B
• Datasets: HuggingFaceFW/fineweb
• HuggingFace checkpoints: AICrossSim

Generation

We recommend using the HuggingFace Transformers library for generation tasks. We provide a script to convert the torchtitan checkpoint to a HuggingFace checkpoint (See this file).

Parameter-Efficient Fine-tuning

• For models larger than 1.1B, we fine-tune pretrained checkpoints.
  • LoRA fine-tuning data
  • LoRA fine-tuning scripts

Model Behavior Simulation

• Random bitflip

  • Post-training bitflip transform
  • Bitflip-aware pretraining

• Optical compute

  • Roberta on GLUE
  • CLM 🚧 WIP

• Spiking neural networks 🚧 TODO

• In-memory compute 🚧 TODO

Hardware-Performance Simulation

🚧 TODO