Eval Factsheets

A web-based tool for generating standardized JSON evaluation factsheets for AI/ML model assessments. This tool helps researchers and practitioners create consistent, comprehensive documentation of their evaluation methodologies.

Quick Start

Try it now: https://evalevalai.com/EvalFactsheets/

Link to the paper: https://arxiv.org/abs/2512.04062

What are Eval Factsheets?

Eval Factsheets provide standardized documentation for AI model evaluations, similar to how Model Cards document model development. They help ensure transparency and reproducibility in evaluation practices by capturing:

• Evaluation Context and Scope
• Evaluation Structure and Method
• Evaluation Alignment

Relationship with Every Eval Ever

This repository serves as an auxiliary tool for the Every Eval Ever (EEE) project. While EEE stores the actual evaluation results and data, Eval Factsheets provides the metadata and documentation for the evaluation methodologies themselves.

Each evaluation in the EEE repository should have an associated factsheet. The factsheet generated by this tool should be placed in the corresponding evaluation's folder at the top level of the EEE repository structure (e.g., data/{eval_name}/factsheet.json).

Features

• Interactive Form Interface: Easy-to-use web form for inputting evaluation details.
• Interactive Database: Explore the csv database with a simple interface.
• Multiple Export Options:
  • Copy to clipboard with one click
  • Download as .json file
• Form Validation: Ensures all required fields are completed
• Responsive Design: Works seamlessly on desktop and mobile devices
• No Installation Required: Browser-based tool, no dependencies needed

How to Use

Basic Usage

1. Navigate to the tool: Visit https://evalevalai.com/EvalFactsheets/
2. Fill in the form: Enter your evaluation details in the provided fields
3. Generate JSON: Click the "Generate JSON" button
4. Export your factsheet:
   • Use "Copy to Clipboard" for quick pasting
   • Click "Download .json" to save the file locally

Use Cases

• Research Papers: Document evaluation methodology for academic publications
• Model Development: Track evaluation procedures during model iterations
• Team Collaboration: Share standardized evaluation details across teams
• Reproducibility: Provide clear documentation for others to replicate evaluations

Contributing

We welcome contributions! Here's how you can help:

Contributing to the EvalFactSheets Database

To contribute a new evaluation factsheet:

1. Generate the Factsheet: Use the Eval Factsheets tool to fill out the details of your evaluation and generate the JSON output.
2. Submit to Every Eval Ever:
   • Go to the Every Eval Ever repository.
   • Locate the folder for your evaluation under data/{eval_name}/.
   • Add your generated factsheet file (e.g., factsheet.json) to this folder.
   • Submit a Pull Request to the EEE repository.

Reporting Issues

If you find a bug or have a feature request:

1. Check if it's already reported in Issues
2. If not, create a new issue with:
   • Clear description of the problem/feature
   • Steps to reproduce (for bugs)
   • Expected vs actual behavior
   • Screenshots if applicable

Development Guidelines

• Keep code clean and well-commented
• Maintain the existing code style
• Test across different browsers
• Update documentation for new features

Attribution

This project is a fork of the original EvalFactsheets repository by Meta Research. We acknowledge and thank the original authors for their work.

License

This project is licensed under the cc-by-nc License - see the LICENSE.md file for details.

---

Citation: If you use this tool in your research, please cite:

@misc{bordes2025evalfactsheetsstructuredframework,
      title={Eval Factsheets: A Structured Framework for Documenting AI Evaluations}, 
      author={Florian Bordes and Candace Ross and Justine T Kao and Evangelia Spiliopoulou and Adina Williams},
      year={2025},
      eprint={2512.04062},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2512.04062}, 
}