Defects4REST Version 1.0.0

A comprehensive benchmark and bug mining framework for systematically deploying, testing, and analyzing reproducible real-world bugs in REST API applications. The benchmark includes 110 defects across 12 real-world open-source projects along with deployment scripts, replication steps, and detailed metadata.

Features

• Single-command Deployment — Instantly deploy either the buggy or the patched version of the system.
• Pre-configured, Bug-specific Environments — For each bug, the benchmark initializes the setup required to reproduce it, such as admin accounts, API tokens, test data, or other necessary configuration when applicable.
• Step-by-Step Guides — Detailed, Bug-specific replication steps provided for all 110 bugs.

Supported Projects

Project	Bugs
awx	5
dolibarr	25
enviroCar-server	4
flowable-engine	5
kafka-rest	3
mastodon	5
netbox	6
nocodb	6
podman	23
restcountries	16
seaweedfs	9
signal-cli-rest-api	3

See the complete list of all bugs here

Installation

Prerequisites

Linux / macOS:

• Python 3.9+
• Docker and Docker Compose
• Git
• Go 1.16+

Windows

• Python 3.9+
• Docker and Docker Compose
• Git
• Go 1.16+
• WSL2 installed with Ubuntu
• Docker Desktop with WSL integration enabled

Project-Specific System Requirements (Podman)

Note: The following requirements apply only when replicating defects from the Podman subject.

• Linux environment:
  • Native Linux
  • Windows Subsystem for Linux (WSL)
  • Linux virtual machine on macOS (e.g., Podman machine)
• sudo privileges
• System packages:

sudo apt-get update

sudo apt-get install -y conmon btrfs-progs gcc git golang-go go-md2man iptables libassuan-dev libbtrfs-dev libc6-dev libdevmapper-dev libglib2.0-dev libgpgme-dev libgpg-error-dev libprotobuf-dev libprotobuf-c-dev libseccomp-dev libselinux1-dev libsystemd-dev make netavark passt pkg-config runc uidmap

Install from Source

Recommended: Create a new Python virtual environment before installing.

# Create and activate virtual environment (recommended)
python -m venv defects4rest-env
source defects4rest-env/bin/activate  # On Windows: defects4rest-env\Scripts\activate

# Clone the repository
git clone https://github.com/ANSWER-OSU/Defects4REST.git
cd Defects4REST

# Install in development mode
pip install -e .

# Verify installation
defects4rest --help

Once installed, you can run defects4rest from any directory as long as the virtual environment is activated.

Updating to Latest Version

If you have an older version installed, pull the latest changes and reinstall:

cd Defects4REST
git pull
pip install -e .

Note: Just pulling the repository won't update the installed package, you must rerun pip install -e . again to install the cloned version.

Usage

The CLI provides two commands: info to view bug details and checkout to deploy environments.

# View bug details
defects4rest info -p <project> -i <issue>

# Deploy buggy version
defects4rest checkout -p <project> -i <issue> --buggy

# Deploy patched version
defects4rest checkout -p <project> -i <issue> --patched

# Cleanup
defects4rest checkout -p <project> -i <issue> --clean

See detailed documentation: info | checkout

OpenAPI Specifications

For each bug, the OpenAPI specifications are available at:

bug_replication/<project>/<project>#<issue>/<project>#<issue>_spec.json/yaml

(e.g., bug_replication/awx/awx#2311130/awx#2311130_spec.json)

REST API Defect Taxonomy

Defect Type	Sub Defect Type
Configuration and Environment Issues (T1)	Container and Resource Quota Handling Errors (ST1)
	Job Execution and Workflow Configuration Defects (ST2)
	Environment-Specific Behavior and Configuration Bugs (ST3)
Data Validation and Query Processing Errors (T2)	Schema and Payload Validation Errors in POST APIs (ST4)
	Query Filter and Search Parameter Handling Errors (ST5)
Authentication, Authorization, and Session Management Issues (T3)	Authentication and Token Management Errors (ST6)
	Session, Token, and Account Lifecycle Management Errors (ST7)
Integration, Middleware, and Runtime Environment Failures (T4)	Middleware Integration Failures in REST APIs (ST8)
	Process Signal and Grouping Issues in Containerized APIs (ST9)
	Runtime and Dependency Errors (ST10)
Data Storage, Access, and Volume Errors (T5)	Volume and File Upload/Access Errors (ST11)
	Database/Table User Access Handling Errors (ST12)
Distributed Systems and Cluster Failures (T6)	Index and Cluster Coordination Failures (ST13)

Versioning

Defects4REST uses a semantic versioning scheme (major.minor.patch):

Change	major	minor	patch
Addition/Deletion of projects	X
Addition of new bugs to existing projects		X
Fixes and documentation changes			X

Contributing

We welcome contributions! See our guides:

• Adding New Bugs — Step-by-step guide to add new defects

License

This project is licensed under the MIT License - see the LICENSE file for details.