Simple Grimoire

A version of the Grimoire algorithm implemented on top of a simple python coverage fuzzer.

The code contains modifications of code found within the original Grimoire repository (https://github.com/RUB-SysSec/grimoire), which is licensed under the AGPL-3.0 license.

Getting Started

System Requirements

We have successfully run SimpleGrimoire on:

• Windows
• Mac (Intel): 16GB RAM
• Mac (M1)
• Linux: 64 core, 132GB RAM VM

Setup

Download Python3: https://www.python.org/downloads/ Install the required python packages.

Usage

Running Grimoire and Evaluation

The eval.sh script runs Grimoire on benchmarks and then runs evaluation on the benchmarks. Run the following on the command line:

./eval.sh <benchmark>

Replace <benchmark> with:

• nothing, to run all single-file benchmarks
• calculator (source)
• cgidecode (source)
• mathexpr (source)
• microjson (source)
  • Note: with the current implementation, program runs out of memory when creating Lark parser.
• sexpr (source)
• urlparse (source)
• apimd (source)
  • Multi-file benchmark, with apimd_parser.py as the entry point. Original repo: https://github.com/KmolYuan/apimd

Running Grimoire Fuzzer only

Run the following on command line:

Run with: `python3 main.py <benchmark.path>.<benchmark> --fuzzer GRIMOIRE --time <seconds> --input_dir <benchmark seed path>/<benchmark>`

Benchmarks:

• calculator (source)
• cgidecode (source)
• mathexpr (source)
• microjson (source)
• sexpr (source)
• urlparse (source)

eg.

python3 main.py new_benchmarks.calculator --fuzzer GRIMOIRE --time 10 --input_dir new_benchmarks/unified_train_set/calculator

Running Evaluation only

To run this, you will need to have an existing output folder from running Grimoire on the benchmark. Run the following on command line:

python3 eval.py --benchmarks_dir new_benchmarks --benchmark <benchmark> --output_parent_dir output/<benchmark output folder> --golden_input_parent_dir new_benchmarks/unified_test_set

Replace <benchmark> with a benchmark name, and <benchmark output folder> with the folder created from running Grimoire on the benchmark.

If successful, a benckmark_output.txt file will be generated in the same output folder, with precision and recall information at the bottom.

Documentation

Grimoire fuzzer: details of implementation of the Grimoire algorithm.

Grimoire Pipeline: details of benchmark evaluation.

EBNF Generation: details of generating an EBNF grammar from Grimoire's generalized inputs.

Replacement Classes: overview of replacement classes used by Replacement Grimoire.