Microservice System Testing Tool

This project is a tool designed to test a microservice system. The most novel part of this tool is its ability to utilize traces collected from the system to guide the testing process.

Table of Contents

• Introduction
• Features
• Architecture
• Dependencies
• Installation
• Preparation
• Integration with other tools
• Usage
• Configuration
• License

Introduction

Microservice systems are complex and require thorough testing to ensure reliability and performance. This tool leverages system traces to guide the testing process, providing a more efficient and effective way to identify issues and ensure system robustness.

Features

• Trace-Guided Testing: Utilizes system traces to guide the testing process.
• Comprehensive Coverage: Ensures thorough testing of all microservice endpoints.
• Automated Test Case Generation: Automatically generates test cases based on system traces.
• Detailed Reporting: Provides detailed reports on test coverage and results, including endpoints coverage and internal service coverage.

Architecture

You can get details of architecture design here

Dependencies

The tool is still under development, below is our current dependencies:

• X86-64 architecture
• Ubuntu 22.04 LTS
• Go 1.23

We plan to support more architectures and operating systems in the future.

Installation

To install the tool, follow these steps:

1. Clone the repository:

2. Install dependencies:

   go mod tidy

Preparation

1. Prepare the OpenAPI specification file for the system under test. The tool supports OpenAPI 3 by default.
2. Prepare a protobuf file for all RPC which internal services use, or the OpenAPI specification file for all REST APIs which internal services use.

• We use protoc-gen-openapi to convert protobuf to openapi.
• You should annotate the proto file, and you can refer to this issue.
• If you use OpenAPI spec, you should ensure that the OpenAPI spec is in the same format as a file generated by the tool (including operationId). Note that you must add tag APIType_HTTP to all HTTP APIs, as we treat internal service APIs as RPC APIs by default. You can refer to the example at the end of this section.

3. Optionally, you can provide API dependencies to enhance fuzzing, including system API dependencies and internal service API dependencies (a map from service name to single service dependencies). We support parse dependency file generated by RESTler now, you can refer to its Github Repo for more information.
4. Ensure that your service includes a unique trace ID in the headers of each response. You can configure the header key in the settings.

Internal service API Spec example:

/v1/cart/add:
    post:
        tags:
            - CartService
            - APIType_gRPC
        operationId: CartService_AddItem
        requestBody:
            content:
                application/json:
                    schema:
                        $ref: '#/components/schemas/AddItemRequest'
            required: true
        responses:
            "200":
                description: OK
                content:
                    application/json:
                        schema:
                            $ref: '#/components/schemas/Empty'
            default:
                description: Default error response
                content:
                    application/json:
                        schema:
                            $ref: '#/components/schemas/Status'

Integration with Other Tools

Restler

Restler is a tool designed to generate REST API test cases from OpenAPI specifications. During the compilation process, it produces a dependency file for the system's APIs. Our tool can parse this dependency file to enhance the fuzzing process. For more details, refer to the --dependency-file and --dependency-file-type options.

CoSREST

CoSREST is a tool that leverages Large Language Models (LLMs) to generate REST API test cases from OpenAPI specifications. It enhances test case generation by using LLMs to make decisions on input space partitioning (e.g., parameter generation), which improves both efficiency and effectiveness. Our tool supports parsing the space partition file generated by CoSREST and converts it into a fuzz value dictionary. This helps the fuzzer create more diverse and valid test cases. For more details, refer to our script

Usage

To use the tool, follow these steps:

1. Build the project:

   make build

2. Run the tool:

   make run

3. Clean the project:

   make clean

   Or you can clean build and output files separately:

   make clean-build
   make clean-output

In addition, we provide vscode tasks to use the tool. You can build, run and debug the project by selecting the task Run in vscode. See the .vscode/tasks.json file for more details.

Configuration

The tool can be configured using command-line arguments. The following options are available:

• --config-file: Path to the config file. If an argument is provided in both the config file and command line, the config file argument will be used.
• --dependency-file: Path to the dependency file generated by other tools or manually.
• --dependency-file-type: Type of the dependency file. Currently only supports 'Restler'. Required if --dependency-file is provided.
• --enable-energy-operation: Enable energy (priority) of test operations. If true, energy affects the test operation selection when extending the test scenario.
• --enable-energy-scenario: Enable energy (priority) of test scenarios. If true, energy affects the test scenario selection when starting a new test loop.
• --extra-headers: Extra headers to be added to the request, in the format of stringified JSON, e.g., {"header1": "value1", "header2": "value2"}.
• --fuzz-value-dict-file: Path to the file containing the dictionary of fuzz values, in JSON format. Each element is a dictionary with name (string) and value (any JSON).
• --fuzzer-budget: The maximum time the fuzzer can run, in seconds (default: 5).
• --fuzzer-type: Type of the fuzzer. Currently only supports 'Basic' (default: Basic).
• --http-client-dial-timeout: Timeout for the HTTP client dial, in seconds (default: 30).
• --http-middleware-script: Path to the script file that contains the HTTP middleware functions.
• --internal-service-api-dependency-file: Path to the internal service API dependency file generated by other tools or manually. It should be a map of service name to a list of API dependencies.
• --internal-service-openapi-spec: Path to the internal service OpenAPI specification file (required).
• --log-level: Log level: debug, info, warn, error, fatal, panic (default: info).
• --log-to-file: Whether to log to a file (default: false).
• --max-ops-per-scenario: Maximum number of operations to execute in each scenario (default: 1).
• --max-allowed-operation-case-executed-count: Maximum number of times a test operation case can be executed (default: 14).
• --max-allowed-operation-cases: Maximum number of test operation cases in the queue of an API method (default: 7).
• --max-allowed-scenario-executed-count: Maximum number of times a test scenario can be executed (default: 6).
• --max-allowed-scenarios: Maximum number of test scenarios in the queue (default: 114).
• --openapi-spec: Path to the OpenAPI specification file (required).
• --output-dir: Directory to save the output reports (default: ./output).
• --server-base-url: Base URL of the server to test (default: https://www.example.com).
• --trace-backend-type: Type of the trace backend. Currently supports 'Jaeger' and 'Tempo' (default: Jaeger).
• --trace-backend-url: URL of the trace backend (required).
• --trace-id-header-key: The response header key to be used for trace ID (default: X-Trace-Id).
• --use-internal-service-api-dependency: Indicates whether to use the internal service API dependency. If true, the internal service API dependency will be used to enhance the external service API dependency.

You can also use a configuration file with the --config-file option to set the options. The configuration file should be in JSON format. We provide an example configuration file here.

For sensitive information like tokens or private keys, you can use environment variables. Set an environment variable with the same name as the configuration file key (in uppercase) to override the value in the configuration file. For example, you can set SERVER_BASE_URL in a .env file:

SERVER_BASE_URL=http://localhost:6789

About HTTP Middleware Script

The HTTP Middleware Script allows you to intercept and modify HTTP requests and responses using a Starlark script. The script can modify headers, path parameters, query parameters, and the body of the request.

Here is an example of a Starlark script that modifies the headers, path parameters, query parameters, and body of the request:

# Example Starlark script
headers = {"Authorization": "Bearer new_token"}
pathParams = {"id": "123"}
queryParams = {"search": "new_query"}
body = "[1, 2, 3]"

For more information on Starlark, see the Starlark documentation.

License

This project is licensed under the GPL-3.0 License - see the LICENSE file for details.