Skip to content

CRITICAL: Test name extraction via CLI args is unreliable #31

@siphonite

Description

@siphonite

Problem

The orchestrator attempts to infer the test name by parsing std::env::args(). This logic is fragile and prone to failure.

Code reference:
https://github.com/siphonite/first/blob/main/src/orchestrator.rs#L328-L348

fn extract_test_name() -> Option<String> {
    // ...
    // Skip the executable path, look for something that looks like a test name
    for arg in args.iter().skip(1) {
        // ... (skipping flags)
        return Some(arg.clone());
    }
    None
}

This fails when:

  • No explicit test filter is provided.
  • Flags are reordered or complex.
  • Cargo behavior changes (e.g., passing extra arguments).

If extraction fails, child processes may run all tests (default behavior when no test name is given), causing recursive spawning or cross-test interference.

Impact

  • Exponential test spawning: If a child process meant to run a specific test instead runs all tests, it will spawn its own children, leading to a fork bomb.
  • Cross-test contamination: Tests interfering with each other's state.
  • Potential infinite recursion: The orchestrator spawning itself endlessly.

Acceptance Criteria

  • Test identity must be unambiguous and explicit.
  • Preferred: mandatory FIRST_TEST_NAME environment variable set by the orchestrator when spawning children.
  • Or: robust mechanism (not CLI parsing) to identify the current test.
  • Orchestrator must refuse to run if test identity is unknown.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingdesign

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions