Terminal bench artifacts + adding memory layer #1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Refactor StateManager to use centralized config for state directory - Add state module initialization - Add runs/ directory to .gitignore to exclude benchmark artifacts
edgarpavlovsky
force-pushed
the
e/clean-benchmark
branch
from
20d9e29 to
dffdf8c
Compare
edgarpavlovsky
changed the title
edgarpavlovsky
changed the title
- Created 165 total tests (161 unit + 4 new e2e/integration) - Added test infrastructure (conftest.py, helpers.py) - Enhanced MemoryManager with embedding_model parameter - Added lightweight embedding tests for fast CI - Added E2E hello world subprocess test - Added terminal-bench integration test - Created GitHub Actions workflow with 3 jobs - Updated documentation and added TODO for improvements - Fixed config.py .env loading from repo root - All fast tests passing (163/163)
Lightweight tests are already included in fast tests since they're not marked as slow/e2e/integration. Running them separately caused duplication.
- Marked all tests that load Qwen3 model (~1.2GB) as @pytest.mark.slow - This excludes them from CI fast-tests job - Fast tests now: 127 tests in ~25s (was 163 tests in ~60s) - Slow memory tests: 36 tests (use heavy model) - Lightweight tests: 2 tests (use 80MB model, run in fast job) This prevents CI timeouts and keeps fast tests truly fast.
Temporarily running all tests on feature branches to validate before merging to main.
Use github.head_ref for pull request events to properly detect e/* branches.
- Use subprocess.call() to stream output directly to console - Add --livestream flag for real-time terminal-bench output - Simplify assertions to just check return code - Remove output parsing (terminal-bench handles success/failure) - This provides much better observability during long test runs
Terminal-bench tests need local debugging. E2E tests remain enabled. Fast tests (127 tests): ✅ Running E2E tests (1 test): ✅ Running Integration tests: ⏸️ Disabled for now
Only run on: - Pull requests to main - Direct pushes to main This prevents duplicate runs (once on push, once on PR).
Co-authored-by: edgarpavlovsky <edgarpavlovsky@gmail.com>
Co-authored-by: edgarpavlovsky <edgarpavlovsky@gmail.com>
…ests-in-ci-58aa
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces new documentation and configuration files, restructures the project layout for clarity, and adds support for terminal-bench integration. The main themes are enforcing consistent development rules (especially around Python version and dependency management), improving project organization, and providing detailed instructions for benchmarking and integration.
Development rules and documentation:
.ai-rules.md,.claude.md,.cursorrules, andWARP.mdto standardize AI assistant usage, enforce Python 3.12+ requirement, and mandate the use ofuvfor dependency management across different environments and tools. [1] [2] [3]README.mdto clarify runtime state directory usage, configuration file location, and reorganized the project structure to use asrc/directory for source code and a separatestate/directory for runtime data. [1] [2] [3]Terminal-bench benchmarking integration:
benchmark/README.mdandbenchmark/USAGE.mdwith comprehensive instructions for installing, running, and troubleshooting Fireteam as a terminal-bench agent, including real-time logging, state isolation, and advanced usage tips. [1] [2]benchmark/__init__.pyandbenchmark/adapters/__init__.py, establishing a clear entry point and import structure for terminal-bench integration. [1] [2]Configuration and environment updates:
ANTHROPIC_API_KEYto.env.exampleto support Anthropic API integration for Claude agents.Codebase cleanup:
agents/base.pyfile, likely in favor of the new structure undersrc/agents/.These changes collectively improve developer onboarding, enforce best practices, and enable robust benchmarking and integration workflows for Fireteam.