Large refactor of the code base with support for new training models#233

Open

wvaske wants to merge 207 commits intomlcommons:TF_Modernizationfrom

Commits on Nov 21, 2025

Add initial KV Cache benchmark implementation for MLPerf Storage v3
hazemawadalla
committed

Commits on Dec 9, 2025

feat: Replace legacy spillover logic with Waterfall LRU architecture
hazemawadalla
committed

Commits on Jan 10, 2026

Add storage throughput metric, ShareGPT integration, LMCache validation suite, and production tooling for MLPerf Storage v3
hazemawadalla
committed

Commits on Jan 12, 2026

Refactor BenchmarkRun to use data contract pattern
claude
committed

Commits on Jan 13, 2026

Commits on Jan 15, 2026

Commits on Jan 23, 2026

Add fail-fast dependency validation and fix initialization bugs

Wes Vaske
and
claude
committed
Fix unhashable RunID bug in report generator

wvaske
and
claude
committed
Update CLAUDE.md with test environment directories and examples

wvaske
and
claude
committed
docs: map existing codebase structure

wvaske
and
claude
committed
docs: initialize project

wvaske
and
claude
committed
chore: add project config

wvaske
and
claude
committed
docs: define v3.0 requirements

wvaske
and
claude
committed
docs: create roadmap (10 phases)

wvaske
and
claude
committed
docs(phase-1): research package management foundation
wvaske
committed
docs(01): create phase plan for Package Management Foundation

wvaske
and
claude
committed
docs: revise Plan 01-05 to address benchmark integration gap

wvaske
and
claude
committed
feat(01-02): add uv CPU-only index configuration
wvaske
committed
feat(01-02): add packaging dependency for version parsing
wvaske
committed
feat(01-01): create lockfile module structure with data models
wvaske
committed
docs(01-02): complete CPU-only PyTorch configuration plan
wvaske
committed
docs(01-01): complete lockfile module foundation plan
wvaske
committed
feat(01-03): implement lockfile generator module
wvaske
committed
feat(01-04): create lockfile validator module
wvaske
committed
feat(01-03): update lockfile module exports for generator
wvaske
committed
docs(01-03): complete lockfile generator plan
wvaske
committed
docs(01-04): complete runtime version validation plan
wvaske
committed
feat(01-05): add lockfile CLI argument builder
wvaske
committed
feat(01-05): integrate lockfile subcommand into CLI
wvaske
committed
feat(01-05): add lockfile command handler to main
wvaske
committed
feat(01-05): add --verify-lockfile flag to benchmark commands
wvaske
committed
docs(01-05): complete CLI integration plan - Phase 1 complete
wvaske
committed
docs: mark Phase 1 complete in roadmap

wvaske
and
claude
committed

Commits on Jan 24, 2026

docs(phase-2): research environment validation and fail-fast patterns

wvaske
and
claude
committed
docs(02): create phase plan for Environment Validation and Fail-Fast

wvaske
and
claude
committed
fix(02): revise plans based on checker feedback

wvaske
and
claude
committed
feat(02-01): add environment module with OS detection and install hints
wvaske
committed
test(02-01): add unit tests for environment module
wvaske
committed
docs(02-01): complete environment detection plan
wvaske
committed
feat(02-03): add ValidationIssue dataclass and SSH validators
wvaske
committed
feat(02-02): Add OS-aware dependency checking with install hints

wvaske
and
claude
committed
test(02-03): add comprehensive tests for validators
wvaske
committed
docs(02-02): complete Executable Checking Module plan

wvaske
and
claude
committed
docs(02-03): complete SSH validation and issue collection plan
wvaske
committed
feat(02-04): add comprehensive fail-fast environment validator
wvaske
committed
test(02-04): add comprehensive tests for validate_benchmark_environment
wvaske
committed
docs(02-04): complete pre-run validation orchestration plan
wvaske
committed
feat(02-05): integrate fail-fast validation into main.py

wvaske
and
claude
committed
feat(02-05): add _validate_environment hook to Benchmark base class

wvaske
and
claude
committed
test(02-05): add tests for benchmark validation integration

wvaske
and
claude
committed
docs(02-05): complete fail-fast validation integration plan

wvaske
and
claude
committed
docs(02): complete environment validation and fail-fast phase
wvaske
committed
docs(03): research KV Cache benchmark integration phase

wvaske
and
claude
committed
docs(03): create phase plan for KV Cache benchmark integration

wvaske
and
claude
committed
feat(03-01): add distributed execution arguments to KV cache CLI

wvaske
and
claude
committed
test(03-01): add unit tests for KV cache CLI arguments

wvaske
and
claude
committed
docs(03-01): complete KV cache distributed CLI plan

wvaske
and
claude
committed
feat(03-02): add MPI execution support to KVCacheBenchmark

wvaske
and
claude
committed
test(03-02): add unit tests for KVCacheBenchmark MPI execution

wvaske
and
claude
committed
docs(03-02): complete MPI execution support plan

wvaske
and
claude
committed
feat(03-03): enhance KV cache metadata for history integration
wvaske
committed
test(03-03): add metadata tests for history integration
wvaske
committed
docs(03-03): complete metadata and history integration plan
wvaske
committed
docs(03): complete KV Cache Benchmark Integration phase

wvaske
and
claude
committed
docs(04): research VectorDB benchmark integration phase

wvaske
and
claude
committed
docs(04): create phase plan for VectorDB benchmark integration

wvaske
and
claude
committed
fix(04): revise 04-02 plan based on checker feedback

wvaske
and
claude
committed
feat(04-01): rename vectordb run-search to run for CLI consistency

wvaske
and
claude
committed
feat(04-02): add metadata property to VectorDBBenchmark
wvaske
committed
feat(04-02): add write_metadata calls to execute_run and execute_datagen
wvaske
committed
test(04-01): update vectordb CLI tests for run command rename

wvaske
and
claude
committed
docs(04-01): complete VectorDB CLI rename plan

wvaske
and
claude
committed
docs(04-02): complete VectorDB metadata integration plan
wvaske
committed
test(04-03): add VectorDB CLI argument parsing tests
wvaske
committed
test(04-03): add VectorDB benchmark class tests
wvaske
committed
docs(04-03): complete VectorDB verification and integration plan
wvaske
committed
docs(04): complete VectorDB Benchmark Integration phase

wvaske
and
claude
committed
docs(05): research phase domain
wvaske
committed
docs(05): create phase plan for benchmark validation pipeline

wvaske
and
claude
committed
feat(05-01): add VectorDBRunRulesChecker class
wvaske
committed
feat(05-01): export VectorDBRunRulesChecker from run_checkers
wvaske
committed
feat(05-01): export VectorDBRunRulesChecker from rules package
wvaske
committed
feat(05-02): route kv_cache and vector_database in BenchmarkVerifier
wvaske
committed
feat(05-02): add VECTORDB_REQUIREMENTS to ClosedRequirementsFormatter
wvaske
committed
docs(05-01): complete VectorDBRunRulesChecker plan
wvaske
committed
docs(05-02): complete BenchmarkVerifier routing plan
wvaske
committed
test(05-03): add VectorDBRunRulesChecker unit tests
wvaske
committed
docs(05-03): complete VectorDB rules checker tests plan
wvaske
committed
docs(05): complete Benchmark Validation Pipeline Integration phase

wvaske
and
claude
committed
docs(06): research SSH-based host collection phase

wvaske
and
claude
committed
docs(06): create phase plan for SSH-Based Host Collection

wvaske
and
claude
committed
feat(06-01): add MountInfo, CgroupInfo dataclasses and /proc parsers
wvaske
committed
feat(06-01): update collect_local_system_info with vmstat, mounts, cgroups
wvaske
committed
test(06-01): add unit tests for cluster_collector parsers
wvaske
committed
docs(06-01): complete /proc parsers plan
wvaske
committed
feat(06-02): implement SSHClusterCollector class

wvaske
and
claude
committed
test(06-02): add unit tests for SSHClusterCollector

wvaske
and
claude
committed
docs(06-02): complete SSHClusterCollector implementation plan

wvaske
and
claude
committed
feat(06-03): add ClusterSnapshots dataclass for start/end collection
wvaske
committed
feat(06-03): integrate SSH collection into benchmark base class
wvaske
committed
test(06-03): add tests for collection method selection and snapshots
wvaske
committed
docs(06-03): complete benchmark base integration plan
wvaske
committed
docs(06): complete SSH-Based Host Collection phase

wvaske
and
claude
committed
docs(07): research time-series host data collection phase

wvaske
and
claude
committed
docs(07): create phase plan

wvaske
and
claude
committed
fix(07): revise plans based on checker feedback

wvaske
and
claude
committed
feat(07-01): add TimeSeriesSample and TimeSeriesData dataclasses
wvaske
committed
feat(07-01): add collect_timeseries_sample and TimeSeriesCollector
wvaske
committed
test(07-01): add unit tests for time-series collection
wvaske
committed
docs(07-01): complete core time-series infrastructure plan
wvaske
committed
feat(07-02): add MultiHostTimeSeriesCollector for parallel multi-host collection
wvaske
committed
test(07-02): add unit tests for MultiHostTimeSeriesCollector
wvaske
committed
docs(07-02): complete multi-host time-series collection plan
wvaske
committed
feat(07-03): add time-series CLI arguments to all benchmark parsers

wvaske
and
claude
committed
feat(07-03): integrate time-series collection into Benchmark base class

wvaske
and
claude
committed
test(07-03): add unit tests for time-series benchmark integration

wvaske
and
claude
committed
docs(07-03): complete benchmark time-series integration plan

wvaske
and
claude
committed
docs(07): complete Time-Series Host Data Collection phase

wvaske
and
claude
committed
docs(08): research phase domain for new training models
wvaske
committed
docs(08): create phase plan for new training models

wvaske
and
claude
committed
fix(08): revise plans based on checker feedback

wvaske
and
claude
committed
feat(08-01): add DLRM, RETINANET, FLUX model constants
wvaske
committed
feat(08-01): add DLRM workload configurations
wvaske
committed
feat(08-01): add RetinaNet workload configurations
wvaske
committed
feat(08-01): add Flux workload configurations
wvaske
committed
docs(08-01): complete new training model configurations plan
wvaske
committed
feat(08-02): add model validation rules for new training models
wvaske
committed
test(08-02): add unit tests for new training model validation
wvaske
committed
docs(08-02): complete validation rules for new training models plan
wvaske
committed
docs(08): complete New Training Models phase

wvaske
and
claude
committed

Commits on Jan 25, 2026

docs(09): research DLIO parquet format support

wvaske
and
claude
committed
docs(09): create phase plan for DLIO parquet support

wvaske
and
claude
committed
fix(09): revise plans based on checker feedback

wvaske
and
claude
committed
docs(09-01): complete DLIO parquet format implementation plan

wvaske
and
claude
committed
feat(09-02): add DLRM parquet configuration files

wvaske
and
claude
committed
test(09-02): add unit tests for parquet format validation

wvaske
and
claude
committed
docs(09-02): complete parquet workload configuration plan

wvaske
and
claude
committed
docs(09): complete DLIO Parquet Support phase

wvaske
and
claude
committed
docs(10): research phase domain for progress indication

wvaske
and
claude
committed
docs(10): create phase plan

wvaske
and
claude
committed
fix(10): revise plans based on checker feedback

wvaske
and
claude
committed
feat(10-01): add Rich dependency and create progress module
wvaske
committed
test(10-01): add unit tests for progress module
wvaske
committed
docs(10-01): complete progress indication foundation plan
wvaske
committed
feat(10-02): add stage indicators to benchmark run() method
wvaske
committed
feat(10-02): add spinners to cluster collection methods
wvaske
committed
feat(10-03): add progress indication to main.py validation operations

wvaske
and
claude
committed
test(10-02): add unit tests for progress integration in base.py
wvaske
committed
docs(10-02): complete benchmark progress integration plan
wvaske
committed
docs(10-03): complete main.py progress integration plan
wvaske
committed
docs(10): complete Progress Indication phase
wvaske
committed

Commits on Feb 2, 2026

docs(11): research phase domain

wvaske
and
claude
committed
docs(11): create phase plan for comprehensive parquet support

wvaske
and
claude
committed
feat(11-03): update DLIO dependency to wvaske fork with parquet support

wvaske
and
claude
committed
docs(11-03): complete DLIO fork dependency plan

wvaske
and
claude
committed
docs(11-01): complete parquet config and enum extensions plan

wvaske
and
claude
committed
docs(11-02): complete Parquet Reader/Generator Rewrite plan

wvaske
and
claude
committed
docs(11): complete Comprehensive Parquet Support phase

wvaske
and
claude
committed
chore: add UAT docs, phase context, and uv lockfile

wvaske
and
claude
committed

Commits on Feb 3, 2026

feat: update DLRM parquet configs with schema-driven columns

wvaske
and
claude
committed

Commits on Feb 4, 2026

chore: update test params and fix file permissions

wvaske
and
claude
committed

Commits on Feb 9, 2026

allow claude to bypass cla check (#234)
BarnacleBob
authored

Commits on Feb 15, 2026

Merge pull request #238 from mlcommons/FileSystemGuy-KVCache-revert
FileSystemGuy
authored

Commits on Feb 17, 2026

Merge pull request #239 from mlcommons/FileSystemGuy-claudebot
FileSystemGuy
authored

Commits on Feb 18, 2026