docs: Add comprehensive documentation website and benchmark results#17
Open
WiseMrMusa wants to merge 112 commits intomasterfrom
Open
docs: Add comprehensive documentation website and benchmark results#17WiseMrMusa wants to merge 112 commits intomasterfrom
WiseMrMusa wants to merge 112 commits intomasterfrom
Conversation
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Co-authored-by: Han <tinghan0110@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
…ion 4bac87e for ere packages in Cargo.toml and Cargo.lock
- Introduced Cluster variant in Resource enum for SP1 cluster resources. - Updated ProverResourceType conversion to handle Cluster using default ClusterProverConfig.
- Introduced a new `docker-compose.yml` file for managing SP1 Cluster services including Redis, PostgreSQL, API, Coordinator, CPU, and GPU nodes. - Added `start-sp1-cluster.sh` script for starting the cluster with options for GPU nodes and mixed worker modes. - Created `stop-sp1-cluster.sh` script for stopping the cluster and managing persistent data and images. - Enhanced user experience with logging and help messages in the scripts.
- Added validation in the CLI to ensure that the Cluster resource can only be used with SP1 zkVMs. - Introduced `bail` from `anyhow` for error handling in case of invalid resource configurations.
Adds a new crate that provides pure EVM transaction execution without pre-execution validation or post-execution consensus checks. Key components: - stateless_execution_with_trie: Core execution function returning bool - WitnessDatabase: EVM database backed by StatelessTrie witness data - RethStatelessExecutorGuest: Guest implementation for zkVMs - SP1 and RISC0 entry points for zkVM compilation This enables accurate benchmarking of raw EVM execution cycles in zkVMs by skipping all validation overhead.
Updates both root and ere-guests workspace Cargo.toml to include: - reth-stateless-executor as a workspace member - Required reth dependencies (reth-evm, reth-revm) - Dependency declarations for the new crate
Adds stateless_executor module to benchmark-runner for execution-only benchmarking support: - stateless_executor/reth.rs: Input preparation for Reth executor - Updated lib.rs to export the new module - Made BlockMetadata.block_used_gas public for reuse
…-executor Adds zkVM-specific entry points for all supported platforms: - Airbender: With custom allocator and runtime - OpenVM: With crypto provider installation - Pico: With KZG proof verification crypto provider - ZisK: Standard entry point These mirror the structure of stateless-validator entry points.
Introduces a new stateless_executor module with the following features: - Added stateless_executor.rs for handling execution client variants and input preparation. - Updated reth.rs to utilize StatelessExecutorFixture for input processing. - Enhanced read_benchmark_fixtures_folder to read and parse benchmark fixtures. This lays the groundwork for execution-only benchmarking of stateless programs.
…cutor and validator to allow both input_file and input_folder This commit introduces the ability to specify either an input folder or a single input file for the stateless executor and validator. The `stateless_executor_inputs` and `stateless_validator_inputs` functions are updated to support this new functionality, along with corresponding changes in the CLI to accept an optional input file argument. This improves flexibility in benchmarking by allowing users to work with individual benchmark fixture files directly.
…r stateless executor This commit modifies the CLI to set a default value for the execution client argument, allowing for easier configuration. Additionally, the guest relative path for the Reth execution client is simplified by removing the specific subdirectory, enhancing the path resolution for the stateless executor.
This commit introduces a new `MemoryTracker` struct to monitor memory usage during the proving process. It tracks initial, peak, and average memory usage, and integrates memory sampling into the benchmark runner. The proving metrics now include memory usage statistics, enhancing performance analysis capabilities.
…d runner.rs This commit introduces optional memory tracking capabilities in the benchmark runner. The `Cargo.toml` is updated to include new features for memory tracking, and the `runner.rs` file is modified to conditionally compile memory tracking logic based on the feature flag. This enhances the ability to monitor memory usage during the proving process, aligning with previous enhancements to performance analysis.
…chmark CLI This commit introduces a new `Network` resource type in the CLI, allowing users to specify network proving. It also adds validation to ensure that network proving is only used with SP1, enhancing the robustness of the command line interface for the zkVM benchmarker. The changes improve user experience by preventing misconfigurations related to resource selection.
… in reth.rs This commit updates the input handling in the `stateless_executor_inputs_from_fixtures` function to pass a reference to the benchmark wrapper instead of moving it. This change ensures that the benchmark wrapper can be reused, improving memory efficiency and preventing potential ownership issues.
This commit cleans up the `runner.rs` file by removing the unused `PublicValues` import from the `ere_zkvm_interface` module. This enhances code clarity and maintains a cleaner codebase.
…nd version 4bac87e for ere packages in Cargo.toml and Cargo.lock
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - uses: pnpm/action-setup@v4 |
Check warning
Code scanning / CodeQL
Unpinned tag for a non-immutable Action in workflow Medium
| run: ls -R artifacts | ||
|
|
||
| - name: Release | ||
| uses: softprops/action-gh-release@v2 |
Check warning
Code scanning / CodeQL
Unpinned tag for a non-immutable Action in workflow Medium
Regenerate lockfile to resolve Git merge conflict markers that were preventing the project from building. The conflicts were between different versions of ere dependencies.
… commit hashes This commit modifies the Rust toolchain version in multiple workflow files to use specific commit hashes instead of the stable tag. This change ensures consistency in the toolchain used across different CI jobs, enhancing build reliability.
…e files This commit refactors several files by reorganizing and cleaning up import statements for better readability and consistency. Additionally, it enhances code formatting in the `runner.rs`, `stateless_executor.rs`, `stateless_validator.rs`, and other related files, ensuring a more maintainable codebase. These changes do not alter functionality but improve overall code clarity.
…ional numbers This commit modifies the regex pattern in `compare_proving_times.py` to allow for gas values as rational numbers (e.g., 0.1M, 0.2M). Additionally, the documentation within the script has been updated to reflect this change, ensuring clarity for users regarding the expected format of benchmark directory names.
This commit introduces new documentation files for benchmark results across various gas categories (0.1M, 0.2M, 0.4M, 0.6M, and 0.8M) for both risc0 and sp1 configurations. Each category includes detailed proving metrics, comparisons, and performance statistics, enhancing the overall documentation for users analyzing zkVM performance.
This commit introduces new documentation files for marginal gas benchmark reports for both SP1 and RISC0 configurations, detailing performance metrics and regression results. Additionally, the configuration file has been updated to include a new section in the sidebar for easy navigation to these reports.
…P1 and RISC0 This commit corrects the image paths in the documentation for the marginal gas benchmark reports of SP1 and RISC0, ensuring that all visual data is properly linked to the new directory structure. This change enhances the accessibility of performance metrics and regression results for users.
…chmark reports This commit enhances the configuration for the marginal gas benchmark reports by adding a proxy setup for local development. Additionally, it updates the image paths in the documentation for SP1 and RISC0 to reflect the new directory structure, ensuring that all visual data is correctly linked and accessible for users.
…development This commit introduces a new Vite plugin that rewrites requests for public assets from the basePath to the root, enhancing the development experience by simplifying asset access. The previous proxy configuration has been removed in favor of this more streamlined approach.
…arginal gas benchmark documentation
Add stateless-executor as a documented guest program type alongside stateless-validator, empty-program, and block-encoding-length. Include details about: - Purpose: pure EVM execution benchmarking without validation overhead - Input: same BlockAndWitness fixtures as stateless-validator - Execution client: Reth only (ethrex not supported) - Key difference: skips pre/post validation for raw execution cycles
Create new documentation page explaining the internal architecture of the stateless-executor guest program. Includes: - Module breakdown (execution.rs, guest.rs, witness_db.rs) - Technical explanation of what validation steps are skipped - Execution flow diagram comparing executor vs validator - Relevant code references from the implementation
Create comparison page to help users choose between stateless-executor and stateless-validator guest programs. Includes: - Feature comparison table (validation steps, execution clients, etc.) - Decision flowchart for selecting the right program - Trade-offs: accuracy vs pure execution cycle measurement - Recommended use cases for each
Explicitly document that stateless-executor only supports Reth execution client, unlike stateless-validator which supports both Reth and Ethrex. Updates: - stateless-executor-guide.mdx: add limitation note in prerequisites - gas-categorized-benchmarks.mdx: clarify executor vs validator clients
Create API reference page documenting the public types and interfaces of the stateless-executor guest program. Documents: - RethStatelessExecutorInput: stateless_input and public_keys fields - RethStatelessExecutorOutput: boolean success/failure - StatelessExecutionError: all error variants with explanations - Example JSON input structure
Update the metrics JSON structure example to accurately reflect stateless-executor output, which skips validation. Changes: - Remove validation-specific region cycles from example - Add "execution" cycle scope as the primary metric - Add explanatory note about executor-specific metrics - Clarify what regions are measured vs skipped
Document the StatelessExecutorFixture structure and how it relates to StatelessValidationFixture used by the validator. Includes: - Fixture JSON structure breakdown - Witness data requirements - Fixture compatibility notes between programs - Example fixture with annotations
Remove references to temporary development branches (master-se-docs-03) now that documentation is merged. Updates getting-started.mdx and stateless-executor-guide.mdx to use the default branch.
Add comprehensive guidance on when to use stateless-executor vs other guest programs. Includes: - Ideal use cases: pure EVM cycle measurement, opcode profiling - Trade-offs: no validation overhead vs less realistic workload - Benchmarking implications and accuracy considerations - Decision criteria for choosing executor vs validator
Document zkVM support and any platform-specific behaviors for the stateless-executor guest program. Includes: - Supported zkVMs: SP1, RISC0, OpenVM, Pico, Zisk, Airbender - Compatibility matrix for executor across platforms - Any zkVM-specific limitations or notes - Performance characteristics guidance
Add comprehensive documentation explaining the architectural decisions: GPU Worker Isolation (One GPU per Worker): - Memory Isolation: Dedicated GPU memory prevents OOM errors from affecting others - Fault Tolerance: Worker crashes don't impact other workers - Predictable Performance: No resource contention ensures consistent proving times - Simplified Debugging: Issues traceable to specific GPU/worker pairs GPU Worker Limit (Maximum 8): - Typical Server Configurations: Most GPU servers have 4-8 GPUs - Resource Management: Each worker requires significant CPU and memory - Coordinator Overhead: More workers increases coordination complexity - Recommendation: Run multiple clusters for larger deployments Also add helpful hint when user exceeds GPU limit.
Create new documentation page (www/docs/pages/sp1-cluster.mdx) covering: Quick Start: - Basic commands to get started quickly - Prerequisites verification commands Prerequisites: - Required software and versions - Hardware requirements - NVIDIA Container Toolkit installation guide Configuration: - Environment variables reference - Resource limits customization - Network port configuration - Example .env file Usage: - Start/stop commands with all options - Command line options reference - Viewing logs and status Architecture: - Services overview with descriptions - Data flow diagram - GPU worker design rationale Advanced Topics: - Running multiple clusters on same machine - High availability considerations - Production deployment recommendations Troubleshooting: - Common issues and solutions - Diagnostic commands - Reset procedures Integration: - Using with benchmark scripts - FAQ section Also update vocs.config.ts to add sidebar link for SP1 Cluster Setup.
Remove documentation pages and references for scripts that don't exist: - run-single-file-benchmark.sh - generate_markdown_tables.py - generate_results.sh - export_comparison_csv.py - compare_sp1_risc0.py - simplify_test_names.py - update-docs-with-results.sh Deleted pages: - single-file-benchmark.mdx - export-comparison-csv.mdx - compare-sp1-risc0.mdx - markdown-tables.mdx - simplified-naming.mdx Updated files: - vocs.config.ts (removed sidebar entries) - scripts.mdx (removed sections for non-existent scripts) - download-fixtures.mdx (removed workflow reference) - gas-categorized-fixtures.mdx (removed workflow reference) - gas-categorized-benchmarks.mdx (removed analysis integration references) - benchmark-results.mdx (removed script references)
Eliminate obsolete patch entries for sha2, sha3, k256, and bn from the Cargo.toml file, streamlining the dependency management.
Add 'nm/03-docs' to the list of branches that trigger the deploy-docs workflow, enhancing documentation deployment flexibility.
Remove references to non-existent scripts and streamline the documentation by eliminating outdated links, including those for 'Compare SP1 vs RISC0' and 'Markdown Tables'. This enhances clarity and focuses on relevant content for users.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR builds on
nm/02-scriptsto add a complete documentation website using Vocs, comprehensive benchmark results for multiple zkVMs, and detailed guides for the stateless-executor guest program.Detailed Changes
1. Documentation Website (
www/)Vocs Framework Setup
vocs.config.tspackage.jsontsconfig.jsonpnpm-lock.yamlConfiguration Highlights
2. Documentation Pages (
www/docs/pages/)Core Guides
getting-started.mdxstateless-executor-guide.mdxstateless-executor-architecture.mdxstateless-executor-api.mdxexecutor-vs-validator.mdxFixture & Benchmark Guides
download-fixtures.mdxgas-categorized-fixtures.mdxgas-categorized-benchmarks.mdxscripts.mdxbenchmark-results.mdxSP1 Cluster Documentation
sp1-cluster.mdxTopics covered:
3. Benchmark Results
Marginal Gas Benchmarks
marginal-gas-benchmark/sp1.mdmarginal-gas-benchmark/risc0.mdmarginal-gas-benchmark/index.mdxResults include:
Gas-Categorized Results
Comprehensive results for multiple configurations:
Each result file includes:
4. Visualization Assets (
www/docs/public/)Chart Images (645 PNG files)
bar_time_per_gas.pngbar_cycles_per_gas.pngscatter_*.pngcomparison_*.pngCSV Data Files
sp1_regression_results.csvrisc0_regression_results.csv5. GitHub Actions CI/CD
.github/workflows/(Documentation Deployment)Added workflow for deploying documentation to GitHub Pages:
6. Analysis Scripts
scripts/compare_proving_times.pyCompare proving times across zkVMs:
scripts/convert-metrics-to-markdown.pyConvert JSON metrics to markdown tables:
scripts/test_name_formatter.pyHuman-readable test name formatting:
opcodes_PUSH1_100kPUSH1 (100K ops)precompile_ecrecover_5ecrecover (5 calls)7. Documentation Improvements
README.md Updates
Content Enhancements
Breaking Changes
None. This PR adds documentation without modifying code.