Skip to content

Conversation

@pinges
Copy link
Contributor

@pinges pinges commented Nov 28, 2025

This pull request introduces several improvements and fixes related to snap sync.

Snap sync is using two pipelines now.

  • The first pipeline downloads the headers. The download is starting from a trusted pivot block. For mainnet that trusted pivot block is a safe block that was sent by the CL (FCU). Headers are downloaded backwards from that trusted block and each block can be validated using the parent block hash contained in the previous header. The last header downloaded is block 1, which is validated by the parent block hash of block 2, and in addition validated against the known genesis block by checking whether the parent block hash matches the block hash of the genesis block (block 0).
  • All headers are written to the database and are fully trusted at the end of the header download.
  • The second pipeline now uses the trusted headers to download the bodies and receipts. This download starts from the genesis block or a known checkpoint. Mainnet blocks and receipts by default are downloaded starting at the first block of the Paris hard fork. The first step of the second pipeline retrieves the headers from the database. Blocks and receipts are requested from peers and are validated against the block hash or the receipts root contained in the header.
  • The storing of the bodies and receipts was optimized by doing less validation as these are validated against fully trusted headers now.

Test and Configuration Enhancements:

  • Fixed BFT sync acceptance tests. BFT SNAP and CHECKPOINT sync tests now use snap.

Build and Miscellaneous Fixes:

  • Fixed test resource generation in the Gradle build script to only generate the test blockchain if the binary does not already exist.

Syncing Times compared to latest
*Chain download is 2-3 hours faster than latest, but as state download, heal is slower this does not make a big difference in total sync time.

image

Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
@github-project-automation github-project-automation bot moved this to Backlog in RC 25.12.0 Dec 4, 2025
pinges and others added 27 commits December 15, 2025 10:25
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
…9295)

Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
* Update engine API

Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com>

* Add EngineGetPayloadV6Test

Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com>

* Revert changes to EngineForkchoiceUpdatedV3

Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com>

* Add EngineNewPayloadV5Test

Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com>

* Address code review comments

Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com>

* Fix failing test build

Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com>

* Fix failing test build

Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com>

---------

Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com>
Co-authored-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
* ADD: UInt256 implementation of evm's op MOD

Signed-off-by: Thomas Zamojski <thomas.zamojski@quadratic-labs.com>
Signed-off-by: Nikos Baxevanis <nikos.baxevanis@gmail.com>
Co-authored-by: Nikos Baxevanis <nikos.baxevanis@gmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Should fix support for running builds `./gradlew build` for besu when using a git-worktree setup. Previously there was a problem while parsing the besu version during the build since gradle was confused as to where to lookup for the commit hash.

Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
* add evm options to evmtool too
* Statically initiatise SignatureAlgorithm default instance for reference tests

---------

Signed-off-by: Luis Pinto <luis.pinto@consensys.net>
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Co-authored-by: Simon Dudley <simon.dudley@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
* add block timestamp to logs within transaction receipt and block receipts

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

---------

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
…9304)

Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
* Bump spec tests

Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net>

* changelog

Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net>

---------

Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: Usman Saleem <usman@usmans.info>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
* Add test summary reporting to block-test command

The block-test command now tracks and displays a summary of test
execution results, including total tests, passed/failed counts, and
a list of failed tests with their failure reasons. This improves
usability when running large test suites by providing a clear
overview of test outcomes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Bhargava Shastry <bhargava.shastry@ethereum.org>

* Refactor block-test summary output for better maintainability

Address code review feedback by:
- Extract separator string into SEPARATOR constant to reduce duplication
- Reuse failureReason variable in output statements instead of duplicating
  formatting logic

This improves code maintainability by consolidating the failure message
formatting in a single location.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Bhargava Shastry <bhargava.shastry@ethereum.org>

* Address review feedback: simplify TestResults implementation

- Remove unused testName parameter from recordPass() method
- Remove totalTests field and calculate it dynamically in printSummary()
  from passed + failed counts
- Simplify SEPARATOR constant by removing embedded newline and use
  explicit out.println() for consistent formatting
- Add hasTests() helper method to check if any tests were executed

This reduces redundant state tracking and improves code clarity.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Bhargava Shastry <bhargava.shastry@ethereum.org>

---------

Signed-off-by: Bhargava Shastry <bhargava.shastry@ethereum.org>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Usman Saleem <usman@usmans.info>
Co-authored-by: Simon Dudley <simon.dudley@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
* Implement era1 file export

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Clean up OutputStreamFactory to pass inspections

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* fix javadoc

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* fix javadoc

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* fix more compile checks

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* fix more compile checks

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* fix more compile checks

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Output era1 files to supplied directory

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Implement accumulator and block index

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Fix compile errors

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Fix javadoc

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Couple of fixes, add logging

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Implement our own merkleizer, try many things

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Fix accumulator calculations

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Fix block number to file number calculation

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Correctly use accumulator hash in era1 filename

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Update manual test to use nimbus.team source for expected era1 file

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Add javadoc to new Era1* files

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Fix BlocksSubCommandTest

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Add MerkleizerTest

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Refactor to produce Era1FileWriter

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Add unit test for Era1FileWriter, fix discovered bug in Era1FileWriter

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Refactor to produce Era1Accumulator and add unit test

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Refactor to produce Era1BlockIndexConverter and add unit test

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Unit test Era1BlockExporter

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Fix javadoc issues

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Rename difficultysForFile to difficultiesForFile

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Move network selection outside of loop

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Make blocks export option non-hidden, add changelog entry

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Use network option in era1 block exporter

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Change Era1BlockExporter.export arguments to start/end block, and handle file number conversion internally

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Add additional Merkleizer tests and guard against empty leaves

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Fix BlocksSubCommandTest

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Move new rlp encoder classes into encoding package, add encodeWrapped for BlockBodyEncoder, add TransactionReceiptEncodingConfiguration parameter to TransactionReceiptEncoder.encode

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

---------

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>
Signed-off-by: Matilda-Clerke <matilda.clerke@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
…yperledger#9320)

This reverts commit 42306f9.

Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
…yperledger#9194)

Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
* remove parentBeaconBlockRoot from getPayload result to match the specs

Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
…9318)

* add burn-in tasks

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

---------

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
…ledger#9183)

Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>

Allows for enabling and disabling stack, storage, returnData, etc.. for these 2 specific RPC endpoints: debug_traceStandardBlockToFile and debug_traceStandardBadBlockToFile. It does not change any previous defaults for memory which is shown by default.

Example of usage:

curl --location --globoff 'http://localhost:8545' --data '{
    "jsonrpc": "2.0",
    "method": "debug_standardTraceBlockToFile",
    "params": [
        "0x2476b23a03dd87f272724a12213dc4a1ca8f13a1aa9d128a690e1b7d7d312e5e",
        {
            "txHash": "0xdfd5028da0f2f306ae71dc558c26bb1935b13ec3a0e909223eaa77bf761189a8",
            "disableStack": true,
            "disableMemory": false,
            "disableStorage": false
        }
    ],
    "id": 1
}'

curl --location --globoff 'http://localhost:8545/' --data '{
    "jsonrpc": "2.0",
    "method": "debug_standardTraceBadBlockToFile",
    "params": [
        "0x2476b23a03dd87f272724a12213dc4a1ca8f13a1aa9d128a690e1b7d7d312e5e",
        {
            "txHash": "0xdfd5028da0f2f306ae71dc558c26bb1935b13ec3a0e909223eaa77bf761189a8",
            "disableStack": true,
            "disableMemory": false,
            "disableStorage": false
        }
    ],
    "id": 1
}'

Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
…lock to be completed (hyperledger#9332)

Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Also remove sigleton pattern from ReferenceTestProtocolSchedules.

Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
* update netty

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

---------

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
…fer value is zero (hyperledger#9315)

Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
…ng with peer task system usage (hyperledger#9178)

* Remove GetBlockFromPeerTask and RetryingGetBlockFromPeersTask, replacing with peer task system usage

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Fix BackwardSyncContextTest

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Use preferred peer first for block retrieval in BlockPropagationManager

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Add new GetHeadersFromPeerTask constructor which doesn't take a block number and use it in SyncStepStep

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Refactor PeerTaskExecutor usage in BlockPropagationManager for better readability

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Fix a != that should have been ==

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Re-add AbstractBlockPropagationManagerTest.shouldRequestBlockFromOtherPeersIfFirstPeerFails and modify to use mock PeerTaskExecutor

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Re-add BackwardSyncContextTest.whenBlockNotFoundInPeers_shouldRemoveBlockFromQueueAndProgressInNextSession

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Fix BlocksSubCommandTest again, but differently

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Remove commented code

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

---------

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
fab-10 and others added 14 commits December 15, 2025 10:28
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
…tion (hyperledger#9516)

* Fix: Ensure deterministic sub-protocol registration in ConsensusSchedule

Signed-off-by: saeeddawod <saeed.dawod@gmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
feat: Implement callTracer for debug_trace* methods

Major refactoring of the debug tracing infrastructure to support the callTracer
type with Geth-compatible output.

Key changes:
- Refactor DebugCallTracerResult to support hierarchical call stack
- Introduce TracerType enum and TracerOptions simplification
- Rename StructLogTracerConfig to OpCodeTracerConfig for clarity
- Rename tracer result classes for consistency (OpCodeLoggerTracerResult, etc.)
- Implement CallTracerResultConverter for converting trace frames to call traces
- Add TraceFrame Builder pattern with precompile input/output fields
- Handle precompile calls with correct gas calculation and input/output
- Support SELFDESTRUCT, CREATE/CREATE2 operations in call traces
- Handle exceptional halts and soft failures with proper error descriptions
- Add feature flag for extra debug tracers
- Add comprehensive unit and integration tests for callTracer

---------

Signed-off-by: Usman Saleem <usman@usmans.info>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: Karim Taam <karim.t2am@gmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
)

* simple fix

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

---------

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
* Revert "accept client or server timeout for flaky timeout test (hyperledger#9552)"

This reverts commit 4e088d6.

    Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

* Revert "feat: Add RPC timeout interruption with CompletableFuture-based implementation (hyperledger#9117)"

This reverts commit 01bfb85.

    Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

---------

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
@pinges pinges marked this pull request as ready for review December 18, 2025 06:50
Copilot AI review requested due to automatic review settings December 18, 2025 06:50
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@pinges pinges requested a review from Copilot December 19, 2025 07:04
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

No open projects
Status: Open PRs

Development

Successfully merging this pull request may close these issues.