New feature: state pruning

[shawn-zil]

This PR introduces the state-pruning feature. It has been tested successfully on a local node in testnet by restoring from a recent checkpoint, enabling prune_interval and monitoring the database statistics.

As an example:

@25474000 - 2795227 nodes (602MB)
@25537402 - 2745840 nodes (535MB)

The nodes here refer to nodes in the Patricia-Merkle-Tree.

As a comparison, a testnet full node has a 6.5GB database storage, and it only grows in size as more and more data is added to the PMT, never reduced. The data is validated by checking account balances across time.

Details are described in docs/state_pruning.md documentation.

Tested on testnet under different scenarios:

1. Successful key migration observed for non-pruning node.
2. Successful key migration and pruning observed for pruning node.
3. Successful pruning observed for new pruning node.

[github-actions]

Bencher Report

Branch	shawn/3047/prune-state
Testbed	self-hosted

Click to view all benchmark results

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
full-blocks-erc20-transfers/full-blocks-erc20-transfers	📈 view plot 🚷 view threshold	614.29 ms (-7.43%) Baseline: 663.61 ms	796.35 ms (77.14%)
full-blocks-evm-transfers/full-blocks-evm-transfers	📈 view plot 🚷 view threshold	877.26 ms (-6.75%) Baseline: 940.75 ms	1,170.35 ms (74.96%)
full-blocks-giant-deploy/full-blocks-giant-deploy	📈 view plot 🚷 view threshold	4,954.30 ms (-4.51%) Baseline: 5,188.08 ms	6,052.46 ms (81.86%)
full-blocks-scilla-add/full-blocks-scilla-add	📈 view plot 🚷 view threshold	1,004.40 ms (-2.39%) Baseline: 1,029.04 ms	1,189.70 ms (84.42%)
full-blocks-scilla-load/full-blocks-scilla-load	📈 view plot 🚷 view threshold	5,319.00 ms (+3.74%) Baseline: 5,127.43 ms	6,463.96 ms (82.29%)
full-blocks-scilla-store/full-blocks-scilla-store	📈 view plot 🚷 view threshold	4,101.90 ms (-0.19%) Baseline: 4,109.71 ms	5,049.61 ms (81.23%)
full-blocks-zil-transfers/full-blocks-zil-transfers	📈 view plot 🚷 view threshold	892.93 ms (-9.46%) Baseline: 986.25 ms	1,432.22 ms (62.35%)
process-empty/process-empty	📈 view plot 🚷 view threshold	9.82 ms (-2.85%) Baseline: 10.11 ms	10.82 ms (90.77%)

🐰 View full continuous benchmarking report in Bencher

[shawn-zil]

Running this on devnet, we can see the difference in state nodes:

non-pruning API0 node:

Keys in range: 235915
Value size distribution: 
Count: 235915 Average: 346.9504  StdDev: 262.71
Min: 1  Median: 394.6768  Max: 22311
Percentiles: P50: 394.68 P75: 487.36 P99: 576.34 P99.9: 579.67 P99.99: 1361.27

pruning VAL0 (prune_interval=1_000) node:

Keys in range: 92323
Value size distribution: 
Count: 92323 Average: 123.4194  StdDev: 246.66
Min: 1  Median: 138.7620  Max: 22310
Percentiles: P50: 138.76 P75: 157.39 P99: 370.25 P99.9: 562.96 P99.99: 16460.18

This node holds just one snapshot of the state because the prune_interval < epoch.

pruning VAL2 (prune_interval=10_000) node:

Keys in range: 276979
Value size distribution: 
Count: 276979 Average: 123.4369  StdDev: 246.67
Min: 1  Median: 138.7644  Max: 22310
Percentiles: P50: 138.76 P75: 157.39 P99: 370.75 P99.9: 563.02 P99.99: 16459.88

This node holds three snapshots of the state because the prune_interval ~ 3 * epoch.

So, we should modify the code slightly to factor in the prune_interval before making a snapshot so that the node only holds one snapshot at a time, regardless of prune_interval

[shawn-zil]

Have encountered a missing trie node error - warrants more investigation/experimentation.

{"timestamp":"2026-02-23T03:43:41.644250Z","level":"INFO","fields":{"message":"handling block proposal 64fdc51f47589426e1a516d013ad12e921d1165bdad81923d853a2352aaaea73","block_view":591513,"block_number":591475,"txns":250},"target":"zilliqa::consensus","line_number":884}
{"timestamp":"2026-02-23T03:43:42.569194Z","level":"INFO","fields":{"message":"executing txn","hash":"c24bc511694dc70034512484d8074b2d04d5282c5f5bc3f4163467896605e861","from":"0xe45d5b4699fb41d45ef972126edfb3c81c5f71c2","to":"Some(0xb73194ed1924b6e439e29451c53cda8b52e824ba)","txn":"Legacy(TxLegacy { chain_id: Some(33469), nonce: 2300, gas_price: 4761904800000, gas_limit: 21000, to: Call(0xb73194ed1924b6e439e29451c53cda8b52e824ba), value: 10000000000000000, input: 0x })"},"target":"zilliqa::exec","line_number":850}
{"timestamp":"2026-02-23T03:43:42.570266Z","level":"INFO","fields":{"message":"executing txn","hash":"843bce822ddd9059de38243a52eeb62761725df4c8cc9a3838666fc5c27b0561","from":"0xe45d5b4699fb41d45ef972126edfb3c81c5f71c2","to":"Some(0x68399b383fcecfca829788ec40431644158c8b98)","txn":"Legacy(TxLegacy { chain_id: Some(33469), nonce: 2301, gas_price: 4761904800000, gas_limit: 21000, to: Call(0x68399b383fcecfca829788ec40431644158c8b98), value: 10000000000000000, input: 0x })"},"target":"zilliqa::exec","line_number":850}
{"timestamp":"2026-02-23T03:43:42.571563Z","level":"INFO","fields":{"message":"executing txn","hash":"762361dd72b1eb3c629c660a1c6567c93b913a588426cb5afff5af7249567b51","from":"0xe45d5b4699fb41d45ef972126edfb3c81c5f71c2","to":"Some(0xc5a4f3616b314bffcf909b2bb196d4abe4ef7b9e)","txn":"Legacy(TxLegacy { chain_id: Some(33469), nonce: 2302, gas_price: 4761904800000, gas_limit: 21000, to: Call(0xc5a4f3616b314bffcf909b2bb196d4abe4ef7b9e), value: 10000000000000000, input: 0x })"},"target":"zilliqa::exec","line_number":850}
{"timestamp":"2026-02-23T03:43:42.572811Z","level":"INFO","fields":{"message":"executing txn","hash":"38ee31c8095bbb5b350823b514c12d8a35cc595aea5296278ac72a45ecf49535","from":"0xe45d5b4699fb41d45ef972126edfb3c81c5f71c2","to":"Some(0x356e3f4e4415fbb4e5cdbdd3056967add3afb97f)","txn":"Legacy(TxLegacy { chain_id: Some(33469), nonce: 2303, gas_price: 4761904800000, gas_limit: 21000, to: Call(0x356e3f4e4415fbb4e5cdbdd3056967add3afb97f), value: 10000000000000000, input: 0x })"},"target":"zilliqa::exec","line_number":850}
{"timestamp":"2026-02-23T03:43:42.575270Z","level":"INFO","fields":{"message":"executing txn","hash":"858de9133c69db1e040d9012c42aca8ab3deb603bd62e07e19d804159982fda0","from":"0xe45d5b4699fb41d45ef972126edfb3c81c5f71c2","to":"Some(0xede048021f53a046644caccc05edd67c2ac19408)","txn":"Legacy(TxLegacy { chain_id: Some(33469), nonce: 2304, gas_price: 4761904800000, gas_limit: 21000, to: Call(0xede048021f53a046644caccc05edd67c2ac19408), value: 10000000000000000, input: 0x })"},"target":"zilliqa::exec","line_number":850}
{"timestamp":"2026-02-23T03:43:42.576423Z","level":"INFO","fields":{"message":"executing txn","hash":"ee257b58c7c90baae20863c8c01787131ccc9c06629a1ffd5561de3a36c6edc2","from":"0xe45d5b4699fb41d45ef972126edfb3c81c5f71c2","to":"Some(0xe39b98b712bc88bd69fe60b4daab3802c5b358a1)","txn":"Legacy(TxLegacy { chain_id: Some(33469), nonce: 2305, gas_price: 4761904800000, gas_limit: 21000, to: Call(0xe39b98b712bc88bd69fe60b4daab3802c5b358a1), value: 10000000000000000, input: 0x })"},"target":"zilliqa::exec","line_number":850}
{"timestamp":"2026-02-23T03:43:42.577289Z","level":"INFO","fields":{"message":"executing txn","hash":"f07b1a8fa7511225b35821c91e882f2367174594a4e10476dc006b06503971fa","from":"0xe45d5b4699fb41d45ef972126edfb3c81c5f71c2","to":"Some(0x926db1b304f29b3ee805f343bd8257606cc986c8)","txn":"Legacy(TxLegacy { chain_id: Some(33469), nonce: 2306, gas_price: 4761904800000, gas_limit: 21000, to: Call(0x926db1b304f29b3ee805f343bd8257606cc986c8), value: 10000000000000000, input: 0x })"},"target":"zilliqa::exec","line_number":850}
{"timestamp":"2026-02-23T03:43:42.578262Z","level":"INFO","fields":{"message":"executing txn","hash":"bdb7f0403c61ec9dc63307d6fa6755e36bd941aadce113790a69a39d5a2d66e3","from":"0xe45d5b4699fb41d45ef972126edfb3c81c5f71c2","to":"Some(0x0f7d9f099b0ad00684a93339be320989672dd3a4)","txn":"Legacy(TxLegacy { chain_id: Some(33469), nonce: 2307, gas_price: 4761904800000, gas_limit: 21000, to: Call(0x0f7d9f099b0ad00684a93339be320989672dd3a4), value: 10000000000000000, input: 0x })"},"target":"zilliqa::exec","line_number":850}
{"timestamp":"2026-02-23T03:43:42.579183Z","level":"INFO","fields":{"message":"executing txn","hash":"08bddc26037342d8f7011a5c587b3ef8369a329ac4b1a80b65ae4a68d2f7882e","from":"0xe45d5b4699fb41d45ef972126edfb3c81c5f71c2","to":"Some(0xfda99df1b6487f5a127396e21f11177d492482cc)","txn":"Legacy(TxLegacy { chain_id: Some(33469), nonce: 2308, gas_price: 4761904800000, gas_limit: 21000, to: Call(0xfda99df1b6487f5a127396e21f11177d492482cc), value: 10000000000000000, input: 0x })"},"target":"zilliqa::exec","line_number":850}
{"timestamp":"2026-02-23T03:43:42.580050Z","level":"INFO","fields":{"message":"executing txn","hash":"764b9cc0ca6fa51e86f2fa8cf60ff8f35959df8629338c00d7795df46c9e608c","from":"0xe45d5b4699fb41d45ef972126edfb3c81c5f71c2","to":"Some(0x4eb43bf7974932fb7785cfc16d85d368ecc50010)","txn":"Legacy(TxLegacy { chain_id: Some(33469), nonce: 2309, gas_price: 4761904800000, gas_limit: 21000, to: Call(0x4eb43bf7974932fb7785cfc16d85d368ecc50010), value: 10000000000000000, input: 0x })"},"target":"zilliqa::exec","line_number":850}
{"timestamp":"2026-02-23T03:43:42.580799Z","level":"INFO","fields":{"message":"executing txn","hash":"3a618ff2c2dd0d35f2f9409eef91201c10d68d23fcff96ceb9ed5e75e0a8bf12","from":"0xe45d5b4699fb41d45ef972126edfb3c81c5f71c2","to":"Some(0xd3e34eaff1879b4fa58ace38e528450d4094cdac)","txn":"Legacy(TxLegacy { chain_id: Some(33469), nonce: 2310, gas_price: 4761904800000, gas_limit: 21000, to: Call(0xd3e34eaff1879b4fa58ace38e528450d4094cdac), value: 10000000000000000, input: 0x })"},"target":"zilliqa::exec","line_number":850}
{"timestamp":"2026-02-23T03:43:42.581615Z","level":"INFO","fields":{"message":"executing txn","hash":"083379cd1db24e3f5d2bdb48d90765c9d8a1210186a0d7c5bf23b5a4194bd006","from":"0xe45d5b4699fb41d45ef972126edfb3c81c5f71c2","to":"Some(0xe122e3309a700b8c245b84f18c020bdf003f8945)","txn":"Legacy(TxLegacy { chain_id: Some(33469), nonce: 2311, gas_price: 4761904800000, gas_limit: 21000, to: Call(0xe122e3309a700b8c245b84f18c020bdf003f8945), value: 10000000000000000, input: 0x })"},"target":"zilliqa::exec","line_number":850}
{"timestamp":"2026-02-23T03:43:42.582372Z","level":"INFO","fields":{"message":"executing txn","hash":"b2b63b58ec59ac911725d1b07c903ac13dc2e2f9f98560d163eee798ad5251a4","from":"0xe45d5b4699fb41d45ef972126edfb3c81c5f71c2","to":"Some(0x7aa74c31169ecf9c6e345221da8343c2d946441c)","txn":"Legacy(TxLegacy { chain_id: Some(33469), nonce: 2312, gas_price: 4761904800000, gas_limit: 21000, to: Call(0x7aa74c31169ecf9c6e345221da8343c2d946441c), value: 10000000000000000, input: 0x })"},"target":"zilliqa::exec","line_number":850}
{"timestamp":"2026-02-23T03:43:42.583384Z","level":"INFO","fields":{"message":"executing txn","hash":"c9d8ac8bb7e04e5281e9b19dd250b3205727100f7bc9cda9fca47b3e20c4d51f","from":"0xd61d28cf7efdde292683c244ca69910aaf9348ef","to":"Some(0x1572986ab4da8ca8b5b6efd88e9600071b60d8de)","txn":"Legacy(TxLegacy { chain_id: Some(33469), nonce: 2300, gas_price: 4761904800000, gas_limit: 21000, to: Call(0x1572986ab4da8ca8b5b6efd88e9600071b60d8de), value: 10000000000000000, input: 0x })"},"target":"zilliqa::exec","line_number":850}
{"timestamp":"2026-02-23T03:43:42.584141Z","level":"INFO","fields":{"message":"executing txn","hash":"620bb6cbc43842444f656965280c1ec7c9610adb5ff98ab92583311040c95a5a","from":"0xd61d28cf7efdde292683c244ca69910aaf9348ef","to":"Some(0x242ec973bb3433cea7ddb27b7e8c1585e94683c0)","txn":"Legacy(TxLegacy { chain_id: Some(33469), nonce: 2301, gas_price: 4761904800000, gas_limit: 21000, to: Call(0x242ec973bb3433cea7ddb27b7e8c1585e94683c0), value: 10000000000000000, input: 0x })"},"target":"zilliqa::exec","line_number":850}
{"timestamp":"2026-02-23T03:43:42.584950Z","level":"INFO","fields":{"message":"executing txn","hash":"e08161896cceb4bbafc41a46da5fbdd53be2be611be9a08901385ef46204b22a","from":"0xd61d28cf7efdde292683c244ca69910aaf9348ef","to":"Some(0x63f78f6a1eadf9063dbf265fadd03903ebeaae12)","txn":"Legacy(TxLegacy { chain_id: Some(33469), nonce: 2302, gas_price: 4761904800000, gas_limit: 21000, to: Call(0x63f78f6a1eadf9063dbf265fadd03903ebeaae12), value: 10000000000000000, input: 0x })"},"target":"zilliqa::exec","line_number":850}
{"timestamp":"2026-02-23T03:43:42.585403Z","level":"WARN","fields":{"message":"transaction failed to execute","hash":"e08161896cceb4bbafc41a46da5fbdd53be2be611be9a08901385ef46204b22a","error":"database error: trie error: missing node\n\nCaused by:\n    trie error: missing node\n\nStack backtrace:\n   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.101/src/backtrace.rs:27:14\n   1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual\n             at ./rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/result.rs:2184:27\n   2: zilliqa::exec::<impl zilliqa::state::State>::apply_transaction_evm\n             at ./zilliqa/zilliqa/src/exec.rs:679:17\n   3: zilliqa::exec::<impl zilliqa::state::State>::apply_transaction\n             at ./zilliqa/zilliqa/src/exec.rs:885:73\n   4: zilliqa::consensus::Consensus::apply_transaction_at\n             at ./zilliqa/zilliqa/src/consensus.rs:1176:19\n   5: zilliqa::consensus::Consensus::execute_block\n             at ./zilliqa/zilliqa/src/consensus.rs:3334:26\n   6: zilliqa::consensus::Consensus::proposal\n             at ./zilliqa/zilliqa/src/consensus.rs:979:18\n   7: zilliqa::consensus::Consensus::receive_block\n             at ./zilliqa/zilliqa/src/consensus.rs:2703:27\n   8: zilliqa::node::Node::handle_injected_proposal\n             at ./zilliqa/zilliqa/src/node.rs:1114:34\n   9: zilliqa::node::Node::handle_request\n             at ./zilliqa/zilliqa/src/node.rs:367:22\n  10: zilliqa::node_launcher::NodeLauncher::start_shard_node::{{closure}}\n             at ./zilliqa/zilliqa/src/node_launcher.rs:283:47\n  11: zilliqa::p2p_node::P2pNode::add_shard_node::{{closure}}::{{closure}}\n             at ./zilliqa/zilliqa/src/p2p_node.rs:262:57\n  12: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/task/core.rs:374:24\n  13: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/loom/std/unsafe_cell.rs:16:9\n  14: tokio::runtime::task::core::Core<T,S>::poll\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/task/core.rs:363:30\n  15: tokio::runtime::task::harness::poll_future::{{closure}}\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/task/harness.rs:535:30\n  16: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n             at ./rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/panic/unwind_safe.rs:274:9\n  17: std::panicking::catch_unwind::do_call\n             at ./rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/std/src/panicking.rs:590:40\n  18: std::panicking::catch_unwind\n             at ./rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/std/src/panicking.rs:553:19\n  19: std::panic::catch_unwind\n             at ./rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/std/src/panic.rs:359:14\n  20: tokio::runtime::task::harness::poll_future\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/task/harness.rs:523:18\n  21: tokio::runtime::task::harness::Harness<T,S>::poll_inner\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/task/harness.rs:210:27\n  22: tokio::runtime::task::harness::Harness<T,S>::poll\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/task/harness.rs:155:20\n  23: tokio::runtime::task::raw::RawTask::poll\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/task/raw.rs:267:18\n  24: tokio::runtime::task::LocalNotified<S>::run\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/task/mod.rs:515:13\n  25: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/scheduler/multi_thread/worker.rs:643:18\n  26: tokio::task::coop::with_budget\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/task/coop/mod.rs:167:5\n  27: tokio::task::coop::budget\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/task/coop/mod.rs:133:5\n  28: tokio::runtime::scheduler::multi_thread::worker::Context::run_task\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/scheduler/multi_thread/worker.rs:634:9\n  29: tokio::runtime::scheduler::multi_thread::worker::Context::run\n  30: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/scheduler/multi_thread/worker.rs:532:24\n  31: tokio::runtime::context::scoped::Scoped<T>::set\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/context/scoped.rs:40:9\n  32: tokio::runtime::context::set_scheduler::{{closure}}\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/context.rs:176:38\n  33: std::thread::local::LocalKey<T>::try_with\n             at ./rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/std/src/thread/local.rs:508:12\n  34: std::thread::local::LocalKey<T>::with\n             at ./rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/std/src/thread/local.rs:472:20\n  35: tokio::runtime::context::set_scheduler\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/context.rs:176:17\n  36: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/scheduler/multi_thread/worker.rs:527:9\n  37: tokio::runtime::context::runtime::enter_runtime\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/context/runtime.rs:65:16\n  38: tokio::runtime::scheduler::multi_thread::worker::run\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/scheduler/multi_thread/worker.rs:519:5\n  39: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/scheduler/multi_thread/worker.rs:485:45\n  40: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/blocking/task.rs:42:21\n  41: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/task/core.rs:374:24\n  42: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/loom/std/unsafe_cell.rs:16:9\n  43: tokio::runtime::task::core::Core<T,S>::poll\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/task/core.rs:363:30\n  44: tokio::runtime::task::harness::poll_future::{{closure}}\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/task/harness.rs:535:30\n  45: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n             at ./rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/panic/unwind_safe.rs:274:9\n  46: std::panicking::catch_unwind::do_call\n             at ./rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/std/src/panicking.rs:590:40\n  47: std::panicking::catch_unwind\n             at ./rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/std/src/panicking.rs:553:19\n  48: std::panic::catch_unwind\n             at ./rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/std/src/panic.rs:359:14\n  49: tokio::runtime::task::harness::poll_future\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/task/harness.rs:523:18\n  50: tokio::runtime::task::harness::Harness<T,S>::poll_inner\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/task/harness.rs:210:27\n  51: tokio::runtime::task::harness::Harness<T,S>::poll\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/task/harness.rs:155:20\n  52: tokio::runtime::task::raw::RawTask::poll\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/task/raw.rs:267:18\n  53: tokio::runtime::task::UnownedTask<S>::run\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/task/mod.rs:552:13\n  54: tokio::runtime::blocking::pool::Task::run\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/blocking/pool.rs:161:19\n  55: tokio::runtime::blocking::pool::Inner::run\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/blocking/pool.rs:516:22\n  56: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}\n             at ./usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.49.0/src/runtime/blocking/pool.rs:474:47\n  57: std::sys::backtrace::__rust_begin_short_backtrace\n             at ./rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/std/src/sys/backtrace.rs:158:18\n  58: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}\n             at ./rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/std/src/thread/mod.rs:562:17\n  59: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n             at ./rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/panic/unwind_safe.rs:274:9\n  60: std::panicking::catch_unwind::do_call\n             at ./rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/std/src/panicking.rs:590:40\n  61: std::panicking::catch_unwind\n             at ./rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/std/src/panicking.rs:553:19\n  62: std::panic::catch_unwind\n             at ./rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/std/src/panic.rs:359:14\n  63: std::thread::Builder::spawn_unchecked_::{{closure}}\n             at ./rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/std/src/thread/mod.rs:560:30\n  64: core::ops::function::FnOnce::call_once{{vtable.shim}}\n             at ./rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/ops/function.rs:250:5\n  65: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once\n             at ./rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/alloc/src/boxed.rs:2005:9\n  66: std::sys::thread::unix::Thread::new::thread_start\n             at ./rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/std/src/sys/thread/unix.rs:126:17\n  67: <unknown>\n  68: <unknown>"},"target":"zilliqa::consensus","line_number":1180}

Update: no smoking gun - but symptoms are consistent with normal database corruption e.g. if a node got killed before the data is flushed. So, added some flushing after snapshot.

[shawn-zil]

While we cannot determine exactly when a compaction will prune the nodes, things eventually get pruned e.g.

For VAL1 node:

Keys in range: 389374
Value size distribution: 
Count: 389374 Average: 150.5088  StdDev: 251.00
Min: 1  Median: 141.7327  Max: 22310
Percentiles: P50: 141.73 P75: 161.97 P99: 552.47 P99.9: 577.75 P99.99: 15227.30

Keys in range: 101062
Value size distribution: 
Count: 101062 Average: 278.4282  StdDev: 236.80
Min: 1  Median: 168.1757  Max: 22310
Percentiles: P50: 168.18 P75: 441.53 P99: 574.50 P99.9: 579.49 P99.99: 579.99

For VAL0:

Keys in range: 279785
Value size distribution: 
Count: 279785 Average: 198.3408  StdDev: 253.69
Min: 1  Median: 148.5379  Max: 22310
Percentiles: P50: 148.54 P75: 198.60 P99: 569.05 P99.9: 579.06 P99.99: 5141.22

Keys in range: 142266
Value size distribution: 
Count: 142266 Average: 232.9204  StdDev: 241.08
Min: 1  Median: 155.3322  Max: 22310
Percentiles: P50: 155.33 P75: 385.87 P99: 572.31 P99.9: 579.31 P99.99: 1036.28