Make compression level adaptive and configurable #NIT-4153

[mahdy-nasr]

In this PR we are changing current compression rates and static thresholds to something more configurable by operators.

The previous --node.batch-poster.compression-level flag is deprecated in this PR and replaced by:
--node.batch-poster.compression-levels which allow user to set json array in the following form:

[{"backlog": <int>, "level": <int>, "recompression-level": <int>}, ...]

Now you can define by which threshold backlog , which compression level should be used and which block/recompression level should be used. There are validation rules on this flag like you shouldn't have increasing compression level with i->n. (more details on the PR code and on the issue #NIT-4153

[github-actions]

❌ 8 Tests Failed:

Tests completed	Failed	Passed	Skipped
4480	8	4472	0

View the top 3 failed tests by shortest run time

TestMultiGasConstraintsStorage

Stack Traces | 0.000s run time

=== RUN   TestMultiGasConstraintsStorage
=== PAUSE TestMultiGasConstraintsStorage
=== CONT  TestMultiGasConstraintsStorage
    constraints_test.go:377: 
        	Error Trace:	/home/runner/work/nitro/nitro/precompiles/constraints_test.go:377
        	Error:      	Not equal: 
        	            	expected: 0x1
        	            	actual  : 0x3
        	Test:       	TestMultiGasConstraintsStorage
--- FAIL: TestMultiGasConstraintsStorage (0.00s)

TestMultiGasConstraintsStorage

Stack Traces | 0.000s run time

=== RUN   TestMultiGasConstraintsStorage
=== PAUSE TestMultiGasConstraintsStorage
=== CONT  TestMultiGasConstraintsStorage
INFO [12-18|17:23:22.485] Load database journal from disk
INFO [12-18|17:23:22.485] State snapshot generator is not found
INFO [12-18|17:23:22.485] Starting snapshot generation             root=56e81f..63b421 accounts=0 slots=0 storage=0.00B dangling=0 elapsed="1.4µs"
INFO [12-18|17:23:22.485] Initialized path database                triecache=16.00MiB statecache=16.00MiB buffer=64.00MiB state-history="last 90000 blocks"
INFO [12-18|17:23:22.485] Resuming snapshot generation             root=56e81f..63b421 accounts=0 slots=0 storage=0.00B dangling=0 elapsed="145.58µs"
INFO [12-18|17:23:22.486] Generated snapshot                       accounts=0 slots=0 storage=0.00B dangling=0 elapsed="263.153µs"
    constraints_test.go:377: 
        	Error Trace:	/home/runner/work/nitro/nitro/precompiles/constraints_test.go:377
        	Error:      	Not equal: 
        	            	expected: 0x1
        	            	actual  : 0x3
        	Test:       	TestMultiGasConstraintsStorage
--- FAIL: TestMultiGasConstraintsStorage (0.00s)

TestMultiGasConstraintsStorage

Stack Traces | 0.030s run time

=== RUN   TestMultiGasConstraintsStorage
=== PAUSE TestMultiGasConstraintsStorage
=== CONT  TestMultiGasConstraintsStorage
INFO [12-18|17:03:28.120] Load database journal from disk
    constraints_test.go:377: 
        	Error Trace:	/home/runner/work/nitro/nitro/precompiles/constraints_test.go:377
        	Error:      	Not equal: 
        	            	expected: 0x1
        	            	actual  : 0x3
        	Test:       	TestMultiGasConstraintsStorage
--- FAIL: TestMultiGasConstraintsStorage (0.03s)

📣 Thoughts on this report? Let Codecov know! | Powered by Codecov

[mahdy-nasr]

will fix the lint and other stuff

[Tristan-Wilson]

Looks largely good once build/test issues are fixed, just some minor requests.

[Tristan-Wilson]

I see that CompressionLevels is set later when we merge the old and new configs, it's a bit non-obvious to readers, is there a way to set it here?

[mahdy-nasr]

CompressionLevel is set here for a reason: it defines the default value for the CLI argument. But for CompressionLevels would effectively remain unused. I think remove all of them would be a better choice.

For ResolveCompressionLevel, the intent is to first check whether either of the two arguments was explicitly set. Having default values makes it harder to distinguish between “unset” and “explicitly set,” which complicates that logic.

From my perspective, leaving both fields unset and having a single, clear path for determining where the configuration comes from is easier to reason about and less error-prone.

To achieve this, I can remove CompressionLevel: 0 altogether (since it already defaults to zero).
What do you think?

[Tristan-Wilson]

Still some errors like this in the logs

=== FAIL: system_tests TestChallengeProtocolBOLDReadInboxChallenge (18.67s)
panic: invalid cloned config: cannot specify both compression-level (deprecated) and compression-levels; use only compression-levels [recovered, repanicked]

[codecov]

Codecov Report

❌ Patch coverage is 20.98765% with 64 lines in your changes missing coverage. Please review.
✅ Project coverage is 29.37%. Comparing base (5b59a1a) to head (a7eb894).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4145      +/-   ##
==========================================
- Coverage   33.35%   29.37%   -3.98%     
==========================================
  Files         453      454       +1     
  Lines       55536    55601      +65     
==========================================
- Hits        18524    16335    -2189     
- Misses      33774    36303    +2529     
+ Partials     3238     2963     -275