Merge ZcashFoundation/zebra v2.4.2 into zsa1 (conflict resolutions + minimal follow-ups) - new #88
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ation#8978) * Remove temporary dependency patches We need to enable the `legacy-api` feature of `incrementalmerkletree` to be able to serialize note commitment trees using an old serialization format for the `z_gettreestate` RPC. * Use spaces instead of a tab * Bump ECC deps to match `zcashd`
* Run `cargo update` * chore: Release * Update `release-crates-dry-run.sh` * Update `ESTIMATED_RELEASE_HEIGHT` * Update `CHANGELOG.md` * Update `ESTIMATED_RELEASE_HEIGHT`
* book: add section about private testnet testing * Apply suggestions from code review Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Arya <aryasolhi@gmail.com> --------- Co-authored-by: Pili Guerra <mpguerra@users.noreply.github.com> Co-authored-by: Arya <aryasolhi@gmail.com>
…Foundation#8986) Previously, most of our deployed instances needed to sync the whole blockchain from genesis, but after implementing the mounting of cached states for the release instances, this is no longer required. Main changes: - Reduce the boot disk size for CD images to 10GB (in CI Zebra might need to rebuild based on test flags, requiring more disk space) - Use `pd-standard` instead of `pd-ssd` for the boot disk - Use `pd-balanced` instead of `pd-ssd` for the mounted disk (where most of the reads and writes happens) - Change our `GCP_SMALL_MACHINE` from `c2-standard-4` (vCPUs: 4, RAM: 16 GiB) to `c2d-standard-2` (vCPUs: 2, RAM: 8 GiB) - Keep long running tests `is_long_test` with `GCP_LARGE_MACHINE` (`c2d-standard-16`) and other with the new `GCP_SMALL_MACHINE` configuration (`c2d-standard-2`)
…ndation#8987) In some cases Zebra logs might not output the database version, and thus we should avoid creating a disk without a version. Before this change, a disk was created without a db version number, just indicating a `-v-`, that caused other tests to fail as an actual version was not found in their regexes.
…cashFoundation#8993) Bumps the devops group with 5 updates in the / directory: | Package | From | To | | --- | --- | --- | | [actions/checkout](https://github.com/actions/checkout) | `4.2.1` | `4.2.2` | | [google-github-actions/auth](https://github.com/google-github-actions/auth) | `2.1.6` | `2.1.7` | | [google-github-actions/setup-gcloud](https://github.com/google-github-actions/setup-gcloud) | `2.1.1` | `2.1.2` | | [jontze/action-mdbook](https://github.com/jontze/action-mdbook) | `3.0.0` | `3.0.1` | | [docker/scout-action](https://github.com/docker/scout-action) | `1.14.0` | `1.15.0` | Updates `actions/checkout` from 4.2.1 to 4.2.2 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v4.2.1...v4.2.2) Updates `google-github-actions/auth` from 2.1.6 to 2.1.7 - [Release notes](https://github.com/google-github-actions/auth/releases) - [Changelog](https://github.com/google-github-actions/auth/blob/main/CHANGELOG.md) - [Commits](google-github-actions/auth@v2.1.6...v2.1.7) Updates `google-github-actions/setup-gcloud` from 2.1.1 to 2.1.2 - [Release notes](https://github.com/google-github-actions/setup-gcloud/releases) - [Changelog](https://github.com/google-github-actions/setup-gcloud/blob/main/CHANGELOG.md) - [Commits](google-github-actions/setup-gcloud@v2.1.1...v2.1.2) Updates `jontze/action-mdbook` from 3.0.0 to 3.0.1 - [Release notes](https://github.com/jontze/action-mdbook/releases) - [Changelog](https://github.com/jontze/action-mdbook/blob/master/CHANGELOG.md) - [Commits](jontze/action-mdbook@v3.0.0...v3.0.1) Updates `docker/scout-action` from 1.14.0 to 1.15.0 - [Release notes](https://github.com/docker/scout-action/releases) - [Commits](docker/scout-action@v1.14.0...v1.15.0) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-patch dependency-group: devops - dependency-name: google-github-actions/auth dependency-type: direct:production update-type: version-update:semver-patch dependency-group: devops - dependency-name: google-github-actions/setup-gcloud dependency-type: direct:production update-type: version-update:semver-patch dependency-group: devops - dependency-name: jontze/action-mdbook dependency-type: direct:production update-type: version-update:semver-patch dependency-group: devops - dependency-name: docker/scout-action dependency-type: direct:production update-type: version-update:semver-minor dependency-group: devops ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* add pub functionality for zaino * updated doc comment with review suggestion
…on#9005) This is an initial step before completely removing Mergify, to keep the duality betwen the two to avoid downtime with queue management
…cashFoundation#9002) Bumps the devops group with 1 update: [rlespinasse/github-slug-action](https://github.com/rlespinasse/github-slug-action). Updates `rlespinasse/github-slug-action` from 4 to 5 - [Release notes](https://github.com/rlespinasse/github-slug-action/releases) - [Commits](rlespinasse/github-slug-action@v4...v5) --- updated-dependencies: - dependency-name: rlespinasse/github-slug-action dependency-type: direct:production update-type: version-update:semver-major dependency-group: devops ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: pinglanlu <pinglanlu@outlook.com>
* typo 0005-treestate.md * typo xxxx-block-subsidy.md * typo continuous-delivery.md * typo docker.md * typo shielded-scan.md
…ndation#9033) While using Mergify's configuration checker, this issue was not raised, but it's now showing in some PR's summaries, as in https://github.com/ZcashFoundation/zebra/pull/9026/checks?check_run_id=33066692861
…tion#9007) * chore: update CHANGELOG to better convey the 2.0.0 issue * Apply suggestions from code review Co-authored-by: Arya <aryasolhi@gmail.com> --------- Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com> Co-authored-by: Arya <aryasolhi@gmail.com>
…cashFoundation#8857) * Adds a parameter to `zebra_consensus::router::init()` for accepting a mempool setup argument, adds and uses an `init_test()` fn for passing a closed channel receiver in tests where no mempool service is needed in the transaction verifier. * Adds a `mempool` argument to the transaction::Verifier constructor (and a `new_for_tests()` constructor for convenience) * Removes `Clone` impl on `transaction::Verifier` to add mempool oneshot receiver, updates tests. * Adds TODOs * updates transaction verifier's poll_ready() method to setup the mempool service handle. * Updates VerifiedSet struct used in mempool storage * Updates mempool service and its `Storage` to use the updated `VerifiedSet` `transactions()` return type. * updates `created_outputs` when inserting or removing a transaction from the mempool's verified set * Adds a TODO, updates field docs * Updates `spent_utxos()` to query the mempool for unspent outputs * Adds `spent_mempool_outpoints` as a field on tx verifier mempool response * Updates mempool `Downloads` to return the spent_mempool_outpoints from the tx verifier response * Updates `Storage.insert()` to accept a list of spent mempool transaction outputs * Adds transaction dependencies when inserting a tx in `VerifiedSet` * polls mempool svc from tx verifier when a mempool tx that creates transparent outputs has been verified. adds a TODO for adding a `pending_outputs` field to the mempool Storage * Adds `pending_outputs` field on mempool Storage and responds to pending outputs requests when inserting new transactions into the mempool's verified set * replaces `UnminedTxId` type with `transaction::Hash` in mempool's verified set * prune pending outputs when rejecting and removing same effects. * Remove dependent transactions from verified set when removing a tx * updates tests * appeases clippy. * removes unused `len()` method * fixes doc links * Adds transaction dependencies to the `FullTransactions` response, let the caller handle it (required to avoid moving zip317 tx selection code to mempool) * updates block template construction to avoid including transactions unless their dependencies have already been added. * updates tests * Replaces placeholder setup channel with one that sends the mempool svc to the tx verifier, adds a timeout layer, adds a TODO about a concurrency bug * Use a single query to check for unspent outputs in the mempool * Updates `getblocktemplate` method to consider dependencies when sorting transactions for the final template * fixes clippy lints, removes unnecessary Option in UnspentOutput response variant * renames type alias and method, adds a TODO to use iteration instead of recursion * Adds mempool_removes_dependent_transactions() test * Updates Storage and VerifiedSet clear() methods to clear pending_outputs, created_outputs, and transaction_dependencies, adds TODO to use iteration instead of recursion. * removes outdated TODO * Adds a TODO for reporting queued transaction verification results from the mempool from the poll_ready() method * Adds `mempool_responds_to_await_output` test * updates mempool_responds_to_await_output test * Uses iteration instead of recursion in verified set's remove() method and zip317 mod's dependencies_depth() method * Adds a mempool_request_with_mempool_output_is_accepted test for the transaction verifier * Moves delay duration before polling the mempool to a constant, uses a shorter timeout for mempool output lookups, adds a `poll_count` to MockService, and updates `mempool_request_with_unmined_output_spends_is_accepted` to check that the transaction verifier polls the mempool after verifying a mempool transaction with transparent outputs * adds long_poll_input_mempool_tx_ids_are_sorted test * Adds a `excludes_tx_with_unselected_dependencies` test * Updates a TODO * moves `TransactionDependencies` struct to `zebra-node-services` * Updates `FullTransactions` response variant's `transaction_dependencies` type * updates zip317 transaction selection for block templates to include dependent transactions * Moves and refactors zip317 tx selection test to its own module, adds an `unmined_transactions_in_blocks()` method on network * Removes `unmined_transactions_in_blocks()` test utility fn from mempool Storage test module and replaces calls to it with calls to the new test method on Network * Fixes spelling mistake * Adds `includes_tx_with_selected_dependencies` test * fixes zip317 block construction issue * Fixes vectors test * Update zebra-node-services/src/mempool.rs * restores `tip_rejected_exact` type * updates affected tests * Documents the new argument in `Storage::insert()`, updates outdated comment * Update zebrad/src/components/mempool/storage/verified_set.rs * fixes potential issue with calling buffered mempool's poll_ready() method without calling it. * Avoids removing dependent transactions of transactions that have been mined onto the best chain. * Updates `spent_utxos()` method documentation * Avoids sorting getblocktemplate transactions in non-test compilations * documents PendingOutputs struct * Apply suggestions from code review Co-authored-by: Marek <mail@marek.onl> * cargo fmt * Applies suggestions from code review Avoids unnecessarily rejecting dependent transactions of randomly evicted mempool transactions. Updates `TransactionDependencies::remove_all()` to omit provided transaction id from the list of removed transaction ids. * Applies suggestions from code review. * Adds minor comments * Update zebrad/src/components/mempool/storage/verified_set.rs Co-authored-by: Marek <mail@marek.onl> * Remove an outdated comment (ZcashFoundation#9013) --------- Co-authored-by: Marek <mail@marek.onl>
* Adds getblockheader RPC method * Updates snapshots, adds hash/height/next_block_hash fields to verbose response * updates getblock snapshot * updates getblockheader response type to hex-encode fields, adds ToHex impl for sapling::tree::Root, adds snapshot and vector tests for new RPC method, adds snapshots. * rustfmt * fixes snapshots, matches zcashd more closely * fixes vectors test * updates lwd failure messages (probably doesn't matter, but seems better to handle it now than risk debugging it later) * fixes getblock_rpc test, fixes/reverses finalsaplingroot field byte-order. * fixes vector test, addresses remaining differences with zcashd (except the `chainwork` field), updates snapshots, and avoids a possible panic when there's a chain reorg between state queries. * Adds a comment noting that the `relative_to_network()` method was copied from zcashd * Apply suggestions from code review Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com> --------- Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
A LWD test was expecting the `ZEBRA_TEST_LIGHTWALLETD` to be set, but this variable is needed for all LWD tests and not specifically for `lightwalletd_integration`. We had to rename this variable on a buggy `elif` statement in our Docker entrypoint. This was avoiding most LWD tests to run correctly.
* Update the reference Sapling treestate Zebra's treestate serialization differs from `zcashd` in the following way: `zcashd` omits the serialization of empty trailing ommers, while Zebra doesn't. This means that `zcashd` serializes the Sapling treestate for height 419_201 as 019eb30778ddeea84c72e69e07a1689f3c8def3dc0a1939f0edcbe47279069d9310002000150715810d52caf35471d10feb487213fbd95ff209122225b7b65d27a7fb1a44d Whereas Zebra serializes it as 019eb30778ddeea84c72e69e07a1689f3c8def3dc0a1939f0edcbe47279069d931001f000150715810d52caf35471d10feb487213fbd95ff209122225b7b65d27a7fb1a44d0000000000000000000000000000000000000000000000000000000000 Serialization Format ==================== The serialized treestate consists of optional, hex-encoded, 32-byte hashes. If the hash is not present, it is serialized as byte 0, i.e., `0x00`. If the hash is present, it is prefixed by byte 1, i.e. `0x01`. The first two hashes in the serialized treestate are the treestate's left and right leaves. These are followed by the serialized length of the vector of ommers. This length is serialized as 1, 3, 5, or 9 bytes. If the length is less than 253, it is serialized as a single byte. The length is then followed by the serialized ommers. We can now parse the first string, produced by `zcashd`: - `0119eb30778ddeea84c72e69e07a1689f3c8def3dc0a1939f0edcbe47279069d931` is the serialized left leaf, - `00` is the serialized right leaf, - `02` is the serialized length of the vector of ommers, - `00` is the serialized first ommer, - `0150715810d52caf35471d10feb487213fbd95ff209122225b7b65d27a7fb1a44d` is the serialized second ommer. And the second one, produced by Zebra: - `0119eb30778ddeea84c72e69e07a1689f3c8def3dc0a1939f0edcbe47279069d931` is the serialized left leaf, - `00` is the serialized right leaf, - `1f` is the serialized length of the vector of ommers, - `00` is the serialized first ommer, - `0150715810d52caf35471d10feb487213fbd95ff209122225b7b65d27a7fb1a44d` is the serialized second ommer - `0000000000000000000000000000000000000000000000000000000000` are the remaining 29 serialized ommers. Note that both serializations represent the same treestate. * Remove a new line char
… on release PRs (ZcashFoundation#8992) * Adds a test to check for crates in the Cargo.lock file that are being pulled in from a git source. * add `check_no_git_refs_in_cargo_lock` to CI * try skip instead of exclude --------- Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Allows non-local impls for abscissa derive macros and fixes other clippy lints. * Fixes formatting
…hFoundation#9006) * rpc: align getblock with zcashd behaviour * Removes handling for verbosity = 3 in getblock method, adds finalorchardroot field, removes unnecessary state request. (ZcashFoundation#9008) * align final(sapling|orchard)root with zcashd behaviour * fix test * Apply suggestions from code review Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com> * restore getblock docs; remove unneeded TODOs * Update zebra-rpc/src/methods.rs Co-authored-by: Arya <aryasolhi@gmail.com> * get rif of cloning self (ZcashFoundation#9044) --------- Co-authored-by: Arya <aryasolhi@gmail.com> Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
…t transactions in the mempool (ZcashFoundation#9067) * change(mempool): Return verification result after attempting to insert transactions in the mempool (ZcashFoundation#8901) * respond with mempool verification result after a transaction has been inserted or has failed to be inserted into the mempool * returns mempool verification errors early, and fixes handling for cancellations or timeouts. * Adds a comment in test warning against code reuse with buffered services. * De-duplicates handling for timeout errors
…on#9068) - Removed deprecated actions and simplified default queue rules. - Adjusted batch sizes for urgent and batched queues. - Consolidated priority rules to enhance clarity and efficiency. - Updated conditions for moving pull requests to different queues based on labels and reviews.
…hFoundation#9629) * use a regtest miner address * give more time to zebra start * cleanup what we can after addnode * fix timeout for connections --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
…on#9654) * feat(rpc): Sync all chains in `TrustedChainSync::sync` Adds a `ReadRequest` variant for listening to changes in the non-finalized state. * Implements `non_finalized_state_change()` gRPC method * Uses gRPC-method handler-method tracing::Span when logging in spawned tasks. * Moves logic to `NonFinalizedBlocksListener::spawn`,moves channel buffer size to a constnat, and adds correctness notes. * updates `TrustedChainSync` to use new grpc method * refactors `sync()` method * Fixes an issue where any blocks committed to the non-finalized state in TrustedChainSync were being sent as the best chain tip Fixes an issue where finalized chain tip changes were being missed in the chain_tip_change in TrustedChainSync * rustfmt * Applies suggestions from code review
…ds to TransactionObject (ZcashFoundation#9652) * Adds binding_sig, joinsplit_pub_key and joinsplit_sig fields to TransactionObject. Updates get raw transaction test. * clippy * Corrects unit test, fmt, clippy * Adds hex field serialization test, corrects mispelled words in doc comment, adds opthex serde attribute and HexSignature HexBytes serialization wrappers. * fmt, clippy, check * Resolves conflicts in serialization test * adjustments to match zcashd * address comment --------- Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
Bumps the ecc group with 2 updates: [zcash_protocol](https://github.com/zcash/librustzcash) and [zcash_script](https://github.com/ZcashFoundation/zcash_script). Updates `zcash_protocol` from 0.5.2 to 0.5.3 - [Release notes](https://github.com/zcash/librustzcash/releases) - [Commits](zcash/librustzcash@zcash_protocol-0.5.2...zcash_protocol-0.5.3) Updates `zcash_script` from 0.3.1 to 0.3.2 - [Release notes](https://github.com/ZcashFoundation/zcash_script/releases) - [Changelog](https://github.com/ZcashFoundation/zcash_script/blob/master/CHANGELOG.md) - [Commits](ZcashFoundation/zcash_script@v0.3.1...v0.3.2) --- updated-dependencies: - dependency-name: zcash_protocol dependency-version: 0.5.3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: ecc - dependency-name: zcash_script dependency-version: 0.3.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: ecc ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* remove zebra-scan * removed zebra-grpc * remove shielded scan book sections * Apply suggestions from code review Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com> --------- Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
* Support V4 coinbase * Don't skip Canopy in Regtest tests * Add network accessor method to RpcImpl * Support Canopy commitment bytes and adjust tests * Set indexer_listen_addr to localhost in trusted_chain_sync test * Don't skip Canopy in Regtest tests * Support Canopy commitment bytes and adjust tests * Update zebrad/tests/acceptance.rs Co-authored-by: Arya <aryasolhi@gmail.com> * Reorder imports * address comments --------- Co-authored-by: Arya <aryasolhi@gmail.com> Co-authored-by: Conrado Gouvea <conrado@zfnd.org>
* add CHANGELOGs * bump versions * update checklist; EoS * remove removed changelog entries * Apply suggestions from code review Co-authored-by: Arya <aryasolhi@gmail.com> * address review comments * bump zebrad * update changelog with last merged PRs * update checkpoints * cargo upgrade * Apply suggestions from code review Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com> * fix release dates of crates * fix testnet checkpoint * fix checkpoint test --------- Co-authored-by: Arya <aryasolhi@gmail.com> Co-authored-by: Alfredo Garcia <oxarbitrage@gmail.com>
…tion#9712) The create-state-image job condition was using `!cancelled() && !failure()` which checks workflow-level status, not job-specific results. This caused disk images to be created even when the test-result job was cancelled, failed, or timed out, leading to corrupted cached states. Change condition to `needs.test-result.result == 'success'` to only create disk images when the specific test actually succeeded.
…ation#9709) * fix(state): fix 2.4.0 DB upgrade; add warning if impacted * use usize::MAX instead of 0 for the dummy location * fix import * fix max values
…dation#9711) - Remove release exclusion from get-disk-name job condition - Make disk-finding conditional within sub-find-cached-disks workflow - Add release defaults step to provide empty cached disk outputs The deploy-nodes job was being skipped for releases because it depended on get-disk-name which excluded release events. Now get-disk-name runs for releases but skips disk lookup and provides default outputs, allowing deploy-nodes to proceed with production deployment flow.
* bump to 2.4.1 * checklist improvements, changelog fixes
This commit applies the cumulative post-merge fixes on top of `zsa1-v2.4.2-conflicts-new` by copying the file states from the tip of `zsa1-v2.4.2-fixes` (old stack), without re-resolving conflicts. Context: - Base: zsa1-v2.4.2-conflicts-new (upstream v2.4.2 import) - Source of fixes: zsa1-v2.4.2-fixes@<OLD_SHA> Contents (intent): - Conflict resolutions carried over from the old stack - Build and test follow-ups required after v2.4.2 import - No intentional behavior changes beyond integration fixes Notes: - This is a squash-style carryover to keep the new PR stack reviewable. - For provenance, see the original fixes branch at <OLD_SHA>. Refs: upstream v2.4.2, zsa1
…m zebra-chain and zebra-network
….rs and specify temporary values for them
…7 activation heights
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR updates our
zsa1branch to upstreamv2.4.2. It includes: