SKU/molecule #1234
Draft
SKU/molecule #1234
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…es w/o content_length (#1235)
…arlier than expected (#1237)
* Kicked off webhooks domain: described webhook subscription aggregate * Added 'WebhookEvent' simple aggregate * Sketched subscription event store methods * Sketched WebhookDelivery (fact) and it's repository * Couple more methods in delivery repository * Improved ID types. Introduced TaskAttemptId type * Introduced concept of `TaskAttempt` in task, and reshuffled state structure around it without changing event store * Drafter retrying logic and retry policies at task aggregate level. Not integrated or materialized yet. * Drafted retries in task snapshots * Working sqlite migration * Stabilized existing tests * Unit tests added for retry policies * Task scheduler config for retry policy + service-layer tests for retries * Implemented repository for webhook events (3 incarnations, no tests yet) * Unit tests implemented for WebhookEvent repositories. Detecting duplicate ID when creating events. * SQL for webhook subscriptions and deliveries * Drafted in-memory repo for webhook subscriptions * Sketched in-memory webhook delivery repository * Drafted Postgres/SQlite implementations fo webhook delivery repository * Drafted Postgres webhook subscription event store implementation * Implemented SQlite version of webhook subscription store * Tests for webhook delivery repository * Basic coverage for webhook subscription event store * Merge corrections * Finished tests for repository layer * Test corrections * Drafted webhook event bridge service * Sketched webhooks service layer, including: - webhook logical plan in Task System, propagation through planner/runner - webhook sender: populates delivery object, generates headers, sends webhook, updates the delivery object with webhook response - webhook signer: implementation of RFC9421 * udeps cleaned * Wrote very naive webhook signing and sender tests * Removed "ref" field from webhook event + linter fixed * WebhookSender => WebhookDeliveryWorker * Separated webhook outbox bride and event builder * Sending more universal DATASET.REF.UPDATED event with "blockRef" field * Separated WebhookSender from WebhookDelivery worker (to increase testability) * WebhookOutboxBridge => WebhookDeliveryScheduler * Drafted webhook secret generator * Added outbox consumer that removes subscriptions of a removed dataset * MInor correction in sending test * Sketched webhook subscriptions GQL api (no tests yet, incomplete) * Self-review question * Merge corrections * merge corrections * Removed task retries. Binding webhook delivery 1-to-1 to task * First test for subscriptions API * Extended test coverage for webhook subscription create GQL entry. Extracted creation use case * Reorganized GQL api handlers to use use cases more * Reorganized webhook subscriptions API to be more dataset bound to avoid separate security checking * Reshuffled update operations in GQL for webhook subscriptions * Idempotent subscription reactions * More subscription GQL API tests * Webhook subscription use case tests * Tests for WebhookDatasetRemovalHandler * Tests for webhook delivery scheduler service * Tests for webhook delivery worker * Merge corrections * Will lint/codegen pass with project toolchain? * try fixing lint/codegen * Removed obsolete comment * Recovered original formatting of GQL schema
* Custom workflow for SQLX migrations dev branch * GQL: Account Deletion API (#1242) * GQL: AccountMut::delete_account(): scaffolding * validate_password(): use .len() * DidSecretKeyRowModel: use type fullpath for DidEntityType * schema.gql: update * AccountMut::delete_account_by_name(): use DeleteAccountUseCase * AccountMut: use AccountName scalar instead of String * AccountMut: use Email scalar instead of String * AccountService::delete_account_by_name(): add * make sqlx-local-setup: fix * did_secret_keys: drop creator_id * CreateAccountUseCaseImpl::execute(): update fallback email generation * AccountService::delete_account_by_name(): add [2] * AccountRepository::delete_account_by_name(): implement * account_messages.rs -> account_lifecycle_message.rs * DeleteAccountUseCaseImpl: send AccountLifecycleMessage::deleted() * AccountRepository::delete_account_by_name(): return the removed account not only id * AccountLifecycleMessageDeleted::display_name: add * DatasetAccountDeletionHandler: introduce * DatasetRegistry::all_dataset_handles_by_owner_id(): implement * DatasetRegistry::all_dataset_handles_by_owner() -> all_dataset_handles_by_owner_name() * DatasetAccountDeletionHandler: introduce [2] * DidSecretKeyRepository::delete_did_secret_key(): add * kamu-account-services: group message producers and consumers * messaging_outbox::prelude: introduce * messaging_outbox::prelude: introduce * DidSecretService: handle DatasetLifecycleMessage * access_tokens: add ON DELETE CASCADE for account_id * accounts_passwords: add account_id * Linter fixes * CreateAccountUseCaseImpl::generate_email(): extract * PasswordHashRepository::save_password_hash(): add "account_id" * Makefile: resort vertically db crates * AccountLifecycleMessageDeleted: add "email" field * DidSecretKeyRepository::get_did_secret_key(): add * test_insert_and_locate_did_secret_keys(): update * test_create_account(): fixed * test_create_dataset_from_snapshot_creates_did_secret_key(): fixed * test_update_email_bad_email(): fixed * Makefile: remove db crate duplicates * sqlx: add cached queries * AccountsMut::create_account(): absorb * AccountMut::delete(): rethink * AccountMut: access checks * AccountMut: update tracings * DeleteAccountUseCaseImpl: only admins * GQL: AccountMut::modify_password(): re-think * DeleteAccountUseCaseImpl: allow self-deletion * Self-review * CHANGELOG.md: update * DeleteAccountUseCaseImpl::authenticated(): renamed from unauthenticated() * Integration fixes * SelfDeletionIsForbidden: remove unused struct * AccountRepository::delete_account_by_name(): do not return deleted account * OsoDatasetAuthorizer: move dill macros to struct declaration * DeleteAccountUseCase::execute(): take &Account as argument * DeleteAccountUseCaseImpl: use utils::AccountAuthorizationHelper * DatasetRegistry::all_dataset_handles_by_owner_id(): return odfOwnedDatasetHandleStream * Revert "DatasetRegistry::all_dataset_handles_by_owner_id(): return odfOwnedDatasetHandleStream" This reverts commit 8ac0da8. * DatasetAccountDeletionHandler::handle_account_lifecycle_deleted_message(): add a PERF note * CI: fix codegen action * CHANGELOG.md: update * test_delete_account_use_case_impl(): implement * GQL: test_accounts: add tests * CHANGELOG.md: update * Tests fixes * sqlx: update cached queries * Remove password logic from account service level (#1243) * Refactor password logic * Update changelog * GQL: Collection API, `extra_data` validation (#1246) * CollectionMut: take dataset by a ref * VersionedFileMut: take dataset by a ref * GQL: Add ExtraData scalar * GQL: use ExtraData scalar * Typo fixes * Fix tests * GQL: ExtraData scalar: add tests * CHANGELOG.md: update * Allow MIT-0 license usage * kamu-adapter-graphql: correct feature gate * Release (minor): 0.238.0 (#1248) --------- Co-authored-by: Sergei Zaychenko <szaychenko@kamu.dev> Co-authored-by: Roman Boiko <roman.bv20@gmail.com>
* Upgrade to new rustc and 2024 edition (#1254) * Search by account name (#1253) * Search filters also by account name * Update changelog * Fix review comments. Iter 1 * Replace format by to_string() * Fix fmt * Wallet based authentication: Phase 1 (#1239) * EvmWalletAuthenticationProvider: scaffolding * kamu-datasets: remove extra dep (itertools) * kamu-adapter-auth-web3: implement Web3WalletAuthenticationProvider (w/o nonce checking) * kamu --show-error-stack-trace * APIServerRunCommand: get token after HTTP server initialization * kamu-cli: activate Web3WalletAuthenticationProvider * kamu-adapter-graphql: extract auth_mut/ * kamu-adapter-auth-web3: ChecksumWalletAddress -> ChecksumEvmWalletAddress * GQL: AuthWeb3Mut::nonce(): implement * kamu-web3: introduce Web3AuthNonceRepository * kamu-web3-services: introduce Web3NonceServiceImpl * kamu-cli: register kamu-web3-services * kamu-auth-web3: update EIP_4361_EXPECTED_STATEMENT text * schema.gql: update * Web3NonceServiceImpl: impl InitOnStartup * kamu-web -> kamu-auth-web * kamu-auth-web3-inmem: implement * kamu-auth-web3-repo-tests: implement * kamu-auth-web3-inmem: tests * AuthWeb3Mut: nonce() -> eip4361_auth_nonce() * kamu-auth-web3-postgres: scaffolding * kamu-auth-web3-postgres: implement * Web3AuthenticationNonceEntity: expired_at -> expires_at * kamu-auth-web3-sqlite: implement * sqlx: update cached queries * Web3AuthenticationNonce -> Web3AuthenticationEip4361Nonce * Web3AuthenticationNonceEntity -> Web3AuthenticationEip4361NonceEntity * Web3AuthenticationEip4361NonceEntity -> Web3AuthEip4361NonceEntity * Web3AuthNonceRepository -> Web3AuthEip4361NonceRepository * Web3NonceService -> Web3AuthEip4361NonceService * Web3NonceServiceImpl -> Web3AuthEip4361NonceServiceImpl * Add auth_eip4361 to filenames * EvmWalletAddressConvertor: tests * Web3AuthEip4361NonceRepository::consume_nonce(): implement * Web3WalletAuthenticationProvider: verify nonce * PostgresWeb3AuthNonceRepository -> PostgresWeb3AuthEip4361NonceRepository * InMemoryWeb3AuthNonceRepository -> InMemoryWeb3AuthEip4361NonceRepository * SqliteWeb3AuthNonceRepository -> SqliteWeb3AuthEip4361NonceRepository * kamu-cli: register db repos * kamu-cli: move kamu_adapter_auth_web3::register_dependencies() to configure_server_catalog() * Changes after merging * test_login_enabled_methods(): fix * Self-review * Fix typos * clippy fixes * GQL: Account::account_type(): add * Account::prepare_account_name_for_storage(): save checksummed wallet address caseness * odf::AccountID: the type as an enum (initial migration) * AccountID: as_did() -> as_did_odf() * DidPkh: implement * odf::AccountID: tests * Web3WalletAuthenticationProvider::login(): generate did:pkh: account ID * Fixes after merging * test_read_shapefile_geom(): fix test * Web3WalletAuthenticationProvider: tests * Self-review * odf::AccountID::to_stack_string(): implement * odf::AccountID::as_stack_string(): implement * ToStackString: implement * odf::AccountID::as_id_without_did_prefix(): implement * Self-review * Self-review [2] * AccountProvider: introduce * lazy_static: remove dep * GQL: Account::account_provider(): return AccountProvider enum * GQL: Eip4361AuthNonce: scalar * Web3WalletAuthenticationProviderHarness: add signature generation notes * test_signature_verified(): add * kamu-cli: register kamu_adapter_auth_web3 with kamu_adapter_oauth * AccountService: create_account() -> create_password_account() * LoginPasswordAuthProvider::login(): update comment * Web3AuthenticationEip4361Nonce: use regex * opendatafabric-metadata: add "did-pkh" feature * kamu-accounts-repo-tests: remove unused dep * kamu-cli: fix "web-ui" build * GQL: AuthMut::login(): use AccountProvider for "login_method" argument * Unittests fixes * CHANGELOG.md: update * Release (minor): 0.240.0 (#1255) * Update `sqlx` to `0.8.6`, vol.2 (#1222) * sqlx: 0.8.5 * images: sqlx-cli@0.8.5 * CHANGELOG.md: update * sqlx: 0.8.6 * sqlx: 0.8.6 [2] * images/sqlx-cli: read versions from the repo * Hotfix: `web3-wallet` authorization provider: interactive login use case support (Device Flow) (#1257) * Web3WalletAuthenticationProvider: fix device flow * Release (patch): 0.240.1 * Post-merge changes --------- Co-authored-by: Sergii Mikhtoniuk <mikhtoniuk@gmail.com> Co-authored-by: Roman Boiko <roman.bv20@gmail.com>
When creating datasets on user's behalf, ensure naming scheme for the users matches parent account name.
* StackString::make_ascii_lowercase(): implement * GQL: Search::query() * GQL: Search::query(): tests * CHANGELOG.md: update * Update tracing-subscriber * MoleculeMut::project_account_name(): generate lowercase project account name * Add migrations
# Conflicts: # CHANGELOG.md
* Drop archetype inference from schema compatibility option * Update merge crate (#1392) * Update merge crate * Cleanup * aws-lc-rs: unpin version (#1403) * Experiment: GQL: enable Apollo Tracing only based on `x-trace-graphql=1` header (#1402) * GraphqlTracingLayer: implement * kamu-cli: use GraphqlTracingLayer * CHANGELOG.md: update * CHANGELOG.md: PR links after changelog items * Add notes re cheap clones * Fix unstable ordering of dataset entry listings (#1405) * Investigation: potential unstable ordering of dataset entry listings * More ordering fixed * CHANGELOG.md: update --------- Co-authored-by: Sergei Zaychenko <zaychenko.sergei@gmail.com> * Handle iterative ingest process in separate transactions (#1388) * Add per dataset pull transaction * Update tests * Add refresh methods * Update changelog * Move loop to use case * Add transactional static methods * Extend new macro comments * Fix grammar * Pull iteration outside transaction * Fix transform transactional * Make sync service non transactional * Avoid metadata state recalculating * Remove additional cloning * Fix unwrap failure * Added `deliver_timeout` configuration for webhoo config (#1411) * Added deliver_timeout configuration for webhoo config * Fix review comments * v0.250.0 * Remove MacOS amd64 support from installer test * Clean up database sorting (#1413) * Cleanup order by clauses * Clean order by queries * SQLX: avoid using untyped row interfaces like SqliteRow, PgRow (#1414) * Outbox queries refactored * get_events_multi * SqliteFlowTriggerEventStore::stream_all_active_flow_bindings * SqliteDatasetKeyBlockRepository::match_datasets_having_blocks * SqliteDatasetEntryRepository::get_multiple_dataset_entries * PostgresRebacRepository * SQLite AccountRepositry + SQLiteRebacRepository * Remaining cases eliminated * Deps + changelog * Flow process state projection model and GQL API (individual flow badges, dashboard cards) (#1401) Flow process state projection model and GQL API (individual flow badges, dashboard cards): Flow process is a sequence of flow runs, and it's state is automatically projected Flow badges are statuses of flow processes (primary process + associated webhook channels) Flow dashboards are sortable and filterable card lists with pagination support that represent multiple automated flow runs and facilitate effective monitoring and triaging activities for the platform Unaffected with manual launches, only represent automated processes Higher level integration events model, which replaces FlowProgressMessage: Notify about detected flow failure Notify about change of flow process effective state Notify separately about automatic stops (too high failure rate or unrecoverable error) Projection decides on auto-stopping triggers and whether to schedule next periodic flows Flow agent only invokes success propagation on task completion, and does not interfeer with scheduling logic. Introduced FlowEventCompleted event that wraps the flow, and transports outcome + late activation causes Internal event-sourcing projection mechanism within the flow system: Follows changes of 3 original event streams (flow configs, triggers, and runs) Shared event identifiers between events of the source streams Union-based view of merged flow stream (without physical copying) Synchronizes projections based on the merged event stream Postgres: guaranteeing proper event ordering via transaction identifiers tracking utilized LISTEN/NOTIFY mechanism to propagate changes Sqlite: incremental listening timeout approach, no risk of event ordering issues In-memory: using broadcast signals to notify about new events Flow agent tests are now based on the similar projection mechanism instead of query snapshots Introduced abstraction for background agents: API server creates a collection of Tokio tasks instead of directly listing particular agents. Introduced automated outbox wiping procedure in Postgres Optimized database indices after large flow system refactoring Removed obsolete "allPaused" operation from triggers at dataset level Tests: refactoring - extracted common GQL harnesses for all flow tests * v0.251.0 * Store encrypted webhook secrets (#1416) * Store encrypted webhook secrets * Add rust sec tmp ignore * Remove create_secret method * Add rotate webhook secret GQL method * Update changelog * Remove create secret event * Removed obsolete files --------- Co-authored-by: Sergii Mikhtoniuk <mikhtoniuk@gmail.com> Co-authored-by: Roman Boiko <roman.bv20@gmail.com> Co-authored-by: Sergei Zaychenko <zaychenko.sergei@gmail.com> Co-authored-by: Sergei Zaychenko <szaychenko@kamu.dev>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
2nd delivery includes:
moleculeGQL API group