Feature/ingest v2 by ag-ramachandran · Pull Request #439 · Azure/azure-kusto-java

ag-ramachandran · 2025-09-09T05:59:32Z

Initial work on the IngestV2 API

Copilot

Pull Request Overview

This PR introduces the initial work on the IngestV2 API, implementing a new Kotlin-based ingestion service alongside the existing Java implementation. The major change is upgrading the project from Java 8 to Java 11 and updating various dependencies to support the new ingest-v2 module.

Project upgraded from Java 8 to Java 11 with dependency version updates
New ingest-v2 module written in Kotlin with modern async capabilities
Centralized dependency management moved to root POM for consistency

Reviewed Changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
pom.xml	Upgraded Java version, updated dependencies, added ingest-v2 module, centralized dependency management
ingest/pom.xml	Removed duplicate dependency management, cleaned up version specifications
data/pom.xml	Removed duplicate dependency management, cleaned up version specifications
ingest-v2/pom.xml	Complete new module configuration with Kotlin, Ktor, and OpenAPI generation
ingest-v2/src/main/resources/openapi.yaml	OpenAPI specification for the new ingest REST API
ingest-v2/src/main/kotlin/...	Core Kotlin implementation files for authentication, retry policies, data sources, and API clients
ingest-v2/src/test/kotlin/...	Unit tests for retry policies and configuration API

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

...main/kotlin/com/microsoft/azure/kusto/ingest/v2/common/auth/AzCliTokenCredentialsProvider.kt

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/source/BlobSource.kt

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/source/LocalSource.kt

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/common/utils/PathUtils.kt

github-actions · 2025-09-09T06:00:40Z

Test Results

530 tests ±0 521 ✅ ±0 8m 16s ⏱️ + 5m 13s
31 suites ±0 9 💤 ±0
31 files ±0 0 ❌ ±0

Results for commit f61924c. ± Comparison against base commit 85e4eec.

♻️ This comment has been updated with latest results.

* added code for wellknown kusto endpoints * addressed review comments

* fixed review comments --------- Co-authored-by: ag-ramachandran <ramacg@microsoft.com>

* * Fix method signature for uploads * * Minor edit to ConfigurationCache to determine refresh interval logic * * Additional tests for preferred upload combinations * * Reformat tests * Add tests for duration

Copilot

Pull request overview

Copilot reviewed 110 out of 116 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...t-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/builders/BaseIngestClientBuilder.kt

AsafMah · 2026-01-28T12:10:13Z

...t-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/auth/endpoints/FastSuffixMatcher.kt

@@ -0,0 +1,119 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+package com.microsoft.azure.kusto.ingest.v2.auth.endpoints


I don't like that we have to duplicate these entire classes.

I know we discussed before dependencies on the data package, but at this point we need to reconsider if it leaves us with this much duplicate code.

...t-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/builders/BaseIngestClientBuilder.kt

AsafMah · 2026-01-28T12:14:00Z

...t-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/builders/BaseIngestClientBuilder.kt

+
+    protected abstract fun self(): T
+
+    fun withAuthentication(credential: TokenCredential): T {


In .net, we have withNoAuthentication, here you don't, and instead throw an error if it's not called.

Either it should be part of the main ctor, or should work like c#

AsafMah · 2026-01-28T12:15:53Z

...t-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/builders/BaseIngestClientBuilder.kt

+    }
+
+    companion object {
+        protected fun normalizeAndCheckEngineUrl(clusterUrl: String): String {


These methods are wrong.

See the java getIngestionEndpoint, we have more rules there for special urls. It's also more efficient without constantly building regexes.

Another reason I think to depend on kusto-data.

samples/src/main/java/ingestv2/QueuedIngestV2.java

AsafMah · 2026-01-28T12:23:09Z

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/client/QueuedIngestClient.kt

+     * sent directly to the Data Management service.
+     */
+    @JvmName("ingestAsync")
+    fun ingestAsyncJava(


So all of these methods are not part of the interface?

So if someone uses the interface the can't use it in java?

AsafMah · 2026-01-28T12:34:02Z

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/uploader/ContainerUploaderBase.kt

+     * Atomic counter for round-robin container selection. Increments on each
+     * upload to distribute load evenly across containers.
+     */
+    private val containerIndexCounter = AtomicInteger(0)


Maybe I'm missing something, but that's not what we want -
This has a counter per uploader.

Here, if uploader A and uploader B (sharing the same configurationcache) upload, they will both use storage 0 first.

The state should be a part of the specific list inside configuration cache - so uploader A will use storage 0, and uploader B will use storage 1. See how we do it in c#

ag-ramachandran requested review from AsafMah, Copilot and tanmaya-panda1 September 9, 2025 05:59

ag-ramachandran had a problem deploying to build September 9, 2025 05:59 — with GitHub Actions Failure

ag-ramachandran had a problem deploying to build September 9, 2025 05:59 — with GitHub Actions Error

Copilot AI reviewed Sep 9, 2025

View reviewed changes

ag-ramachandran had a problem deploying to build September 9, 2025 10:10 — with GitHub Actions Error

ag-ramachandran had a problem deploying to build September 9, 2025 10:10 — with GitHub Actions Failure

ag-ramachandran had a problem deploying to build September 9, 2025 10:10 — with GitHub Actions Error

ag-ramachandran had a problem deploying to build September 9, 2025 16:39 — with GitHub Actions Error

ag-ramachandran had a problem deploying to build September 9, 2025 16:39 — with GitHub Actions Failure

ag-ramachandran had a problem deploying to build September 9, 2025 16:39 — with GitHub Actions Error

ag-ramachandran had a problem deploying to build September 10, 2025 01:24 — with GitHub Actions Error

ag-ramachandran had a problem deploying to build September 10, 2025 01:24 — with GitHub Actions Failure

ag-ramachandran had a problem deploying to build September 10, 2025 01:24 — with GitHub Actions Error

ag-ramachandran temporarily deployed to build September 10, 2025 01:57 — with GitHub Actions Inactive

ag-ramachandran temporarily deployed to build September 10, 2025 12:15 — with GitHub Actions Inactive

ag-ramachandran temporarily deployed to build September 10, 2025 12:20 — with GitHub Actions Inactive

AsafMah marked this pull request as ready for review September 18, 2025 09:12

tanmaya-panda1 temporarily deployed to build January 15, 2026 19:48 — with GitHub Actions Inactive

added code for wellknown kusto endpoints (#456)

8a41876

* added code for wellknown kusto endpoints * addressed review comments

ag-ramachandran temporarily deployed to build January 20, 2026 09:30 — with GitHub Actions Inactive

Users/tanmayapanda/address review comments (#458)

66de6e7

* fixed review comments --------- Co-authored-by: ag-ramachandran <ramacg@microsoft.com>

ag-ramachandran temporarily deployed to build January 20, 2026 11:13 — with GitHub Actions Inactive

addressed review comments (#460)

4c3d8f6

ag-ramachandran temporarily deployed to build January 20, 2026 11:54 — with GitHub Actions Inactive

Users/ramacg/refactor managed uploader (#455)

5f296db

* * Fix method signature for uploads * * Minor edit to ConfigurationCache to determine refresh interval logic * * Additional tests for preferred upload combinations * * Reformat tests * Add tests for duration

ag-ramachandran had a problem deploying to build January 21, 2026 05:03 — with GitHub Actions Error

ag-ramachandran had a problem deploying to build January 21, 2026 05:03 — with GitHub Actions Failure

* Fix JDK version in POM (#461)

b1ec311

ag-ramachandran temporarily deployed to build January 21, 2026 05:16 — with GitHub Actions Inactive

tanmaya-panda1 approved these changes Jan 21, 2026

View reviewed changes

ag-ramachandran requested a review from Copilot January 21, 2026 07:46

Copilot started reviewing on behalf of ag-ramachandran January 21, 2026 07:46 View session

Copilot AI reviewed Jan 21, 2026

View reviewed changes

added private link and removed tuple/Pair (#462)

f61924c

ag-ramachandran temporarily deployed to build January 21, 2026 09:17 — with GitHub Actions Inactive

AsafMah requested changes Jan 28, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/ingest v2#439

Feature/ingest v2#439
ag-ramachandran wants to merge 50 commits intomasterfrom
feature/IngestV2

ag-ramachandran commented Sep 9, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Sep 9, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

AsafMah Jan 28, 2026

Uh oh!

Uh oh!

AsafMah Jan 28, 2026

Uh oh!

AsafMah Jan 28, 2026

Uh oh!

Uh oh!

AsafMah Jan 28, 2026

Uh oh!

AsafMah Jan 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants


		protected abstract fun self(): T

		fun withAuthentication(credential: TokenCredential): T {

Conversation

ag-ramachandran commented Sep 9, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

AsafMah Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AsafMah Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

AsafMah Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AsafMah Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

AsafMah Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions bot commented Sep 9, 2025 •

edited

Loading