Skip to content

Users/tanmayapanda/address review comments#458

Merged
ag-ramachandran merged 3 commits intofeature/IngestV2from
users/tanmayapanda/AddressReviewComments
Jan 20, 2026
Merged

Users/tanmayapanda/address review comments#458
ag-ramachandran merged 3 commits intofeature/IngestV2from
users/tanmayapanda/AddressReviewComments

Conversation

@tanmaya-panda1
Copy link
Collaborator

This pull request introduces significant changes to the ingestion client and uploader logic to simplify and clarify multi-source ingestion, enforce stricter type safety, and improve concurrency handling. The main focus is to restrict multi-source ingestion to only support blob sources, remove internal logic for uploading local sources within multi-ingest, and enhance the uploader's parallelism and load distribution.

Multi-source ingestion API and logic changes:

  • Restricts multi-source ingestion (ingestAsync) to only accept BlobSource objects, both in the interface and implementation, and updates all related documentation to clarify that local sources must be uploaded separately before ingestion. This prevents partial failure scenarios and enforces a clearer, safer workflow. [1] [2] [3] [4]
  • Removes the internal method for uploading local sources (uploadLocalSourcesAsync) from QueuedIngestClient, making it the caller's responsibility to upload local files/streams before calling multi-ingest. [1] [2]

Error handling and messaging improvements:

  • Updates error messages in IngestClientException to use the message parameter for clarity and consistency. [1] [2]

Uploader concurrency and load balancing enhancements:

  • Switches the container selection logic in ContainerUploaderBase from random to round-robin using an atomic counter, ensuring more even load distribution across containers during uploads and retries. [1] [2] [3]
  • Improves parallel upload logic to use a semaphore for true streaming parallelism, allowing new uploads to start as soon as slots are available, rather than processing in fixed-size chunks.

Java interop and API usability:

  • Adds overloaded static factory methods to DefaultConfigurationCache for easier creation from Java, supporting both minimal and full-parameter configurations.
  • Adds @JvmOverloads constructors to FileSource and StreamSource for better Java interoperability and sensible defaults. [1] [2]

These changes collectively make the ingestion API safer, simpler, and more robust, especially in multi-source scenarios.### Added

Changed

Fixed

@github-actions
Copy link

github-actions bot commented Jan 15, 2026

Test Results

530 tests  ±0   521 ✅ ±0   7m 58s ⏱️ -12s
 31 suites ±0     9 💤 ±0 
 31 files   ±0     0 ❌ ±0 

Results for commit 428eb23. ± Comparison against base commit ce36f64.

♻️ This comment has been updated with latest results.

* Atomic counter for round-robin container selection. Increments on each
* upload to distribute load evenly across containers.
*/
private val containerIndexCounter = AtomicInteger(0)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AtomicLong perhaps ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought of so but AtomicLong seems to be an overkill perhaps, since AtomicInteger has already around 2.1 billion positive values. Added Math.floorMod to handle the negative values incase overflow occurs.

@ag-ramachandran ag-ramachandran merged commit 66de6e7 into feature/IngestV2 Jan 20, 2026
5 checks passed
@ag-ramachandran ag-ramachandran deleted the users/tanmayapanda/AddressReviewComments branch January 20, 2026 11:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants