Skip to content

Conversation

@kashishmittal55
Copy link
Collaborator

Add support to enable converting from pyarrow to numpy after reading the parquet file and storing it in result queue.
When this is enabled; it will allow us to save time when we use disk cache. This way whatever is being read from the cache is ready to be consumed by downstream petastorm libraries.
This feature is disabled by default

@arushi297
Copy link
Collaborator

LGTM. Do you think we should enable this by default? Or are there any concern/implication for that?

@codecov
Copy link

codecov bot commented Sep 10, 2025

Codecov Report

❌ Patch coverage is 89.47368% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.40%. Comparing base (64cc551) to head (42e7ba2).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
petastorm/arrow_reader_worker.py 89.47% 2 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #815      +/-   ##
==========================================
+ Coverage   86.29%   86.40%   +0.11%     
==========================================
  Files          84       84              
  Lines        5123     5136      +13     
  Branches      804      807       +3     
==========================================
+ Hits         4421     4438      +17     
+ Misses        560      557       -3     
+ Partials      142      141       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@CLAassistant
Copy link

CLAassistant commented Sep 15, 2025

CLA assistant check
All committers have signed the CLA.

@diyu2025 diyu2025 merged commit ec31c47 into uber:master Sep 15, 2025
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants