Skip to content

Pyfive implementation #189

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
valeriupredoi merged 155 commits into from
Mar 14, 2025
Merged

Pyfive implementation #189

valeriupredoi merged 155 commits into from
Mar 14, 2025

Conversation

@valeriupredoi
Copy link
Collaborator

@valeriupredoi valeriupredoi commented Mar 8, 2024 •
edited
Loading

@bnlawrence I opened a PR with it for ease of checking the tests 🍺

Merged 14 March 2024 merge commit 4af5c46
Great many thanks to @bnlawrence and @davidhassell for making this happen!

Bryan Lawrence and others added 30 commits February 29, 2024 09:00
A few wrinkles to sort out, some changes to pyfive needed, but this i…
e9f9ee6 
…s a pre-train interim commit
bnl_test works with pyfive. The tests are borked because they're ridd…
eb9e853 
…led with zarr dependencies
Gzip decompression working. Will document issues remaining in #188
0d7bf55
Split compression and filters
d73fcd8
Slight doc improvement in hdf2numcodec
0c4272f
Need to turn off decoding filter pipeline if there is none.
c20b259
Support for contiguous data. Passing lots of tests now.
68e49d0
Remove the file context manager for now
b745d5e
A couple more h5py lurkers and one more bug squashed.
74f8662
@valeriupredoi
removed redundant test file
b628a68
@valeriupredoi
removed redundant test cases
1803e7f
@valeriupredoi
making sure we know Pyfive is run haha
f5a375c
@valeriupredoi
fix bigger data test with native pyfive exception
d1c9282
@valeriupredoi
fix package test
69c3dca
@valeriupredoi
mark lock test as xfail
c50c3d2
toy for testing missing test
09b4b3f
Merge branches 'pyfive' and 'pyfive' of github.com:valeriupredoi/PyAc…
03254dc 
…tiveStorage into pyfive
@valeriupredoi
turn off some printing for now
8a7e35b
@valeriupredoi
starting to understand the test flakiness
cacb94e
Addresses issues with missing data and with thread safety (basically …
ce0479c 
…we need to read the chunk cache before sending things off into the weeds).
Merge branch 'pyfive' of github.com:valeriupredoi/PyActiveStorage int…
39884bc 
…o pyfive
Fixed the regression wrt files with no chunking introduced by thread …
02d6120 
…safety.
Made some changes to Active just so we can use masked data as a stand…
2ffd644 
…alone method which I don't like. We would be better to pull that out and do this differently. I also fixed the disgusting test ...
@valeriupredoi
fix test compression
408558f
@valeriupredoi
fix the other compression test
3294052
@valeriupredoi
fix mocker tests
ca30bb2
@valeriupredoi
minor fix to active
c1b2823
@valeriupredoi
ass test todo
3c1b420
@valeriupredoi
fix last test
589c772
@valeriupredoi
checkout and install pyfive in dev mode
c022040
valeriupredoi and others added 22 commits February 28, 2025 12:37
@valeriupredoi @davidhassell
Update activestorage/active.py
b80862d 
Co-authored-by: David Hassell <davidhassell@users.noreply.github.com>
@valeriupredoi
fix test
ebd54e8
@valeriupredoi
test mock s3 dataset
050bc8f
@valeriupredoi
Merge pull request #241 from NCAS-CMS/new_api_pyfive
5177b06 
Improving API: part 1: functionality for input pyfive.high_level.Dataset
@valeriupredoi
force mamba 2
ead3e60
@valeriupredoi
up codecov action to v5
c588b0d
@valeriupredoi
up checkout version and disable artifacts
93f6362
@valeriupredoi
unrun GHA
412607e
@valeriupredoi
Merge pull request #242 from NCAS-CMS/tlc1
4d075a8 
force mamba 2 and get rid of artifacts
@valeriupredoi
Merge branch 'main' into pyfive
0995d0d
@davidhassell
redcutionist not ready, and tests, fixes
2e54ea1
@davidhassell
merge of pyfive
5298b53
@valeriupredoi
fix tests
c6cc75e
@davidhassell
add test1.nc
0ad9a84
@davidhassell
Merge branch 'axis-subset-2' of github.com:NCAS-CMS/PyActiveStorage i…
b94ed4c 
…nto axis-subset-2
@valeriupredoi
fix s3 test
4ba42da
@valeriupredoi
add missing axis kwarg
259bba1
@valeriupredoi
agh am a doofus
3e725c1
@valeriupredoi
explicitly set storage type kwarg in tests
b474d09
@valeriupredoi
and finally the last test I hope
16ba98b
@valeriupredoi
Merge pull request #248 from NCAS-CMS/axis-subset-2
5d26a97 
Reducing over a subset of axes
@valeriupredoi
Merge branch 'main' into pyfive
8a8de8e
@valeriupredoi valeriupredoi marked this pull request as ready for review March 14, 2025 12:12
@valeriupredoi
Copy link
Collaborator Author

valeriupredoi commented Mar 14, 2025

@bnlawrence @davidhassell I'd like to merge this one into main - we should definitely do that since this is the way forward, and am keeping #186 just as a demonstrator - that branch will not get updated with the new main after we merge this

@valeriupredoi valeriupredoi changed the title [DRAFT] Pyfive Masterpiece Pyfive implementation Mar 14, 2025
@davidhassell davidhassell self-requested a review March 14, 2025 13:30
davidhassell
davidhassell approved these changes Mar 14, 2025
View reviewed changes
Copy link
Collaborator

@davidhassell davidhassell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good for me!

@bnlawrence
Copy link
Collaborator

bnlawrence commented Mar 14, 2025

Likewise. Do it.

@valeriupredoi
Copy link
Collaborator Author

valeriupredoi commented Mar 14, 2025

cheers, gents, big one here 🍺

@valeriupredoi valeriupredoi merged commit 4af5c46 into main Mar 14, 2025
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bnlawrence bnlawrence bnlawrence left review comments

@davidhassell davidhassell davidhassell approved these changes

Assignees

No one assigned

Labels

enhancement New feature or request testing testing duh

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@valeriupredoi @bnlawrence @davidhassell