Revisit fsspec/boto3 dependency declarations #496
Description
Context
Lengthy comment above the following lines (pasted below for readability):
Lines 140 to 141 in 5a31a64
We use fsspec's S3 support, which has a runtime dep on s3fs. s3fs itself requires aiobotocore, which in turn requires very specific versions of botocore (because aiobotocore is a giant monkey-patch).
We also use boto3, which also requires botocore, usually with minimum versions closely matching the boto3 version (they're released in near lock step).
If we declare a dep on boto3 directly, this leads to conflicts during dependency resolution when a newer boto3 (from our declaration here) requires a newer botocore than is supported by s3fs → aiobotocore's declarations.
Resolve the issue by using a specially-provided package extra from s3fs (first introduced with 2021.4.0, removed in 2025.12.0) which causes them to declare an explicit dependency on aiobotocore's specially-provided package extra on boto3 so that dependency resolver can figure it out properly.
Note that the upper limit is not future-proof and will likely cause issues down the road. There may be a better combination to use here, but that needs extra digging.
More background:
- Dependency on aiobotocore renders packaging s3fs no longer possible fsspec/s3fs#357
- [batch] Unable to submit builds when installed with latest aiobotocore #133
- Directly specify optional dependencies
awscliandboto3fsspec/s3fs#994
What a mess.
Avoiding 2023.9.1 due to change in auto_mkdir parameter in fsspec/filesystem_spec#1358 that causes the error described in fsspec/s3fs#790
Possible solutions
From @joverlee521 in #495 (comment):
Reading through the upstream issue aio-libs/aiobotocore#1424, modern pip seems to be able to resolve s3fs and boto3 without issues (source). Maybe we can remove this whole mess and just declare s3fs and boto3 dependencies separately and let pip handle it in the future.