tensorstore crashing on anonymous s3 open (macos arch -x86_64, python 3.12)

[fcollman]

I'm getting a crash on my mac with the latest versions of tensorstore trying to access an zarr in a public bucket.

test_tensorstore.py

# /// script
# requires-python = ">=3.10"
# dependencies = [
#   "tensorstore==0.1.74",
# ]
# ///
import tensorstore as ts

print(ts.__file__)

spec = {
    "driver": "zarr",
    "kvstore": "s3://ome-zarr-neuroglancer/1250417704.zarr/5/",
    "context": {"aws_credentials": {"type": "anonymous"}},
}
z = ts.open(spec).result()
print(z)

% uname -a
Darwin OSXLT5XFXF3 23.6.0 Darwin Kernel Version 23.6.0: Mon Jul 29 21:14:30 PDT 2024; root:xnu-10063.141.2~1/RELEASE_ARM64_T6000 arm64

Attached is a crash report generated with

 uv run --python 3.12 --with tensorstore===0.1.74 python -i test_tensorstore.py

Python-2025-04-09-132325.txt

Seems likely related to #227

[laramiel]

Sigh. The same script runs correctly on my mac, and if it's going to be hard for me to reproduce any os-version related issues as I have minimum os requirements:

$ uname -a
Darwin lar-macbookpro3.roam.internal 24.4.0 Darwin Kernel Version 24.4.0: Fri Apr 11 18:33:47 PDT 2025; root:xnu-11417.101.15~117/RELEASE_ARM64_T6000 arm64

Can you try to reproduce this building from source?
https://google.github.io/tensorstore/installation.html#python-api-from-source

---

FWIW, this is crashing on initialization of a static linked list data structure, which seems quite odd. I did move some of the initialization code in ebb8815, which should be included by 0.1.74, but the stack trace is using the older codepath. That seems suspicious to me.

Code path:

void aws_thread_initialize_thread_management(void) {
    aws_linked_list_init(&s_pending_join_managed_threads);
}

/**
 * Initializes the list. List will be empty after this call.
 */
AWS_STATIC_IMPL void aws_linked_list_init(struct aws_linked_list *list) {
    AWS_PRECONDITION(list);
    list->head.next = &list->tail;
    list->head.prev = NULL;
    list->tail.prev = &list->head;
    list->tail.next = NULL;
    AWS_POSTCONDITION(aws_linked_list_is_valid(list));
    AWS_POSTCONDITION(aws_linked_list_empty(list));
}

[laramiel]

Can you help me ensure we're running the same version? I updated the script to print the ts.__file__
If you run sha256sum on it, replacing __init__.py with *, something like the following, what do you get?

If you get anything other than the 0.1.74 version can you retry after running uv cache clean?

---

Version 0.1.74 looks like this:

$ sha256sum ~/.cache/uv/archive-v0/CN-tTXciwtbwgU6d5X8BA/lib/python3.12/site-packages/tensorstore/*
35d349208f6201cec30a7dfdc9efc979938bfc1c8ecb29f51dcf646315e5434d  /Users/lar/.cache/uv/archive-v0/CN-tTXciwtbwgU6d5X8BA/lib/python3.12/site-packages/tensorstore/__init__.py
46ba64a60466d8d443da60ca3f182b84ecef0fef905de22d5aa131aad0600ccf  /Users/lar/.cache/uv/archive-v0/CN-tTXciwtbwgU6d5X8BA/lib/python3.12/site-packages/tensorstore/_tensorstore.cpython-312-darwin.so
13928d281b0d1b18f182dad9cc2933e4a5d366db396c5b2238241504225a2362  /Users/lar/.cache/uv/archive-v0/CN-tTXciwtbwgU6d5X8BA/lib/python3.12/site-packages/tensorstore/LICENSE

Also, if you have xcode installed you can verify the UUID:

$ dwarfdump -u ~/.cache/uv/archive-v0/CN-tTXciwtbwgU6d5X8BA/lib/python3.12/site-packages/tensorstore/_tensorstore.cpython-312-darwin.so
UUID: 13E23557-DD56-39AF-B885-98B4F27EF3D3 (arm64) /Users/lar/.cache/uv/archive-v0/CN-tTXciwtbwgU6d5X8BA/lib/python3.12/site-packages/tensorstore/_tensorstore.cpython-312-darwin.so

---

Version 0.1.73 looks like this:

$ sha256sum /Users/lar/.cache/uv/archive-v0/nbbIMncT_3jRFc7Ig9784/lib/python3.12/site-packages/tensorstore/*
35d349208f6201cec30a7dfdc9efc979938bfc1c8ecb29f51dcf646315e5434d  /Users/lar/.cache/uv/archive-v0/nbbIMncT_3jRFc7Ig9784/lib/python3.12/site-packages/tensorstore/__init__.py
c998180766a04f8cfb8883415940aa77bc43bf8f6da5e1dec0637822b5637b37  /Users/lar/.cache/uv/archive-v0/nbbIMncT_3jRFc7Ig9784/lib/python3.12/site-packages/tensorstore/_tensorstore.cpython-312-darwin.so
13928d281b0d1b18f182dad9cc2933e4a5d366db396c5b2238241504225a2362  /Users/lar/.cache/uv/archive-v0/nbbIMncT_3jRFc7Ig9784/lib/python3.12/site-packages/tensorstore/LICENSE

$ dwarfdump -u /Users/lar/.cache/uv/archive-v0/nbbIMncT_3jRFc7Ig9784/lib/python3.12/site-packages/tensorstore/_tensorstore.cpython-312-darwin.so
UUID: 2063F8EC-EFE4-31D7-BEC6-E0E4FB4EDF4B (arm64) /Users/lar/.cache/uv/archive-v0/nbbIMncT_3jRFc7Ig9784/lib/python3.12/site-packages/tensorstore/_tensorstore.cpython-312-darwin.so

[fcollman]

I tried to reproduce this, but i realize now the crash is happening on import

import tensorstore as ts

print("hi", flush=True)
print(ts.__file__, flush=True)

prints nothing, and generated the attached crash report.

crash_report.txt

[fcollman]

13928d281b0d1b18f182dad9cc2933e4a5d366db396c5b2238241504225a2362  LICENSE
35d349208f6201cec30a7dfdc9efc979938bfc1c8ecb29f51dcf646315e5434d  __init__.py
f0acc29d5215a34a00ab965e5084eb2eb1c824b22bb58ea3cc4c8b26d303784a  _tensorstore.cpython-312-darwin.so
(caveclient) (base) forrestc@OSXLT5XFXF3 tensorstore % pwd
/Users/forrestc/.cache/uv/wheels-v5/pypi/tensorstore/0.1.74-cp312-cp312-macosx_10_14_x86_64/tensorstore

looks like i don't have precisely the same binary as you do

[fcollman]

(caveclient) (base) forrestc@OSXLT5XFXF3 tensorstore % dwarfdump -u _tensorstore.cpython-312-darwin.so 
UUID: F9BA76AF-E97A-387F-8B38-F0AA44DCADEF (x86_64) _tensorstore.cpython-312-darwin.so

I think this is because i am running an x86_64 version of python, mostly because i got this computer before there were reliably arm64 wheels available for everything and I didn't want to deal with compiling everything from scratch myself.

[fcollman]

i can confirm running on a different macbook

% uname -a
Darwin OSXLT2YDL6F 24.5.0 Darwin Kernel Version 24.5.0: Tue Apr 22 19:53:27 PDT 2025; root:xnu-11417.121.6~2/RELEASE_ARM64_T6041 arm64

in which i get the same hashes as you

(base) forrestc@OSXLT2YDL6F tensorstorebug % sha256sum ~/.cache/uv/wheels-v5/pypi/tensorstore/0.1.74-cp312-cp312-macosx_11_0_arm64/tensorstore/*
35d349208f6201cec30a7dfdc9efc979938bfc1c8ecb29f51dcf646315e5434d  /Users/forrestc/.cache/uv/wheels-v5/pypi/tensorstore/0.1.74-cp312-cp312-macosx_11_0_arm64/tensorstore/__init__.py
46ba64a60466d8d443da60ca3f182b84ecef0fef905de22d5aa131aad0600ccf  /Users/forrestc/.cache/uv/wheels-v5/pypi/tensorstore/0.1.74-cp312-cp312-macosx_11_0_arm64/tensorstore/_tensorstore.cpython-312-darwin.so
13928d281b0d1b18f182dad9cc2933e4a5d366db396c5b2238241504225a2362  /Users/forrestc/.cache/uv/wheels-v5/pypi/tensorstore/0.1.74-cp312-cp312-macosx_11_0_arm64/tensorstore/LICENSE

that things work as expected.

i can also confirm that running on the older mac, using an arm64 python 11 that things worked fine with 0.1.74.

Put together, I think this suggests that the problem is running tensorstore compiled for x86 on a macbook in x86 emulation mode. I don't think this is too crazy a use case to try to cover, and i don't know if there is something that can be done to fix it or at least warn users why things are breaking when they do.

[laramiel]

Ok, that's the stack trace that I would expect from 0.1.74. Yeah, x86 vs arm.

0   _tensorstore.cpython-312-darwin.so	       0x111d86619 aws_thread_initialize_thread_management + 25
1   _tensorstore.cpython-312-darwin.so	       0x111d7ef1d aws_common_library_init + 61
2   _tensorstore.cpython-312-darwin.so	       0x111d2c183 absl::lts_20240722::NoDestructor<tensorstore::internal_aws::(anonymous namespace)::AwsApi>::NoDestructor<0>() + 67
3   _tensorstore.cpython-312-darwin.so	       0x111d2bda4 tensorstore::internal_aws::GetAwsAllocator() + 52
4   _tensorstore.cpython-312-darwin.so	       0x111cfaefd tensorstore::(anonymous namespace)::S3KeyValueStoreSpec::DoOpen() const + 45

Can you post sysctl -a | grep cpu

Edit. I haven't run in x86 emulation mode. Do you have a command for that, otherwise I'll try and dig it up.

[fcollman]

(base) forrestc@OSXLT5XFXF3 tensorstorebug % sysctl -a | grep cpu
kern.sched_rt_avoid_cpu0: 0
kern.cpu_checkin_interval: 5000
hw.ncpu: 10
hw.activecpu: 10
hw.perflevel0.physicalcpu: 8
hw.perflevel0.physicalcpu_max: 8
hw.perflevel0.logicalcpu: 8
hw.perflevel0.logicalcpu_max: 8
hw.perflevel0.cpusperl2: 4
hw.perflevel1.physicalcpu: 2
hw.perflevel1.physicalcpu_max: 2
hw.perflevel1.logicalcpu: 2
hw.perflevel1.logicalcpu_max: 2
hw.perflevel1.cpusperl2: 2
hw.physicalcpu: 10
hw.physicalcpu_max: 10
hw.logicalcpu: 10
hw.logicalcpu_max: 10
hw.cputype: 16777228
hw.cpusubtype: 2
hw.cpu64bit_capable: 1
hw.cpufamily: 458787763
hw.cpusubfamily: 5
machdep.cpu.cores_per_package: 10
machdep.cpu.core_count: 10
machdep.cpu.logical_per_package: 10
machdep.cpu.thread_count: 10
machdep.cpu.brand_string: Apple M1 Max

Here's from the machine that i have experienced the crash... i have a newer m4 mac that i just got that i installed arm64 anaconda as the base version.

I don't have a specific command, i generally have installed anaconda and there you can choose between the x86 and arm64 versions, and all subsequent versions will just work. I think the mac will automatically run any x86 compiled executable using emulation mode, so there isn't anything to do other than use python/pip binaries that are x86 and they will install x86 versions of libraries in the python directories for that environment automatically.

[laramiel]

Ok, so I can get close, but python3.10 right now:

$ arch -x86_64 python3 -m venv venv_x86
$ source venv_x86/bin/activate
$ arch -x86_64 python3 -m pip install tensorstore
$ arch -x86_64 python3
Python 3.10.6 (v3.10.6:9c7b4bd164, Aug  1 2022, 17:13:48) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorstore as ts
>>> print(ts.__file__)
/Users/lar/venv_x86/lib/python3.10/site-packages/tensorstore/__init__.py
>>> exit()

$ sha256sum /Users/lar/venv_x86/lib/python3.10/site-packages/tensorstore/*
35d349208f6201cec30a7dfdc9efc979938bfc1c8ecb29f51dcf646315e5434d  /Users/lar/venv_x86/lib/python3.10/site-packages/tensorstore/__init__.py
sha256sum: /Users/lar/venv_x86/lib/python3.10/site-packages/tensorstore/__pycache__: Is a directory
04e6c7b3d2d370b5f2925d1fdb0b2cd4958a00fc99804f5ff560a93e697137c7  /Users/lar/venv_x86/lib/python3.10/site-packages/tensorstore/_tensorstore.cpython-310-darwin.so
13928d281b0d1b18f182dad9cc2933e4a5d366db396c5b2238241504225a2362  /Users/lar/venv_x86/lib/python3.10/site-packages/tensorstore/LICENSE

$ dwarfdump -u /Users/lar/venv_x86/lib/python3.10/site-packages/tensorstore/_tensorstore.cpython-310-darwin.so
UUID: 4AACF774-D574-3BBA-A666-2B883273C955 (x86_64) /Users/lar/venv_x86/lib/python3.10/site-packages/tensorstore/_tensorstore.cpython-310-darwin.so

---

Ok, I can install python 3.12 and use python3.12-intel64

$ python3.12-intel64
Python 3.12.10 (v3.12.10:0cc81280367, Apr  8 2025, 08:47:00) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorstore as ts
>>> print(ts.__file__)
/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/tensorstore/__init__.py

$ dwarfdump -u /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/tensorstore/_tensorstore.cpython-312-darwin.so
UUID: F9BA76AF-E97A-387F-8B38-F0AA44DCADEF (x86_64) /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/tensorstore/_tensorstore.cpython-312-darwin.so

$ sha256sum /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/tensorstore/_tensorstore.cpython-312-darwin.so
f0acc29d5215a34a00ab965e5084eb2eb1c824b22bb58ea3cc4c8b26d303784a  /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/tensorstore/_tensorstore.cpython-312-darwin.so

That seems to match, but no crash. What's the build version output of your python binary?

hw.ncpu: 10
hw.cputype: 16777228
hw.cpusubtype: 2
hw.cpu64bit_capable: 1
hw.cpufamily: 458787763
hw.cpusubfamily: 4
machdep.cpu.cores_per_package: 10
machdep.cpu.core_count: 10
machdep.cpu.logical_per_package: 10
machdep.cpu.thread_count: 10
machdep.cpu.brand_string: Apple M1 Pro

[fcollman]

tensorstorebug % uv run --python 3.12 --with tensorstore===0.1.74 python -i
Python 3.12.10 (main, Apr 8 2025, 11:35:47) [Clang 16.0.0 (clang-1600.0.26.6)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

[fcollman]

so looks like we have narrowed down the problem to the precise kind of python binary that I have installed.. fun times.

[laramiel]

As I see it, the possibilities are:

• Python version. You can try to download the python 3.12 from https://www.python.org/downloads/macos/ and see if that version works for you. Our builds are the same day, but apparently different clang versions.

• Os version. Specifically something related to the rosetta2 x86 translation. The error code is 4, which is bad instruction.

• Some other third thing. :)