fix(codec): handle multi-dimensional ndarrays with Arrow encoding

[bbopen]

Summary

Fixes Arrow encoding failure for multi-dimensional ndarrays (e.g., torch tensors) by implementing a flatten-on-encode, reshape-on-decode strategy.

Problem

PyArrow's pa.array() only handles 1D arrays. Multi-dimensional arrays like 2D torch tensors failed with:

pyarrow.lib.ArrowInvalid: only handle 1-dimensional arrays

This was causing the codec-suite CI to fail.

Solution

Python Side

• Flatten multi-dimensional arrays before pa.array()
• Include original shape in envelope for JS-side reconstruction

JavaScript Side

• Add reshapeArray() utility for reconstructing multi-dim arrays
• Update ndarray decoder to reshape after Arrow decode when shape.length > 1

Why This Approach

Approach	Pros	Cons
Flatten + Reshape ✅	Works today, maintains Arrow efficiency	Extra reshape step
FixedShapeTensorArray	Arrow-native	JS doesn't support it
JSON fallback	Simple	~10x larger payloads

Test Results

✓ serializes scipy sparse matrices
✓ serializes torch tensors        # Was failing, now passes with Arrow
✓ serializes sklearn estimators
✓ handles 1D arrays (no reshape needed)
✓ handles 3D arrays
✓ handles 3D torch tensors with Arrow encoding
✓ handles single-element arrays
✓ handles single-element multi-dimensional arrays
✓ preserves dtype for float arrays
✓ handles 4D tensors (image-like batches)

Test Plan

• Torch tensor test passes with Arrow encoding (no JSON fallback)
• 2D tensor [[1,2],[3,4]] round-trips correctly with shape [2,2]
• 1D arrays pass through without reshape
• 3D arrays (2x3x4) flatten and reshape correctly
• 4D tensors (image-like 2x2x2x2) handled correctly
• Single-element and single-element multi-dim arrays work
• Float dtypes preserved
• Full test suite passes (1232 tests)
• Build succeeds

Closes #161

🤖 Generated with Claude Code

[coderabbitai]

Warning

Rate limit exceeded

@bbopen has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 2 minutes and 25 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

📥 Commits

Reviewing files that changed from the base of the PR and between 2909293 and 3723f05.

📒 Files selected for processing (3)

• runtime/python_bridge.py
• src/utils/codec.ts
• test/runtime_codec_scientific.test.ts

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[bbopen]

lgtm