feat: Zero-Copy Data Transfer with rust-numpy by verdhanyash · Pull Request #73 · etsi-ai/etna

verdhanyash · 2026-02-11T04:38:08Z

Summary

Implements zero-copy data transfer between Python (NumPy) and Rust using rust-numpy's PyReadonlyArray2. This eliminates the pylist_to_vec2 bottleneck that iterated over Python list objects one element at a time, replacing it with direct buffer access to NumPy's contiguous memory.

Closes #17

Changes

Rust Core

lib.rs: Replaced pylist_to_vec2(&Bound<'_, PyList>) with ndarray_to_vec2(PyReadonlyArray2<f32>)
lib.rs: Updated train() and predict() signatures to accept PyReadonlyArray2<'_, f32>
Cargo.toml: Added numpy = 0.27 dependency

Python API

api.py: Passes contiguous float32 NumPy arrays directly to Rust (no .tolist() conversion)
preprocessing.py: Returns NumPy arrays from fit_transform() and transform() instead of .tolist()

Testing

Fixed test_tqdm_progress.py mock patching to work alongside compiled Rust extension
All 28 Rust tests pass
All 38 Python tests pass

Before vs After

Before:

numpy -> .tolist() -> Python list -> PyList -> per-element extract -> Vec<Vec<f32>>

After:

numpy -> PyReadonlyArray2 (zero-copy buffer view) -> bulk row .to_vec() -> Vec<Vec<f32>>

Acceptance Criteria

No manual loops for data conversion in lib.rs
pylist_to_vec2 and PyList completely removed
Training initialization time reduced by >50%% for large arrays
Internal Rust core architecture preserved
All tests pass (28 Rust + 38 Python)

Screenshots

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

github-actions · 2026-02-11T04:40:09Z

Validation Successful!

This pull request has been verified and linked to issue #17. The system is now synchronizing metadata from the referenced issue. Kindly await maintainer review of your changes.

github-actions · 2026-02-11T04:40:09Z

Validation Successful!

This pull request has been verified and linked to issue #17. The system is now synchronizing metadata from the referenced issue. Kindly await maintainer review of your changes.

github-actions · 2026-02-11T04:40:11Z

Thank you for opening this PR! Our automated system is currently verifying the PR requirements.
Internal Discussion: Discord

Copilot

Pull request overview

This PR replaces Python list-based data conversion with numpy array buffer access at the Python-Rust boundary. The implementation uses the rust-numpy crate to accept PyReadonlyArray2<f32> parameters instead of PyList, eliminating the need for element-by-element Python object extraction. The Python preprocessing layer now returns numpy arrays directly, and the API layer ensures contiguous float32 memory layout before passing data to Rust.

Changes:

Added numpy = 0.27 dependency to enable numpy array integration with pyo3
Replaced pylist_to_vec2(PyList) with ndarray_to_vec2(PyReadonlyArray2) for bulk row conversion
Updated Python preprocessing and API to return/pass numpy arrays instead of converting to lists

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
etna_core/Cargo.toml	Added rust-numpy 0.27 dependency for PyReadonlyArray2 support
etna_core/src/lib.rs	Replaced PyList-based conversion with ndarray-based bulk copy; removed PyList import
etna/preprocessing.py	Returns numpy arrays directly instead of calling .tolist()
etna/api.py	Ensures contiguous float32 arrays via np.ascontiguousarray before Rust calls
tests/test_tqdm_progress.py	Fixed mock patching to ensure reloaded module uses mock extension
Cargo.lock	Added transitive dependencies for numpy crate (ndarray, matrixmultiply, num-*, etc.)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

etna_core/src/lib.rs

github-actions · 2026-02-11T04:50:57Z

Validation Successful!

This pull request has been verified and linked to issue #17. The system is now synchronizing metadata from the referenced issue. Kindly await maintainer review of your changes.

github-actions · 2026-02-11T05:32:23Z

Validation Successful!

This pull request has been verified and linked to issue #17. The system is now synchronizing metadata from the referenced issue. Kindly await maintainer review of your changes.

Aamod007

@verdhanyash Good work.

github-actions · 2026-02-12T09:32:46Z

Validation Successful!

This pull request has been verified and linked to issue #17. The system is now synchronizing metadata from the referenced issue. Kindly await maintainer review of your changes.

github-actions · 2026-02-13T03:57:18Z

Validation Successful!

This pull request has been verified and linked to issue #17. The system is now synchronizing metadata from the referenced issue. Kindly await maintainer review of your changes.

github-actions · 2026-02-13T04:00:12Z

Validation Successful!

This pull request has been verified and linked to issue #17. The system is now synchronizing metadata from the referenced issue. Kindly await maintainer review of your changes.

.

github-actions · 2026-02-13T04:07:17Z

Validation Successful!

This pull request has been verified and linked to issue #17. The system is now synchronizing metadata from the referenced issue. Kindly await maintainer review of your changes.

Replace pylist_to_vec2 with numpy::PyReadonlyArray2 for zero-copy buffer access at the Python-Rust bridge. Eliminates per-element Python object extraction and .tolist() overhead. Changes: - etna_core/Cargo.toml: added numpy = 0.27 dependency - etna_core/src/lib.rs: replaced pylist_to_vec2/PyList with ndarray_to_vec2/PyReadonlyArray2 - etna/preprocessing.py: return numpy arrays instead of .tolist() - etna/api.py: pass contiguous float32 arrays directly to Rust - tests/test_tqdm_progress.py: fixed mock patching for compiled extension

Copilot

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

etna_core/src/lib.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

github-actions · 2026-02-13T04:09:35Z

Validation Successful!

This pull request has been verified and linked to issue #17. The system is now synchronizing metadata from the referenced issue. Kindly await maintainer review of your changes.

Aamod007

Great Work @verdhanyash !

github-actions · 2026-02-13T08:59:20Z

This PR has been successfully merged. We appreciate your effort in improving the library and look forward to your future contributions to the Etsi AI ecosystem.

Copilot AI review requested due to automatic review settings February 11, 2026 04:38

github-actions bot requested review from Aamod007, Arsh123344423, Romit23, Satyamgupta2365, SrishtiSonam and debug-soham February 11, 2026 04:40

github-actions bot assigned verdhanyash Feb 11, 2026

github-actions bot added area: bridge Python-Rust interface and PyO3 bindings. Hard Complex architecture or advanced math. performance Speed and memory optimizations. labels Feb 11, 2026

Copilot started reviewing on behalf of verdhanyash February 11, 2026 04:40 View session

Copilot AI reviewed Feb 11, 2026

View reviewed changes

etna_core/src/lib.rs Show resolved Hide resolved

etna_core/src/lib.rs Show resolved Hide resolved

verdhanyash force-pushed the feature/zero-copy-numpy-transfer branch from d685029 to f985953 Compare February 11, 2026 05:32

Aamod007 previously approved these changes Feb 11, 2026

View reviewed changes

github-actions bot requested a review from Aamod007 February 12, 2026 09:32

debug-soham added the SWoC26 Contributions specifically for the Social Winter of Code program. label Feb 12, 2026

verdhanyash force-pushed the feature/zero-copy-numpy-transfer branch from 1c187e2 to 21851d2 Compare February 13, 2026 03:57

Aamod007 requested a review from Copilot February 13, 2026 04:01

Copilot started reviewing on behalf of Aamod007 February 13, 2026 04:02 View session

verdhanyash force-pushed the feature/zero-copy-numpy-transfer branch from 3c76372 to cc1b552 Compare February 13, 2026 04:07

Copilot AI reviewed Feb 13, 2026

View reviewed changes

etna_core/src/lib.rs Show resolved Hide resolved

etna_core/src/lib.rs Outdated Show resolved Hide resolved

Update etna_core/src/lib.rs

611070a

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Aamod007 approved these changes Feb 13, 2026

View reviewed changes

debug-soham approved these changes Feb 13, 2026

View reviewed changes

debug-soham merged commit 2c18033 into etsi-ai:main Feb 13, 2026
2 checks passed

Comments

Conversation

verdhanyash commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Rust Core

Python API

Testing

Before vs After

Acceptance Criteria

Screenshots

Uh oh!

github-actions bot commented Feb 11, 2026

Validation Successful!

Uh oh!

github-actions bot commented Feb 11, 2026

Validation Successful!

Uh oh!

github-actions bot commented Feb 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Feb 11, 2026

Validation Successful!

Uh oh!

github-actions bot commented Feb 11, 2026

Validation Successful!

Uh oh!

Aamod007 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 12, 2026

Validation Successful!

Uh oh!

github-actions bot commented Feb 13, 2026

Validation Successful!

Uh oh!

github-actions bot commented Feb 13, 2026

Validation Successful!

Uh oh!

github-actions bot commented Feb 13, 2026

Validation Successful!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Feb 13, 2026

Validation Successful!

Uh oh!

Aamod007 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Feb 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

verdhanyash commented Feb 11, 2026 •

edited

Loading