feat: Add support for SOURCE packages by toprinse · Pull Request #244 · Quantco/pixi-pack

toprinse · 2025-11-24T14:27:24Z

Motivation

This PR adds support for building packages from source, a feature that is currently missing. See issue #80 for background and motivation.

When a local package is found in the pixi.lock of the main package, start a pixi-build command to build the local dependencies and add them as injected packages in the environment.tar

Changes

Introduces a pixi-build command to automatically build local packages from the main pixi.lock
Adding integration test to demonstrate the the added code is working as expected.

feat: try to add intergation tests

fix: remove unused fixture fix: fix typo fix: fix typo fix: fix typo fix: fix test on arm-64, osx and win builds fix: fix test on arm-64, osx and win builds fix: add valid checksum for other architectures fix: fix test using correct checksum fix: fix typo error fix: fix typo

src/pack.rs

fix: fix typo fix: fix typo

pavelzw

Looks very solid already, thanks!
I have a couple of comments

examples/local-build/local-pkg/pixi.toml

pavelzw · 2025-11-30T09:51:25Z

src/pack.rs

+                let pkg_dir = base_dir.join(match &source_data.location {
+                    UrlOrPath::Path(p) => p.to_path().to_string(),
+                    UrlOrPath::Url(u) => {
+                        anyhow::bail!("Unexpected URL for local package source: {}", u)


Is this a thing that will be supported eventually @Hofer-Julian? Depending on other pixi projects from git is also not supported yet, right?

src/pack.rs

tests/integration_test.rs

src/pack.rs

fix: fix typo for ci precommit fix: fix typo for ci precommit fix: fix typo for ci precommit

toprinse · 2025-12-01T12:06:15Z

Thanks @pavelzw ! So I made the requested changes, could you review the new version and tell me if there are some changes that are not relevant or things I could have done better ?

examples/local-build/pixi.toml

pavelzw · 2025-12-01T17:29:36Z

src/pack.rs

+        .arg(output_dir)
+        .arg("--build-platform")
+        .arg(platform)
+        .current_dir(manifest_dir);


with things like PIXI_PROJECT_ROOT (set by pixi shell) i think it's safer to explicitly use pixi build --path

also, another thing i noticed:

❯ cd examples/local-build/ ❯ cargo run --bin pixi-pack -- Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.99s Running `/Users/pavel/projects/pixi-pack/target/debug/pixi-pack` ⠁ Building local-build-main-pkg-0.1.0-hbf21a9e_0.conda ❌ Build failed with exit code: Some(1) Command: pixi build --output-dir /var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/.tmpZQnitN --build-platform osx-arm64 (in /Users/pavel/projects/pixi-pack/examples) 📤 stderr: Error: × the source directory '/Users/pavel/projects/pixi-pack/examples', does not contain a supported manifest help: Ensure that the source directory contains a valid pixi.toml, pyproject.toml, recipe.yaml, package.xml or mojoproject.toml file. Error: Failed to build package in /Users/pavel/projects/pixi-pack/examples ❯ cargo run --bin pixi-pack -- pixi.toml Compiling pixi-pack v0.7.5 (/Users/pavel/projects/pixi-pack) Finished `dev` profile [unoptimized + debuginfo] target(s) in 8.78s Running `/Users/pavel/projects/pixi-pack/target/debug/pixi-pack pixi.toml` ✅ Built local-build-main-pkg-0.1.0-hbf21a9e_0.conda ✅ Built local-build-local-pkg-0.1.0-hbf21a9e_0.conda ✅ Built local-build-local-pkg-2-0.1.0-hbf21a9e_0.conda ⏳ Downloading 14 packages... 📦 Created pack at /Users/pavel/projects/pixi-pack/examples/local-build/environment.tar with size 7.70 MiB.

something is still wrong with the manifest path...

I updated the base_dir logic to correctly handle manifest paths when building source packages. Previously, when options.manifest_path was a directory (not a file), the code incorrectly used its parent directory as the base, causing build failures with paths like /examples/./pixi.toml. Now it properly uses the manifest directory itself as the base when it's a directory. Also added canonicalize() to resolve relative paths (like ./) and ensure the manifest file exists before attempting to build.

oprinsen@lappc-p1138 local-build ⌥ (add_source_pkg_handling) cargo run --bin pixi-pack -- Compiling pixi-pack v0.7.5 (/home/oprinsen/pixi-pack) Finished `dev` profile [unoptimized + debuginfo] target(s) in 3.62s Running `/home/oprinsen/pixi-pack/target/debug/pixi-pack` DEBUG: options.manifest_path = /home/oprinsen/pixi-pack/examples/local-build DEBUG: source_data.location = Path(Unix(Utf8PathBuf { _encoding: "unix", inner: "." })) DEBUG: base_dir = /home/oprinsen/pixi-pack/examples/local-build ✅ Built local-build-main-pkg-0.1.0-hbf21a9e_0.conda DEBUG: options.manifest_path = /home/oprinsen/pixi-pack/examples/local-build DEBUG: source_data.location = Path(Unix(Utf8PathBuf { _encoding: "unix", inner: "local-pkg" })) DEBUG: base_dir = /home/oprinsen/pixi-pack/examples/local-build ✅ Built local-build-local-pkg-0.1.0-hbf21a9e_0.conda DEBUG: options.manifest_path = /home/oprinsen/pixi-pack/examples/local-build DEBUG: source_data.location = Path(Unix(Utf8PathBuf { _encoding: "unix", inner: "local-pkg2" })) DEBUG: base_dir = /home/oprinsen/pixi-pack/examples/local-build ✅ Built local-build-local-pkg-2-0.1.0-hbf21a9e_0.conda ⏳ Downloading 21 packages... 📦 Created pack at /home/oprinsen/pixi-pack/examples/local-build/environment.tar with size 15.28 MiB. oprinsen@lappc-p1138 local-build ⌥ (add_source_pkg_handling)

src/pack.rs

pavelzw · 2025-12-01T17:50:08Z

pushed some adjustments to your branch; after the manifest dir fixes, this is good to merge 👍🏻

fix: fix typo

toprinse · 2025-12-08T16:05:53Z

Hello @pavelzw, I saw that the tests have failed after merging main into my branch. Is the issue linked to the hash or the build_version of the generated packages which has changed after merging the two branches ? I don't really know how to make the test reliable, as we hardcoded the hash and version in the tests, and I don't really know how to do it in another way. If I can do anything, just let me know !

pavelzw · 2025-12-08T18:46:20Z

yeah this looks really weird to me. sounds like a bug in pixi tbh.
i will create a minimal reproducer and see what the pixi people think of this one

toprinse · 2025-12-18T08:19:14Z

Hello @pavelzw.

So I looked into the issue of the regenerated hashes and it seems that recently the cmake build backend used in the tests has been updated and the hash generation might have been impacted. I'm not sure about how it impacted the tests and it's still unclear to me but for now I added the new generated hashes using the cmake build backend 0.3.5 (maybe it could be intersting to pin the build backend version in the test to avoid new strange behaviors).

Another idea could be to use match patterns instead of the exact version of the package containing the hashes. Instead of looking for conda-meta/local-build-main-pkg-0.1.0-hb0f4dca_0.json we could also use something like starts_with("local-build-main-pkg-0.1.0-") inside the conda-meta directory.

I also updated the pixi version to use 0.61.0 so the API of pixi build command has changed a bit in pack.rs

pavelzw · 2025-12-18T08:34:39Z

Sorry for not coming back to you, lost track of this pr. One thing I'm wondering is why the build string from the pixi.lock didn't match the build string of the built package 🤔 theoretically the lockfile should always match the package exactly and thus represent the correct build string already? Maybe I'm missing something, will investigate later

In general, i think pinning the build backend makes sense in this case 👍🏻

I think it would be nice if we even persisted the build backends and host+build environments used in pixi.lock (optionally): prefix-dev/pixi#5161

toprinse · 2025-12-18T09:20:00Z

Sorry for not coming back to you, lost track of this pr. One thing I'm wondering is why the build string from the pixi.lock didn't match the build string of the built package 🤔 theoretically the lockfile should always match the package exactly and thus represent the correct build string already? Maybe I'm missing something, will investigate later

In general, i think pinning the build backend makes sense in this case 👍🏻

I think it would be nice if we even persisted the build backends and host+build environments used in pixi.lock (optionally): prefix-dev/pixi#5161

No problem, I was also really busy!

I can pin the build backend version in the tests to the latest version (0.3.5).

Regarding the build string mismatch issue, I'm not sure what approach you'd prefer: should we wait for pixi to provide a clear fix for this issue, or would you prefer to merge the PR as-is and address the build string inconsistency once pixi implements the changes mentioned in the issue you linked?

pavelzw · 2025-12-18T09:40:25Z

I can pin the build backend version in the tests to the latest version (0.3.5).

sounds good, maybe also link https://github.com/prefix-dev/pixi/issues/5161 next to the pin.

Regarding the build string mismatch issue

i would like to understand the issue a bit better and at least have an issue to point to. i'll look into creating a minimal reproducer for this problem and let's see what the prefix-dev team has to say

toprinse · 2025-12-18T09:54:39Z

I can pin the build backend version in the tests to the latest version (0.3.5).

sounds good, maybe also link https://github.com/prefix-dev/pixi/issues/5161 next to the pin.

Regarding the build string mismatch issue

i would like to understand the issue a bit better and at least have an issue to point to. i'll look into creating a minimal reproducer for this problem and let's see what the prefix-dev team has to say

Sounds good! I will pin the version and follow the issue tracking on pixi side. I also tried to do a small reproducer but I did not manage to do something showing the exact same issue.

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

regenerate lock file to use pixi 0.61.0 fix: pin the build backend issue

vincent-pollet · 2026-02-05T13:48:07Z

Hello! May I kindly ask what is the status about this PR? Anything that we could help with?

toprinse requested review from delsner and pavelzw as code owners November 24, 2025 14:27

toprinse changed the title ~~feat : Add support for SOURCE packages~~ feat: Add support for SOURCE packages Nov 24, 2025

github-actions bot added the enhancement New feature or request label Nov 24, 2025

feat: update pixi-pack to be able to pack packages from source

a532262

feat: try to add intergation tests

toprinse force-pushed the add_source_pkg_handling branch from e214536 to faf6c1d Compare November 24, 2025 16:01

toprinse force-pushed the add_source_pkg_handling branch from 6abfcac to a9b0110 Compare November 25, 2025 07:49

delsner requested changes Nov 28, 2025

View reviewed changes

src/pack.rs Outdated Show resolved Hide resolved

src/pack.rs Outdated Show resolved Hide resolved

src/pack.rs Outdated Show resolved Hide resolved

fix: remove recursive search and use packages from lock file

0847e0d

fix: fix typo fix: fix typo

toprinse force-pushed the add_source_pkg_handling branch from abf424a to 0847e0d Compare November 29, 2025 13:22

toprinse requested a review from delsner November 29, 2025 13:22

pavelzw reviewed Nov 30, 2025

View reviewed changes

src/pack.rs Outdated Show resolved Hide resolved

fix: fix after review

1377324

fix: fix typo for ci precommit fix: fix typo for ci precommit fix: fix typo for ci precommit

toprinse force-pushed the add_source_pkg_handling branch from a8e2dbc to 1377324 Compare December 1, 2025 12:04

pavelzw added 3 commits December 1, 2025 18:42

refactor local-build structure

482bd9e

explicitly drop

8d7d436

reword

4f94186

pavelzw reviewed Dec 1, 2025

View reviewed changes

fix: fix manifest_dir

177e276

fix: fix typo

toprinse force-pushed the add_source_pkg_handling branch from 6a65aed to 177e276 Compare December 2, 2025 11:04

pavelzw mentioned this pull request Dec 18, 2025

pixi-build used dependencies (i.e. backend + host and build envs) should be persisted in pixi.lock prefix-dev/pixi#5161

Open

pavelzw and others added 4 commits December 18, 2025 11:09

chore: Update shasums after release (Quantco#242)

456da43

ci: bump the gh-actions group with 5 updates (Quantco#247)

3506c08

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

build: Bump rattler, use uv from crates.io (Quantco#243)

2f23e6e

fix: fix tests to match pixi 0.6.1

bea3f62

regenerate lock file to use pixi 0.61.0 fix: pin the build backend issue

toprinse force-pushed the add_source_pkg_handling branch from 9301789 to bea3f62 Compare December 18, 2025 10:18

toprinse and others added 2 commits December 18, 2025 11:33

Merge branch 'main' into add_source_pkg_handling

b6b2895

fix: fix cargo lock

0384d5c

Comments

Conversation

toprinse commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pavelzw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pavelzw Nov 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

toprinse commented Dec 1, 2025

Uh oh!

Uh oh!

pavelzw Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

pavelzw Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

toprinse Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pavelzw commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

toprinse commented Dec 8, 2025

Uh oh!

pavelzw commented Dec 8, 2025

Uh oh!

toprinse commented Dec 18, 2025

Uh oh!

pavelzw commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

toprinse commented Dec 18, 2025

Uh oh!

pavelzw commented Dec 18, 2025

Uh oh!

toprinse commented Dec 18, 2025

Uh oh!

vincent-pollet commented Feb 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

toprinse commented Nov 24, 2025 •

edited

Loading

pavelzw Dec 1, 2025 •

edited

Loading

toprinse Dec 2, 2025 •

edited

Loading

pavelzw commented Dec 1, 2025 •

edited

Loading

pavelzw commented Dec 18, 2025 •

edited

Loading