Investigation into `pixi` as a build tool and package manager

[JeppeKlitgaard]

Introduction

In preparation for the user meeting next week I have spent some time fiddling around with CIL and bumped into a few places where I believe the ergonomics of using and contributing to CIL could be improved.

I have had a look at pixi, which aims improve upon ergonomics, speed, and reproducibility in conda-based projects, not dissimilar to what uv has done for PEP517/518 Python projects.

This issue is meant to discuss the potential use of pixi in CIL.

A WIP branch with an initial Pixi configuration can be found here: https://github.com/JeppeKlitgaard/CIL/tree/experiment-pixi

Pro/Con of Pixi

Pro:

• Significantly easier development setup for CIL
• Reproducible environments
• Very actively developed and incredible support from their development team (Pixi fails to use channel with zstd error prefix-dev/pixi#5398, https://discord.com/channels/1082332781146800168/1082338253925003385/1464556088014667927)
• Developers include the creators of mamba and rattler
• Built-in task runner
  • This provides a uniform, reproducible and self-documenting way for contributors to run tests, documentation builds, linting, etc
• Relatively good (and improving) conda-PEP517/518 interoperability

Cons:

• Still relatively new, though production ready by their own accounts
  • Notably has been adopted by some mature scientific python libraries, see xarray
• Conda builds don't support dynamic versions yet (I think)
• Would require contributors to use pixi as part of the development process
  • This means disturbing the core contributors existing workflows and may be no-go for this reason
  • Though: would likely benefit new contributors and lower contribution barrier-to-entry
  • Though: should be noted that pixi is exceedingly easy to install
  • Though: users of CIL can continue to use conda without any changes required

Areas where pixi could help

It may be that this is not the tool for CIL, but I think it could help alleviate some long-standing issues that I found from quickly browsing the issue tracker:

• conda: self-host packages #1870: could help with building and uploading conda packages easily
• Use pixi for CI instead of conda #2171: faster CI runs
• Dev install instructions create broken version.py #2255: reproducible and easier dev installation
• Instructions for developing CIL on windows are unclear #2245: should be able to handle most build-tooling setups, even on Windows. Also possible in conda but may be easier here
• Use conda package to create CIL development and test environment #2188: Would solve this very elegantly
• upgrade build framework #1875: Has good tooling to continue the good work already done here
• docker build fail #2161: Reproducible builds could prevent sudden failures like this
• Autoyapf GitHub Action #1297: Reproducible dev environments become really important when introducing linting to ensure consistent rules
• Run GHA on different environment configuration #2103: Could provide a sane solution to this
• Github actions should use updated cmake command with python executable path set #2082: I believe this would be handled automatically if actions leveraged Pixi configuration
• Warning that anaconda channel is used in GHA #2100, github actions have issues regarding defaults channel being added implicitly #2051: Pixi would not fix this, but may get a configuration where this is not a problem during implementation of Pixi
• PyPi Wheel #2097: Pixi is best-in-class at collocation of dependencies and configuration between PyPi tooling and conda tooling. Would still require dependencies available on pypi though
• boa has been superseded by rattler-build #2035: Look! rattler-build, this is born out of the pixi eco-system. Would be solved by switching conda build system to Pixi
• Update doc comments, typehinting, and code structure #1683: A rugged, long-lasting implementation of this would benefit from pixi task runners and reproducible environments
• Contributor guide #1233: Pixi task runner helps make this sort of thing a lot easier (instead of contributor potentially running wrong test command in misconfigured environment, pixi run test would always run in a reproducible environment)
• Coding conventions #1023: Easier to enforce if conventions are enforced with pixi run lint
• Add auto-formatter and formatting for python code #1229: Very important that auto-formatter version is consistent in all developer and CI environments to prevent output mismatch, solved by reproducible builds and central configuration

[JeppeKlitgaard]

I am increasingly confident that Pixi would be a really good bet for CIL. Having done everything on a Linux box, I was able to get it to install on Windows without any changes.

Mac support is more tricky since ccpi-regularisers only has conda packages for win64 and linux-64. I think this is an issue best dealt with separately from any pixi consideration as it relates to ccpi-regularisers itself.