Wheel building & PyPI publishing #109

mlapaglia · 2025-09-15T13:26:24Z

Add CI/CD workflows for automated wheel building and PyPI publishing

Builds wheels for Linux and macOS across Python 3.8-3.13
Publishes to TestPyPI on main branch pushes
Publishes to PyPI on GitHub releases
Uses trusted publishing for secure authentication

Notes:

The mac runners seem to take quite a while, we might want to limit this just to ubuntu which is much faster.

- Add build-and-publish.yml workflow for multi-platform wheel building using cibuildwheel - Add release.yml workflow for automatic GitHub release creation - Update pyproject.toml with cibuildwheel configuration - Add release.py script for version management and tagging - Add RELEASE.md documentation for the release process Features: - Builds wheels for Linux and macOS across Python 3.8-3.13 - Publishes to TestPyPI on main branch pushes - Publishes to PyPI on GitHub releases - Uses trusted publishing for secure authentication

ThomasWaldmann · 2025-09-15T15:53:23Z

pyproject.toml

+
+[tool.cibuildwheel.macos]
+before-all = ["brew install macfuse pkg-config"]
+environment = { PKG_CONFIG_PATH = "/usr/local/lib/pkgconfig:/opt/homebrew/lib/pkgconfig" }


add an LF here?

ThomasWaldmann · 2025-09-15T17:45:56Z

RELEASE.md

+### 1. Build and Publish (`build-and-publish.yml`)
+
+This workflow runs on:
+- Push to `main` branch → Publishes to TestPyPI


does that also run when merging a PR?

if so, it would publish potentially a lot on TestPyPI - do they discard stuff automatically after a while?

yes, i removed this so it will only release packages when you publish a release (pre-release goes to test pypi, official releases go to pypi)

RELEASE.md

ThomasWaldmann · 2025-09-15T17:55:14Z

I think we should offer wheels for all popular and supported/supportable platforms, including macOS. Yeah, the macOS runners are slow, guess they are overloaded and microsoft can't afford more hardware from apple.

mlapaglia · 2025-09-18T12:20:52Z

I think we should offer wheels for all popular and supported/supportable platforms, including macOS. Yeah, the macOS runners are slow, guess they are overloaded and microsoft can't afford more hardware from apple.

i just mean for your wallet, macos runners are more expensive that the linux ones. i'll leave them in for now and address your issues.

ThomasWaldmann · 2025-09-18T13:01:43Z

@mlapaglia For FOSS repos, the runners are free.

To minimize load on the macOS runners (which are overloaded and slow already), I configured test runs in some of my repos so that the macOS runs depend on the linux runs being successful. Not sure how it is best done for wheel building though.

mlapaglia · 2025-09-24T12:21:08Z

if you need any other changes lmk ❤️

Nikratio · 2025-12-28T13:25:29Z

Apologies for the long delay here. This is useful functionality and I have some time for this over the next 7 days, so hopefully we can get this sorted.

Could you clarify if any AI coding agents were used to create this branch, and if so document which agent was used and to which extent you have reviewed and tested the code yourself?

It'd be great if you could also rebase this onto main and resolve the conflicts.

Nikratio · 2025-12-28T13:22:57Z

RELEASE.md

I think this should be integrated into developer-notes/release_process.rst instead.

Nikratio · 2025-12-28T13:26:28Z

.github/workflows/build-and-publish.yml

+      - name: Build wheels
+        run: python -m cibuildwheel --output-dir wheelhouse
+        env:
+          CIBW_BUILD: cp38-* cp39-* cp310-* cp311-* cp312-* cp313-*


Please remove end-of-life distributions.

Nikratio · 2025-12-28T13:26:45Z

.github/workflows/build-and-publish.yml

+        env:
+          CIBW_BUILD: cp38-* cp39-* cp310-* cp311-* cp312-* cp313-*
+
+          CIBW_SKIP: "*-win32 *-manylinux_i686 *-musllinux*"


Why skip manylinux?

Nikratio · 2025-12-28T13:27:36Z

.github/workflows/build-and-publish.yml

+      name: pypi
+      url: https://pypi.org/p/pyfuse3
+    permissions:
+      id-token: write


Could you clarify how the permissioning works? Do we need to get a token from PyPi and store it somewhere where Github can access it?

Nikratio · 2025-12-28T13:29:14Z

.github/workflows/release.yml

+          # Extract changelog for this version from Changes.rst
+          if [ -f "Changes.rst" ]; then
+            # Try to extract the section for this version
+            awk '/^pyfuse3 '"${{ steps.version.outputs.version }}"'/{flag=1; next} /^pyfuse3 [0-9]/{flag=0} flag' Changes.rst > changelog.txt


Could you add some output to the PR that illustrates that this parsing has been tested?

Also, can we make the job fail if parsing fails?

Nikratio · 2025-12-28T13:30:52Z

pyproject.toml

 strict = true
+
+[tool.cibuildwheel]
+build = "cp38-* cp39-* cp310-* cp311-* cp312-* cp313-*"


This seems to repeat code from the Github workflow.

Nikratio · 2025-12-28T13:31:42Z

scripts/release.py

+import subprocess
+import sys
+from pathlib import Path
+


I think it'd be better to extract the version from the Git tags (since we're effectively treating them as the new source of truth now).

Nikratio · 2025-12-28T13:33:39Z

scripts/release.py

+    with open(setup_py) as f:
+        content = f.read()
+
+    match = re.search(r"PYFUSE3_VERSION = ['\"]([^'\"]+)['\"]", content)


The PYFUSE3_VERSION variable should probably be removed, and instead we should use dynamic versioning (https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html)

Nikratio · 2025-12-28T13:34:42Z

scripts/release.py

As far as I can tell, this script just reads and updates the version number in .py files and then creates a git tag. So if we instead use git tags as the version source, I think this script can be removed completely.

Nikratio · 2025-12-28T13:35:09Z

test/test_examples.py

This feels like an unrelated change that should go into a separate PR.

mlapaglia added 7 commits September 15, 2025 12:35

comment cleanup

96cd2f5

move it to pyproject

8afb21c

test fix for timing

bdfaec1

correct dev library

e437141

more tolerance, update build version

1c5a353

remove comments

5892d2b

mlapaglia mentioned this pull request Sep 15, 2025

building wheels #108

Open

ThomasWaldmann reviewed Sep 15, 2025

View reviewed changes

RELEASE.md Show resolved Hide resolved

ThomasWaldmann reviewed Sep 15, 2025

View reviewed changes

RELEASE.md Show resolved Hide resolved

dont build on push to main

c0dca76

mlapaglia mentioned this pull request Sep 25, 2025

pip install in venv failing mlapaglia/Borgitory#92

Closed

Nikratio requested changes Dec 28, 2025

View reviewed changes

Wheel building & PyPI publishing #109

Are you sure you want to change the base?

Wheel building & PyPI publishing #109

Conversation

mlapaglia commented Sep 15, 2025

Add CI/CD workflows for automated wheel building and PyPI publishing

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ThomasWaldmann commented Sep 15, 2025

Uh oh!

mlapaglia commented Sep 18, 2025

Uh oh!

ThomasWaldmann commented Sep 18, 2025

Uh oh!

mlapaglia commented Sep 24, 2025

Uh oh!

Nikratio commented Dec 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants