-
Notifications
You must be signed in to change notification settings - Fork 51
Wheel building & PyPI publishing #109
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
524bced
96cd2f5
8afb21c
bdfaec1
e437141
1c5a353
5892d2b
c0dca76
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,131 @@ | ||
| name: Build and Publish | ||
|
|
||
| on: | ||
| push: | ||
| tags: [ 'v*' ] | ||
| pull_request: | ||
| branches: [ main ] | ||
| release: | ||
| types: [ published ] | ||
|
|
||
| env: | ||
| FORCE_COLOR: 1 | ||
|
|
||
| jobs: | ||
| build_wheels: | ||
| name: Build wheels on ${{ matrix.os }} | ||
| runs-on: ${{ matrix.os }} | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| os: [ubuntu-24.04, macos-13, macos-14] | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Install Linux dependencies | ||
| if: runner.os == 'Linux' | ||
| run: | | ||
| sudo apt-get update | ||
| sudo apt-get install -y libattr1-dev libfuse3-dev fuse3 pkg-config | ||
| - name: Install macOS dependencies | ||
| if: runner.os == 'macOS' | ||
| run: | | ||
| brew install macfuse pkg-config | ||
| - name: Set up Python | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: '3.12' | ||
|
|
||
| - name: Install cibuildwheel | ||
| run: python -m pip install cibuildwheel==2.16.5 | ||
|
|
||
| - name: Build wheels | ||
| run: python -m cibuildwheel --output-dir wheelhouse | ||
| env: | ||
| CIBW_BUILD: cp38-* cp39-* cp310-* cp311-* cp312-* cp313-* | ||
|
|
||
| CIBW_SKIP: "*-win32 *-manylinux_i686 *-musllinux*" | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why skip manylinux? |
||
|
|
||
| - uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: cibw-wheels-${{ matrix.os }}-${{ strategy.job-index }} | ||
| path: ./wheelhouse/*.whl | ||
|
|
||
| build_sdist: | ||
| name: Build source distribution | ||
| runs-on: ubuntu-24.04 | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Install Linux dependencies | ||
| run: | | ||
| sudo apt-get update | ||
| sudo apt-get install -y libattr1-dev libfuse3-dev fuse3 pkg-config | ||
| - name: Set up Python | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: '3.12' | ||
|
|
||
| - name: Install build dependencies | ||
| run: | | ||
| python -m pip install --upgrade pip | ||
| pip install setuptools cython build | ||
| - name: Build Cython files | ||
| run: python setup.py build_cython | ||
|
|
||
| - name: Build sdist | ||
| run: python -m build --sdist | ||
|
|
||
| - uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: cibw-sdist | ||
| path: dist/*.tar.gz | ||
|
|
||
| publish_to_pypi: | ||
| name: Publish to PyPI | ||
| needs: [build_wheels, build_sdist] | ||
| runs-on: ubuntu-24.04 | ||
| if: github.event_name == 'release' && github.event.action == 'published' | ||
| environment: | ||
| name: pypi | ||
| url: https://pypi.org/p/pyfuse3 | ||
| permissions: | ||
| id-token: write | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could you clarify how the permissioning works? Do we need to get a token from PyPi and store it somewhere where Github can access it? |
||
|
|
||
| steps: | ||
| - uses: actions/download-artifact@v4 | ||
| with: | ||
| pattern: cibw-* | ||
| path: dist | ||
| merge-multiple: true | ||
|
|
||
| - name: Publish to PyPI | ||
| uses: pypa/gh-action-pypi-publish@release/v1 | ||
|
|
||
| publish_to_testpypi: | ||
| name: Publish to TestPyPI | ||
| needs: [build_wheels, build_sdist] | ||
| runs-on: ubuntu-24.04 | ||
| if: github.event_name == 'push' && github.ref == 'refs/heads/main' | ||
| environment: | ||
| name: testpypi | ||
| url: https://test.pypi.org/p/pyfuse3 | ||
| permissions: | ||
| id-token: write | ||
|
|
||
| steps: | ||
| - uses: actions/download-artifact@v4 | ||
| with: | ||
| pattern: cibw-* | ||
| path: dist | ||
| merge-multiple: true | ||
|
|
||
| - name: Publish to TestPyPI | ||
| uses: pypa/gh-action-pypi-publish@release/v1 | ||
| with: | ||
| repository-url: https://test.pypi.org/legacy/ | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| name: Create Release | ||
|
|
||
| on: | ||
| push: | ||
| tags: | ||
| - 'v*' | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This does not seem to match the current tagging schema...? |
||
|
|
||
| permissions: | ||
| contents: write | ||
|
|
||
| jobs: | ||
| create-release: | ||
| name: Create GitHub Release | ||
| runs-on: ubuntu-24.04 | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Extract version from tag | ||
| id: version | ||
| run: echo "version=${GITHUB_REF#refs/tags/v}" >> $GITHUB_OUTPUT | ||
|
|
||
| - name: Extract changelog | ||
| id: changelog | ||
| run: | | ||
| # Extract changelog for this version from Changes.rst | ||
| if [ -f "Changes.rst" ]; then | ||
| # Try to extract the section for this version | ||
| awk '/^pyfuse3 '"${{ steps.version.outputs.version }}"'/{flag=1; next} /^pyfuse3 [0-9]/{flag=0} flag' Changes.rst > changelog.txt | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could you add some output to the PR that illustrates that this parsing has been tested? Also, can we make the job fail if parsing fails? |
||
| if [ -s changelog.txt ]; then | ||
| echo "changelog<<EOF" >> $GITHUB_OUTPUT | ||
| cat changelog.txt >> $GITHUB_OUTPUT | ||
| echo "EOF" >> $GITHUB_OUTPUT | ||
| else | ||
| echo "changelog=Release ${{ steps.version.outputs.version }}" >> $GITHUB_OUTPUT | ||
| fi | ||
| else | ||
| echo "changelog=Release ${{ steps.version.outputs.version }}" >> $GITHUB_OUTPUT | ||
| fi | ||
| - name: Create Release | ||
| uses: actions/create-release@v1 | ||
| env: | ||
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| with: | ||
| tag_name: ${{ github.ref }} | ||
| release_name: pyfuse3 ${{ steps.version.outputs.version }} | ||
| body: ${{ steps.changelog.outputs.changelog }} | ||
| draft: false | ||
| prerelease: ${{ contains(steps.version.outputs.version, 'rc') || contains(steps.version.outputs.version, 'beta') || contains(steps.version.outputs.version, 'alpha') }} | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this should be integrated into developer-notes/release_process.rst instead. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,147 @@ | ||
| # Release Process for pyfuse3 | ||
|
|
||
| This document describes the automated release process for pyfuse3. | ||
|
|
||
| ## Overview | ||
|
|
||
| The project uses GitHub Actions to automatically build wheels and publish to PyPI when releases are created. The process supports: | ||
|
|
||
| - **Multi-platform wheel building** (Linux, macOS) using cibuildwheel | ||
| - **Automatic PyPI publishing** on GitHub releases | ||
| - **TestPyPI publishing** on main branch pushes | ||
| - **Source distribution building** | ||
|
|
||
| ## Release Workflows | ||
|
|
||
| ### 1. Build and Publish (`build-and-publish.yml`) | ||
|
|
||
| This workflow runs on: | ||
| - GitHub releases → Publishes to PyPI | ||
| - Pull requests → Builds wheels for testing | ||
|
|
||
| **Jobs:** | ||
| - `build_wheels`: Builds wheels for Linux and macOS using cibuildwheel | ||
| - `build_sdist`: Builds source distribution | ||
| - `publish_to_pypi`: Publishes to PyPI on releases (requires trusted publishing) | ||
ThomasWaldmann marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - `publish_to_testpypi`: Publishes to TestPyPI on main branch pushes | ||
|
|
||
| ### 2. Release Creation (`release.yml`) | ||
|
|
||
| Automatically creates GitHub releases when version tags are pushed. | ||
|
|
||
| ### 3. Testing (`test.yml`) | ||
|
|
||
| Runs the existing test suite across multiple Python versions. | ||
|
|
||
| ## Making a Release | ||
|
|
||
| ### Option 1: Using the Release Script | ||
|
|
||
| ```bash | ||
| # Create a new release | ||
| python scripts/release.py 3.4.1 | ||
|
|
||
| # Dry run to see what would happen | ||
| python scripts/release.py 3.4.1 --dry-run | ||
|
|
||
| # Update version without creating tag | ||
| python scripts/release.py 3.4.1 --no-tag | ||
| ``` | ||
|
|
||
| ### Option 2: Manual Process | ||
|
|
||
| 1. **Update version** in `setup.py`: | ||
| ```python | ||
| PYFUSE3_VERSION = '3.4.1' | ||
| ``` | ||
|
|
||
| 2. **Commit and tag**: | ||
| ```bash | ||
| git add setup.py | ||
| git commit -m "Bump version to 3.4.1" | ||
| git tag -a v3.4.1 -m "Release 3.4.1" | ||
| ``` | ||
|
|
||
| 3. **Push**: | ||
| ```bash | ||
| git push origin main | ||
| git push origin v3.4.1 | ||
| ``` | ||
|
|
||
| ## Repository Setup Requirements | ||
|
|
||
| ### GitHub Repository Settings | ||
|
|
||
| 1. **Enable GitHub Actions** in repository settings | ||
|
|
||
| 2. **Configure PyPI Trusted Publishing**: | ||
ThomasWaldmann marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - Go to PyPI → Account Settings → Publishing | ||
| - Add trusted publisher for your GitHub repository | ||
| - Set environment name to `release` | ||
|
|
||
| 3. **Configure TestPyPI Trusted Publishing** (optional): | ||
| - Same process for test.pypi.org | ||
| - Set environment name to `test-release` | ||
|
|
||
| 4. **Create GitHub Environments**: | ||
| - Go to repository Settings → Environments | ||
| - Create `release` environment (for production releases) | ||
| - Create `test-release` environment (for test releases) | ||
| - Enable "Required reviewers" if desired | ||
|
|
||
| ### Dependencies | ||
|
|
||
| The build process requires these system dependencies: | ||
| - **Linux**: `libfuse3-dev`, `pkg-config` | ||
| - **macOS**: `macfuse`, `pkg-config` (via Homebrew) | ||
|
|
||
| These are automatically installed by the workflow. | ||
|
|
||
| ## Build Configuration | ||
|
|
||
| The build is configured via `pyproject.toml` using cibuildwheel: | ||
|
|
||
| - **Python versions**: 3.8, 3.9, 3.10, 3.11, 3.12, 3.13 | ||
| - **Platforms**: Linux (x86_64), macOS (x86_64, arm64) | ||
| - **Skip**: 32-bit builds, musl Linux builds | ||
| - **Dependencies**: Automatically installs FUSE development libraries | ||
|
|
||
| ## Testing | ||
|
|
||
| Each built wheel is tested by importing pyfuse3 to ensure basic functionality. | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| ### Build Failures | ||
|
|
||
| 1. **FUSE dependency issues**: Check that system dependencies are properly installed | ||
| 2. **Cython compilation**: Ensure Cython files are built before wheel creation | ||
| 3. **Platform-specific issues**: Check cibuildwheel logs for platform-specific errors | ||
|
|
||
| ### Publishing Failures | ||
|
|
||
| 1. **Authentication**: Ensure trusted publishing is configured correctly | ||
| 2. **Duplicate versions**: PyPI doesn't allow re-uploading the same version | ||
| 3. **Environment protection**: Check GitHub environment settings | ||
|
|
||
| ### Local Testing | ||
|
|
||
| Test the release process locally: | ||
|
|
||
| ```bash | ||
| # Install cibuildwheel | ||
| pip install cibuildwheel | ||
|
|
||
| # Build wheels locally | ||
| python -m cibuildwheel --output-dir wheelhouse | ||
|
|
||
| # Test installation | ||
| pip install wheelhouse/*.whl | ||
| python -c "import pyfuse3; print('Success!')" | ||
| ``` | ||
|
|
||
| ## Monitoring | ||
|
|
||
| - **GitHub Actions**: Monitor workflow runs in the Actions tab | ||
| - **PyPI**: Check package page for successful uploads | ||
| - **TestPyPI**: Verify test uploads work correctly | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,5 @@ | ||
| [build-system] | ||
| requires = ["setuptools"] | ||
| requires = ["setuptools", "cython"] | ||
| build-backend = "setuptools.build_meta" | ||
|
|
||
| [tool.mypy] | ||
|
|
@@ -8,3 +8,29 @@ packages = ["pyfuse3"] | |
| modules = ["_pyfuse3", "pyfuse3_asyncio"] | ||
| namespace_packages = false | ||
| strict = true | ||
|
|
||
| [tool.cibuildwheel] | ||
| build = "cp38-* cp39-* cp310-* cp311-* cp312-* cp313-*" | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This seems to repeat code from the Github workflow. |
||
| skip = "*-win32 *-manylinux_i686 *-musllinux*" | ||
|
|
||
| # Use newer manylinux images to avoid CentOS 7 mirror issues | ||
| manylinux-x86_64-image = "manylinux_2_28" | ||
| manylinux-aarch64-image = "manylinux_2_28" | ||
|
|
||
| before-build = [ | ||
| "pip install setuptools cython", | ||
| "python setup.py build_cython" | ||
| ] | ||
|
|
||
| test-requires = ["pytest", "pytest-trio", "trio"] | ||
| test-command = "python -c \"import pyfuse3; print('pyfuse3 imported successfully')\"" | ||
|
|
||
| [tool.cibuildwheel.linux] | ||
| before-all = [ | ||
| "dnf install -y fuse3-devel pkgconfig" | ||
| ] | ||
| environment = { PKG_CONFIG_PATH = "/usr/lib/x86_64-linux-gnu/pkgconfig:/usr/lib/pkgconfig:/usr/share/pkgconfig" } | ||
|
|
||
| [tool.cibuildwheel.macos] | ||
| before-all = ["brew install macfuse pkg-config"] | ||
| environment = { PKG_CONFIG_PATH = "/usr/local/lib/pkgconfig:/opt/homebrew/lib/pkgconfig" } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please remove end-of-life distributions.