GH-36411: [C++][Python] Use meson-python for PyArrow build system

[WillAyd]

Rationale for this change

This helps simplify the steps to build pyarrow by leveraging Meson, a build system strongly inspired by Python's syntax. In it's current form, it requires Arrow to be installed on the host system, but in the future we may even be able to have PyArrow build Arrow as a subproject, as needed

What changes are included in this PR?

This PR adds Meson configuration files to the Python code base within Arrow.

Are these changes tested?

Yes

Are there any user-facing changes?

We may want to deprecate the traditional setup.py way of building PyArrow alongside this.

• GitHub Issue: [Python] Improve the pyarrow build system #36411

[github-actions]

⚠️ GitHub issue #36411 has been automatically assigned in GitHub to PR creator.

[eli-schwartz]

Before doing this, include the following code:

# no-op placeholder
arrow_dep = dependency('', required: false)

if get_option('wrap_mode') != 'forcefallback'
  arrow_dep = dependency('arrow', 'Arrow', modules: ['Arrow::arrow_shared'], required: false)
endif

And then shift the rest to look like this:

if not arrow_dep.found()
    cmake = import('cmake')
    # further lookups
    # ...
    arrow_dep = arrow_proj.dependency('arrow_shared')
endif

[eli-schwartz]

What this does:

• check build options for wrap_mode, which is a builtin meson option allowing you to choose whether you wish to resolve bundled dependencies or look for system dependencies. It defaults to finding system dependencies, but when users run meson with --wrap-mode=forcefallback they are asking to explicitly avoid system deps
• first try to find an arrow dependency, using both names it might be available as:
  • "arrow" (pkgconfig)
  • "Arrow" (cmake, yes capitalization does matter), with modules: ensuring we pick up the correct cmake find_package() variable
• if it is not available, required: false means we continue to import the cmake subproject as a fallback

[eli-schwartz]

It is possible to avoid doing if/else checks:

$ cat subprojects/arrow.wrap

[wrap-file]
directory = arrow
method = cmake

[provide]
arrow = arrow_static_dep

However, using wrap files with method=cmake doesn't (currently) allow you to pass your add_cmake_defines. If you didn't need any defines, then you could simply do this:

arrow_dep = dependency('arrow', 'Arrow', modules: ['Arrow::arrow_shared'])

and you would not need any if/else, it would automatically build the cmake subproject if either:

• wrap-mode=forcefallback
• no system arrow was found

[WillAyd]

How does that wrap file work? The directory to the cpp source is in arrow/cpp whereas the wrap file itself will be located in arrow/python/subprojects - how would that resolve to the right directory?

[eli-schwartz]

Since you included a symlink anyway, I chose not to bother including any mechanism for downloading the wrap contents. Meson skips over that because the directory already exists with the correct content.

The key benefit of the wrap file is that it allows specifying in ini syntax:

• the subproject should use the method=cmake automatically, when used via dependency()
• the autogenerated arrow_static_dep (maybe this should be arrow_shared_dep instead?) will fulfill dependency('arrow')

Again, it's missing the necessary cmake defines so it may not be worth pursuing further.

[eli-schwartz]

You could still use setuptools-scm with meson if you want.

project(
    'pyarrow',
    # ...., 
    version: run_command('python3', '-m', 'setuptools_scm', '--force-write-version-files', check: true).stdout().strip(),
)

[WillAyd]

@kou I have made some offline progress on this, but one of the things I am getting stuck on is how the pyarrow C++ modules are being compiled. From what I understand, the current build process will compile Cython modules first (at least lib.pyx) and from that auto-generate lib.h and lib_api.h headers that the pyarrow modules can then reference (?)

Assuming that understanding is correct, where in the process are lib.h and lib_api.h being generated? I found the CMake command that copies them from the source to the build folder, but I can't figure out where they come from in the first place. Any guidance would be appreciated.

[kou]

The following codes may be related:

arrow/cpp/cmake_modules/UseCython.cmake

Lines 120 to 126 in 5e9fce4

	if(${property_is_api})
	set(_generated_files "${output_file}" "${_name}.h" "${_name}_api.h")
	elseif(${property_is_public})
	set(_generated_files "${output_file}" "${_name}.h")
	else()
	set(_generated_files "${output_file}")
	endif()

arrow/python/CMakeLists.txt

Line 674 in 5e9fce4

set_source_files_properties(pyarrow/lib.pyx PROPERTIES CYTHON_API TRUE)

[WillAyd]

Ah nevermind I think I have figured it out. So it looks like Cython generates the header files in the build directory when compiling lib.pyx, so the idea is to copy those header files to a directory structure in the build directory that the sources can resolve to.

I'll have to think about the best way to accomplish that via Meson.

[WillAyd]

As noted in the comment, meson-python advises that you symlink the NOTICE/LICENSE files in the source tree and use a dist script to pack the actual files into the Python sdist

[WillAyd]

@github-actions crossbow submit -g python

[github-actions]

Revision: 62168a0

Submitted crossbow builds: ursacomputing/crossbow @ actions-180cf98f75

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-1.3.4-numpy-1.21.2
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.12-pandas-latest-numpy-1.26
test-conda-python-3.12-pandas-latest-numpy-latest
test-conda-python-3.13
test-conda-python-3.13-pandas-nightly-numpy-nightly
test-conda-python-3.13-pandas-upstream_devel-numpy-nightly
test-conda-python-3.14
test-conda-python-emscripten
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-cuda-python-ubuntu-24.04-cuda-13.0.2
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-42-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

[WillAyd]

Rebased on main and all green. From the crossbow jobs, the freethreading and pandas job failures appear directly related to #48314 and not this PR

The emscripten failure also seems to be unrelated, as it fails on an install step prior to the PyArrow steps

The cuda failures don't seem to have produced logs? If there is something to look into there please let me know