[CI][Python] A new version (77.0.2) of setuptools seems to have broken some of our builds

[raulcd]

Describe the bug, including details regarding any error messages, version, and platform.

Some of our nightly jobs have started failing with:

     File "/arrow-dev/lib/python3.10/site-packages/setuptools/dist.py", line 506, in _find_pattern
      raise InvalidConfigError(f"Pattern {pattern!r} cannot contain '..'")
  setuptools.errors.InvalidConfigError: Pattern '../LICENSE.txt' cannot contain '..'
  error: subprocess-exited-with-error

This is a new check introduced on setuptools 77.0.2 (released some hours ago):
pypa/setuptools@732ec1a

Component(s)

Continuous Integration, Python

[raulcd]

There seems to be a request to change this:

• [Request for Reverting Intentional Breaking Change] New license file validation breaks projects with non-standard layout pypa/setuptools#4892

[raulcd]

Source Release and merge script jobs are also failing due to this (PR jobs failing), example:

• GH-45868: [C++][CI] Fix test for ambiguous initialization on C++ 20 #45871
• GH-45819: [C++] Add OptionalBitmapAnd utility function #45869

[bipin-lekhak]

With setuptools version 77.0.3, the build still fails but the error is different.

SetuptoolsDeprecationWarning: `project.license` as a TOML table is deprecated

ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any
.
.
.
CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
        Could NOT find Python3 (missing: Python3_LIBRARIES Python3_INCLUDE_DIRS
        Development NumPy) (found version "3.10.14")
      Call Stack (most recent call first):
        /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
        /usr/share/cmake-3.16/Modules/FindPython/Support.cmake:2214 (find_package_handle_standard_args)
        /usr/share/cmake-3.16/Modules/FindPython3.cmake:300 (include)
        cmake_modules/FindPython3Alt.cmake:48 (find_package)
        CMakeLists.txt:284 (find_package)


      error: command '/usr/bin/cmake' failed with exit code 1

[raulcd]

A new release of setuptools has been done which temporarily solves the issue (there is a Warning of deprecation being raised).
We should investigate whether we have to do any changes, there are some conversations going on on the PEP that added the limitations for the License file:
https://discuss.python.org/t/pep-639-round-3-improving-license-clarity-with-better-package-metadata/53020/174
@pitrou should we close this and (if necessary) open a new one to follow up once the path forward is clear?

[pitrou]

I see we already have a symlink in the repo for python/cmake_modules, so perhaps we could add symlinks for python/LICENSE.txt and python/NOTICE.txt? What do you think @kou ?

(sidenote: why do we have NOTICE.txt in addition to LICENSE.txt?)

[kou]

In general, we should create python/LICENSE.txt and python/NOTICE.txt instead of symbolic linking to the top-level LICENSE.txt/NOTICE.txt because the top-level LICENSE.txt/NOTICE.txt include unrelated information for python/. (sdist includes only python/ and wheels includes only binaries of cpp/ and python/.)

The Apache 2.0 license doesn't require NOTICE.txt but the ASF policy requires it: https://www.apache.org/legal/src-headers.html#notice

[rok]

top-level LICENSE.txt/NOTICE.txt include unrelated information for python/. (sdist includes only python/ and wheels includes only binaries of cpp/ and python/.)

But it's a symlink to the top position and IMO it is implicitly clear what is being referred to and even if not it's just redundant information. I'll merge the proposed PR, but let's reopen if you think it's an issue.