Python dependencies in MatX

[jwmelto]

Continuing the discussion from cupy/cupy#9464.

I am building a MatX-enabled toolchain container, to support my MatX application. An (unverified) minimal Containerfile looks like this:

FROM nvidia/cuda:13.0.1-devel-ubi9

ARG cuda_arch_list="80"
ENV CUDAARCHS="${cuda_arch_list}"

RUN dnf install -y gcc-toolset-gcc-c++ python3.12
RUN alternatives --install /usr/bin/python3 python3 /usr/bin/python3.12
ENV SCL_MODS='gcc-toolset-13'

RUN scl enable ${SCL_MODS} -- sh -ec '  \
  python3 -m ensurepip \
  ; python3 -m pip install --upgrade pip \
  ; python3 -m pip install -q \
      numpy scipy \
 ' && date "+packages installed %F %T"

##   cupy takes a LONG time to build.
##    - make sure it is built with our compute architecture(s)
RUN echo "Installing MatX Python stuff"                                                         \
&& export CUPY_NVCC_GENERATE_CODE=$(echo $CUDAARCHS                    \
    | awk -F';'                                                                                                  \
       '{for(i=1;i<=NF;i++) printf "%s%sarch=compute_%s,code=sm_%s", (i>1?",":""), "", $i, $i}' \
    )                                                                                                                   \
&& echo "CUPY_NVCC_GENERATE_CODE=$CUPY_NVCC_GENERATE_CODE"             \
&& scl enable ${SCL_MODS} -- sh -ec '                                                           \
    export CC=$(command -v gcc) CXX=$(command -v g++)                      \
    ; python3 -m pip install                                                                       \
            cupy                                                                                     \
    ' && date                                                                                                   \
&& python3 -c "import cupy; print('CuPy import successful')"

(there are probably some other Python packages, like matplotlib; this was taken from the CuPy issue, and was reduced to show the CuPy problem).

I install MatX into the image like this:

ARG matx_src_repo='https://github.com/NVIDIA/MatX.git'
ARG matx_src_tag='v0.9.3'
#####################################
#####           MATX            ####
#####################################
RUN export project=matx                                                                                      \
           project_tag=${matx_src_tag}                                                       \
           project_repo=${matx_src_repo}                                  \
           src=/tmp                                                                             \
&& echo "Installing $project"                                                               \
&& date                                                                                                        \
&& (cd $src                                                                                                 \
     ; git clone                                                                                  \
           --quiet                                                                                  \
           --config advice.detachedHead=false                                       \
           --single-branch                                                                 \
           --branch ${project_tag}                                                  \
           ${project_repo}                                                                  \
           ${project}                                                                            \
     ; cd $project                                                                            \
     ; git rev-parse HEAD                                                             \
    ) && date                                                                                                   \
&& echo "building..."                                                                               \
&& scl enable ${SCL_MODS} -- sh -ec '                                                           \
     cd $src/$project ; mkdir build ; cd build                                       \
     ; export CXXFLAGS=-std=${cxx_standard}                                   \
     ; ${CMAKE}                                                                                              \
         ${CMAKE_FLAGS}                                                                              \
         ${image_CMAKE_FLAGS}                                                                \
         ${cmake_INSTALL_FLAGS}                                                                            \
         -DCMAKE_BUILD_TYPE=Release                                                 \
         -DCMAKE_CUDA_ARCHITECTURES=${CUDAARCHS}                        \
         -DMATX_BUILD_TESTS=OFF                                                                         \
         -DMATX_BUILD_BENCHMARKS=OFF                                                        \
         -DMATX_BUILD_EXAMPLES=OFF                                                 \
         -DMATX_BUILD_DOCS=OFF                                                                         \
         -DMATX_EN_CUDA_LINEINFO=ON                                                            \
         -DMATX_EN_FILEIO=ON                                                                 \
         -DMATX_EN_VISUALIZATION=ON                                               \
         -DMATX_EN_PYBIND11=ON                                                                        \
         -S ..                                                                                                        \
     ; make ${MAKEFLAGS} install                                                          \
     ; /sbin/ldconfig                                                                                      \
     ; cd $src; rm -rf $project                                                                     \
     ' && date

By enabling visualization, I trigger various python dependencies. (I'll also assert that pybind11 is installed, but not relevant to this discussion).

If CuPy is not (successfully) installed, CMake spews the horrible error message that you get when an import fails. @cliffburdick asserts that, since MatX is header-only, it doesn't need to build anything, but with the runtime python bindings (FILEIO, VISUALIZATION), if the packages are not present bad things will happen.

The purpose of this discussion is to clarify what Python dependencies MatX truly has, and the consequences of an incompletely supported Python environment.

My goal for this Containerfile is to isolate all the version dependencies of all the components external to this file, so they can be modified through build-time arguments without having to change the Containerfile.

[cliffburdick]

Hi @jwmelto can you paste the errors CMake is showing without cuPy? As I mentioned in that other thread cuPy is only required for a handful of unit tests, and since you're not building the tests I wouldn't expect it to care. You can see here that it's an optional dependency and should show a warning only.

If it's an error that's stopping your build it would be good to understand why that's being triggered by cuPy.

[jwmelto]

This is what I see today:

-- checking python import module cupy
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'cupy'
-- The optional python package cupy package is not installed. Some unit tests and functionality may not work

That's not horrible; if CuPy is installed but not patched per the original discussion, it throws the verbose exception originally noted:

Traceback (most recent call last):
  File "/usr/local/lib64/python3.12/site-packages/cupy/__init__.py", line 16, in <module>
    from cupy import _core  # NOQA
    ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.12/site-packages/cupy/_core/__init__.py", line 3, in <module>
    from cupy._core import core  # NOQA
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "cupy/_core/core.pyx", line 1, in init cupy._core.core
  File "cupy/_core/_kernel.pyx", line 1, in init cupy._core._kernel
ImportError: /usr/local/lib64/python3.12/site-packages/cupy/_core/_[memory_range.cpython-312-x86_64-linux-gnu.so](http://memory_range.cpython-312-x86_64-linux-gnu.so/): undefined symbol: __gxx_personality_v0
 
The above exception was the direct cause of the following exception:
 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib64/python3.12/site-packages/cupy/__init__.py", line 18, in <module>
    raise ImportError(f'''
ImportError:
================================================================
Failed to import CuPy.
 
If you installed CuPy via wheels (cupy-cudaXXX or cupy-rocm-X-X), make sure that the package matches with the version of CUDA or ROCm installed.
 
On Linux, you may need to set LD_LIBRARY_PATH environment variable depending on how you installed CUDA/ROCm.
On Windows, try setting CUDA_PATH environment variable.
 
Check the Installation Guide for details:
  https://docs.cupy.dev/en/latest/install.html
 
Original error:
  ImportError: /usr/local/lib64/python3.12/site-packages/cupy/_core/_[memory_range.cpython-312-x86_64-linux-gnu.so](http://memory_range.cpython-312-x86_64-linux-gnu.so/): undefined symbol: __gxx_personality_v0
================================================================

To your point: neither of these scenarios stops installation. And this Containerfile has evolved over time, as has MatX, so I can't say that it wasn't scarier earlier. I do try to have error/warning-free builds of all the packages I use, because the casual user lacks the insight to know what is meaningful and what is not.

I'm content to not install CuPy; the current state of things is tolerable.

Thanks.

[jwmelto]

It's also possible that when I started, I was building/running the unit tests, since this was alpha software. Having gained some comfort using it, I opted for a faster image creation process and disabled the unit tests.

[cliffburdick]

I see now. I'm not a cuPy expert so I can't comment on that compilation issue like Leo can, but I will say that we always install cuPy through pip in our containers so I have not tested without that. The original cuPy warning above is what I would expect for it to fail to find that as an optional component.