GPU version of the code doesn't seem to successfully deploy kernels

[aeteijeiro]

Good morning,

when I add the -v switch to any of the benchmark executables, I get the following error:

I have already verified that my GPU, drivers, CUDA, etc. are setup correctly with a simple vector addition program that runs with no problems. The command I am running to build the max pooling benchmark is: "make cuda DATATYPE=FLOAT BLOCKSIZE=4" from max pooling directory. When looking at the output of this make command, I see that it is using g++ at the end, instead of nvcc. Could this be the problem? Simply changing g++ to nvcc in the makefile leads to a rabbit hole of problems.

If this is not the issue, can someone please point me in the right direction? Perhaps I am missing a step.

[aeteijeiro]

Fixed it. I changed the Makefile to use /usr/bin/nvcc for both the CC and NVCC variables.

For any other novices out there: Apparently, originally you could combine normal cpp files with .cu files by compiling the cuda code separately with nvcc -c, then compiling the cpp file with g++ and simultaneously linking in the CUDA object code by putting the object file(s) from the first step in the same line. However, from trying to get a simple vector addition equivalent example going, it seems that cuda12.2 is having problems working with g++ 11.4. In particular, NVCC can find the cuda libraries, but g++ can't. Since I don't know at the moment where said libraries are (they are not in the usual place on my particular install), my fix was to simply change the make file to use nvcc for both cpp code and .cu code compilation, by implementing the aforementioned changes.

Also, and I'm not sure whether this would break anything, but I changed the NVCCFLAGS line to reflect my GPU.

[jaquerinte]

Hi @aeteijeiro sorry for the delay replay, summer holydays is almost a stop here.  We will take a look to that to see if we can reproduce the issue and apply appropriate measures to fix the issue if needed. For the other comment, yes  you should change the NVCCFLAGS to reflect the GPU that you are using.

[aeteijeiro]

@jaquerinte

No worries. If it helps in reproducing the error: I am running Ubuntu 22.04 on a g4dn.xlarge instance from AWS, which has an NVIDIA T4 GPU.