Build glitch: files built, but not present in required directory

[Gastove]

I'm working on building extempore on Fedora 32; I'm running into an issue in which it seems a set of files are being built correctly, but are not being moved to the correct staging directory.

The files in question are librtmidi.so and libsndfile.so. I hit problems with librtmidi.so first:

 71%] moving shared libs into /var/home/gastove/Code/open-source/extempore/libs/platform-shlibs
Error copying file "librtmidi.so" to "/var/home/gastove/Code/open-source/extempore/libs/platform-shlibs".
make[2]: *** [CMakeFiles/external_shlibs_audio.dir/build.make:81: CMakeFiles/external_shlibs_audio] Error 1
make[1]: *** [CMakeFiles/Makefile2:370: CMakeFiles/external_shlibs_audio.dir/all] Error 2
make: *** [Makefile:183: all] Error 2

Line 81 of external_shlibs_audio.dir/build.make is attempting to run:

cd /var/home/gastove/Code/open-source/extempore/build/deps-install/lib && /usr/bin/cmake -E copy librtmidi.so /var/home/gastove/Code/open-source/extempore/libs/platform-shlibs

Surely enough, librtmidi.so is not in the source directory:

ls -la /var/home/gastove/Code/open-source/extempore/build/deps-install/lib
.rwxr-xr-x@  21k gastove 14 Jun 11:16 libkiss_fft.so
.rwxr-xr-x@ 217k gastove 14 Jun 11:16 libnanovg.so
.rwxr-xr-x@  49k gastove 14 Jun 11:16 libportmidi.so
.rwxr-xr-x@ 119k gastove 14 Jun  9:46 libstb_image.so

But it has been built! I can symlink it myself:

gastove@toolbox [gke->kitchen-stadium] ~/Code/open-source/extempore/build/ on v0.8.7 ✓ >( 2 )< λ
$ ln -s (pwd)/deps-install/lib64/librtmidi.so /var/home/gastove/Code/open-source/extempore/build/deps-install/lib/

gastove@toolbox [gke->kitchen-stadium] ~/Code/open-source/extempore/build/ on v0.8.7 ✓ λ
$ el /var/home/gastove/Code/open-source/extempore/build/deps-install/lib/
.rwxr-xr-x@  21k gastove 14 Jun 11:16 libkiss_fft.so
.rwxr-xr-x@ 217k gastove 14 Jun 11:16 libnanovg.so
.rwxr-xr-x@  49k gastove 14 Jun 11:16 libportmidi.so
lrwxrwxrwx@   82 gastove 14 Jun 12:48 librtmidi.so -> /var/home/gastove/Code/open-source/extempore/build/deps-install/lib64/librtmidi.so
.rwxr-xr-x@ 119k gastove 14 Jun  9:46 libstb_image.so

Which gets me the same problem in a slightly different version:

Error copying file "libsndfile.so" to "/var/home/gastove/Code/open-source/extempore/libs/platform-shlibs".
make[2]: *** [CMakeFiles/external_shlibs_audio.dir/build.make:82: CMakeFiles/external_shlibs_audio] Error 1
make[1]: *** [CMakeFiles/Makefile2:370: CMakeFiles/external_shlibs_audio.dir/all] Error 2
make: *** [Makefile:183: all] Error 2
gastove@toolbox [gke->kitchen-stadium] ~/Code/open-source/extempore/build/ on v0.8.7 ✓ λ
$ fd libsndfile.so
deps-install/lib64/libsndfile.so
deps-install/lib64/libsndfile.so.1
deps-install/lib64/libsndfile.so.1.0.29
libsndfile/src/sndfile-build/libsndfile.so
libsndfile/src/sndfile-build/libsndfile.so.1
libsndfile/src/sndfile-build/libsndfile.so.1.0.29

gastove@toolbox [gke->kitchen-stadium] ~/Code/open-source/extempore/build/ on v0.8.7 ✓ λ
$ ln -s (pwd)/deps-install/lib64/libsndfile.so /var/home/gastove/Code/open-source/extempore/build/deps-install/lib/

I am really not good at cmake; this feels like something is missing either a flag or a copy step to get everything in the correct place. I'd be happy to PR it, with a little guidance on where to start digging.

Any advice?

[Gastove]

Having manually shifted the files about, I'm also running in to this point, where the build just hangs:

------------- Extempore --------------
Andrew Sorensen (c) 2010-2020
andrew@moso.com.au, @digego

ARCH           : x86_64-unknown-linux-gnu
CPU            : broadwell
ATTRS          : -sse4a,-avx512bw,+cx16,-tbm,+xsave,-fma4,-avx512vl,+prfchw,+bmi2,+adx,+xsavec,+fsgsbase,+avx,-avx512cd,-avx512pf,-rtm,+popcnt,+fma,+bmi,+aes,+rdrnd,+xsaves,+sse4.1,+sse4.2,+avx2,-avx512er,+sse,+lzcnt,+pclmul,-avx512f,+f16c,+ssse3,+mmx,-pku,+cmov,-xop,+rdseed,+movbe,-hle,+xsaveopt,-sha,+sse2,+sse3,-avx512dq
LLVM           : 3.8.0 MCJIT
Primary        : thread 0
---------------------------------------
INFO: starting utility process...
INFO: server: accepted new connection to utility process
INFO: client: connected to server utility process at localhost:17072
INFO: starting primary process...
INFO: server: accepted new connection to primary process
INFO: client: connected to server primary process at localhost:17073

Evaluating expression: (impc:aot:compile-xtm-file "libs/external/gl/glcore-directbind.xtm")

INFO: started compiling libs/external/gl/glcore-directbind.xtm
Loading xtmbase library... done in 0.724026 seconds
sys:load notification base already loaded
Error: could not find libGL dynamic library

[benswift]

Hey there, yep, the CMake build stuff is... complicated. Especially across Linux distros, it seems. The "missing libGL" error is related to this (although the workaround in this case is to just build with -DEXTERNAL_SHLIBS_GRAPHICS=OFF).

The way to fix it properly is to change the stuff around CMakeLists.txt:713 so that instead of hardcoding the name of the so files it uses variables from the CMake build process for each dep (e.g. libsndfile), returned from the relevant "External Project" CMake build step to get the actual name of the so files. However, there's no standard way of doing this, so it'd be a matter of looking at the CMakeLists.txt for each of those deps and figuring out the name of the relevant variable, then using that in Extempore's CMakeLists.txt. In addition, there might be some "scoping" issues because they're built through ExternalProject_add rather than directly---I don't fully understand all the nuances of that process.

Of course, there might be a nicer way to do this; I know a bit of CMake (and wrote Extempore's build system stuff) but I'm far from a guru.

[benswift]

The good news, though, is that at least we have a CI pipeline these days so it's easier to make sure any changes you've made haven't broken things on other platforms.

[guygastineau]

I am running into this same problem on NixOS. Thank you for making this issue. It is helpful ;)