CPS should support CMake's IMPORTED_LINK_DEPENDENT_LIBRARIES

[mwoehlke-kitware]

CMake has a target property, IMPORTED_LINK_DEPENDENT_LIBRARIES, that describes libraries that may need to be "accessible" at link time, but are not [link] requirements. In particular, these may be private link dependencies that should not be fed to the linker as such, but whose locations may need to be in the linker's search paths.

These are likely also runtime requirements (see also #17, #20), and conceptually sit somewhere in between runtime requirements and link requirements.

CPS should support expressing these in some manner.

See also cmake#27501.

[memsharded]

There are functional tests that implement CMake->CPS->CMake roundtrip for transitive shared libraries that reproduce this case in https://github.com/conan-io/conan/blob/b2f4a0b3faedb72d37076bb83e09717baae71b8b/test/functional/toolchains/cmake/test_cps.py#L290, that are at the moment skipped for the Linux platform, because it fails.

[mwoehlke-kitware]

WDYT about dylib_requires?

[memsharded]

WDYT about dylib_requires?

Sounds great.

[memsharded]

To complement the information above, and possible discussion:

• Tickets Add Runtime-Environment #17, Runtime dependencies #20, are mostly about "runtime" dependencies.
• CMake ticket https://gitlab.kitware.com/cmake/cmake/-/issues/27501 is about build time (link time to be accurate) requirements, as the relationship between the 2 shared libraries in Linux with the mainstream gcc/ld linker, that requires a -rpath-link linker flag, otherwise the build fails
• It seems that the proposed dylib_requires in this ticket automatically implies a "runtime" requirement too. I don't see a way that the dylib_requires wouldn't need the runtime too
• On the other hand, there are some scenarios that pure "runtime" dependencies, without the --rpath-link or dylib_requires requirement could happen. For example for plugin-like systems or explicit user code shared library loading.
• That would imply that for tickets Add Runtime-Environment #17, Runtime dependencies #20, another type of requirement like run_requires could be needed to model those cases

[nickelpro]

I agree with @memsharded that dylib_requires implies a runtime requirement and this isn't a runtime requirement.

This is very, very specifically about how ld.bfd resolves DT_NEEDED at build time. The name shouldn't extend further than that.

Proposing: dylib_link_depends, mirroring CMake's nomenclature. Dropping "requires" makes it distinct and harder to mix up with link_requires, a true linkage requirement.

[nickelpro]

Alternatively, we go whole-hog on what this is actually for and just call it rpath-links or similar.

[memsharded]

Proposing: dylib_link_depends

Sounds good to me too.
But on the other hand, dylib_link_depends still implies runtime_requires, so it is also a kind of "requires" in that sense.
But I like how dylib_link_depends express that the shared library was linking with the other transitive shared library, so totally good for me.

rpath-links

That sounds a bit too specific for the specific linker flag.
Probably, when a Windows DLL depends on another Windows DLL, even if not a requires, not a link_requires, the dylib_link_depends could express this relationship, as opposed to a pure runtime_requires that could come from a plugin-like or dynamically loaded (which could avoid failing at runtime).

Part of this current undefinition comes from the package type module.
The issue is that the same shared library, like a .so inside the same package, can be both things, it can be linked as a dylib, but it can also be dynamically dlopen loaded as a module by a different consumer. It cannot always be defined at the CPS file of the shared library, that is more a property of the consumer "requirement" than a property of the package itself. Maybe even 2 different consumers in the same dependency graph require the shared library differently, and one of them fully links with it, while the other does a dlopen to test for the capability dynamically.

It seems that this is why this is required at the requires level. Taking that into account, maybe:

• dylib_link_requires: A shared lib linking with another shared lib. It will requires rpath-link in ld linker. It also implies dylib_run_requires automatically.
• dylib_run_requires: A shared lib that will need the other shared lib at runtime, either because of direct linkage, or by indirect dynamic loading with dlopen.

I think we probably want for clarity/completeness to map the inverse problem, from different artifacts relationships to CPS "requires" types:

• App -> static library
• App -> dynamic library
• Static library -> Static library (differentiating when the headers are transitive or not (compile_requires)
• Static library -> Shared library
• Static library -> Header library
• Header library -> ...

With the extra exercise to consider the transitivity of several levels.
I can try to provide some summary for this when I manage to continue with the testing I mentioned above.

[mwoehlke-kitware]

I agree with @memsharded that dylib_requires implies a runtime requirement and this isn't a runtime requirement.

Er... it is. (I think you might also be mistaking what @memsharded is claiming.)

I created a simple test project, consisting of one library (bar) which privately links another (foo). On installation, I get a libbar.so which has an unresolved reference to libfoo.so:

$ ldd libbar.so 
        linux-vdso.so.1 (0x00007f0bb2f46000)
        libfoo.so => not found
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f0bb2c00000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f0bb2b11000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f0bb2eef000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f0bb2920000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f0bb2f48000)

IMO, regardless of any RPATH shenanigans CMake may employ, a consumer that tries to dlopen this library is going to fail because libfoo.so can't be found. In my book, that's a runtime requirement.

Now... compile_requires is an exception, but other than that, I think we have a fairly tidy progression:

• runtime_requires: In an RPATH-less environment (i.e. CMake installs), the requirement must appear a search path; i.e. a default search path, or in [[DY]LD_LIBRARY_]PATH or similar.
• bikeshed_requires: Same as runtime_requires, plus the linker may need to know about the requirement in order to resolve transitive library loads.
• link_requires: Same as bikeshed_requires, plus the linker should use the library to resolve symbols; i.e. should be passed to the linker as a normal link dependency.
• requires: Same as link_requires, plus any usage requirements (includes, definitions, etc.) should be communicated to the compiler when compiling any objects which reference the providing component.

There are nuances when talking about non-dylib components, especially if we throw "pure data" components into the mix (which aren't really supported at present, but make sense in a runtime_requires world), but if we look at the situation for purely dylib components, I think the proposed attribute slots neatly in between runtime_requires and link_requires.

Note also another similarity: both runtime_requires and bikeshed_requires describe requirements that may be superfluous if the component lives in a "default" search path.

In any case:

• I have a fairly strong preference for <something>_requires as the name.
• I'd prefer "one word" (or, rather, something with no underscore), but I don't feel as strongly on that point.
  • I'd also prefer avoiding <something>_link_requires unless it's clear that the name is a "lesser form" of the existing link_requires. (Or, rename link_requires, but I'd rather not do that, either, without a truly compelling proposal.)
• I agree that I want something less implementation-specific than "rpath".

So, the question here is how to express something that is a 'link requirement', but only for transitive link resolution, not in the usual sense. It's worth noting that this is specifically a dynamic loader requirement that sometimes spills over onto the linker. That's why I'm inclined to something like dylib_requires or maybe dyld_requires, maybe even leaning toward the latter.

Part of this current undefinition comes from the package type module.

It's probably best to think of a module as a subset of a dylib; specifically, one which a) is not intended to be fed to a linker, and b) does not have an import library. It's that second point that's the main reason module is a separate type. They also exist in a weird space in the absence of runtime requirements, since there isn't a lot you can do with them. (In practice, I think "ask for their location" is about it...) Except they can still have usage requirements, so they actually sit in a weird spot in between dylib or even archive components and interface components.

Taking that into account, maybe [...]

I think we may be proposing the same attributes, except your dylib_run_requires is my runtime_requires, and your dylib_link_requires is my bikeshed_requires.

[mwoehlke-kitware]

Bit of a wrinkle... AFAICT, CMake's IMPORTED_LINK_DEPENDENT_LIBRARIES also supports paths.

Do we want to also add bikeshed_libraries (e.g. dyld_libraries)?

[nickelpro]

I don't think it's a wrinkle. This is specifically about "things you need to pass to -rpath-link". It follows that these may be shared libraries, but they might also be raw paths.

If we want to reduce the complexity on implementers, we can make the entries string-or-object. String is a component representing a shared lib, object is something like:

{
  name: <string>
  is_path: <bool : default False>
}

[mwoehlke-kitware]

Ugh, that's totally novel and harder to implement. Even if we require the object in all cases; object or string is in every way worse for implementations. We already have link_libraries. (We also already discourage use of the same.)