Split a library into multiple modules cannot work properly on Windows (msys2/mingw)

[souk4711]

Issue Description

The Qt library has multiple modules, such as QtCore, QtGui, etc. So I create a gem for each module, such as rubyqt6-qtcore, rubyqt6-qtgui, etc. (rubyqt6-qtgui depends on rubyqt6-core)

It works fine on Linux. But cause some runtime exceptions, like Type QObject is not defined ..., on Windows. Seems the DLL does not share global variables, and does not export any symbols. ref - difference between .so and .dll

In rubyqt6-qtcore:

void Init_qobject(Rice::Module rb_mQt6QtCore)
{
    rb_cQObject = define_class_under<QObject>(rb_mQt6QtCore, "QObject")
        // define methods here.
 }

In rubyqt6-qtgui:

void Init_qaction(Rice::Module rb_mQt6QtGui)
{
    rb_cQAction = define_class_under<QAction, QObject>(rb_mQt6QtGui, "QAction")
        // define methods here

The program load qtcore.so successfully, then load qtgui.so, but raises an exception.

I think, on windows, each DLL holds a copy of Rice::detail::Registries::instance. The Registries::instance in qtgui.so cannot getType of QObject which registered in qtcore.so.

---

First Try (failed)

Try to redefine class QObject on qtgui.so.

It loads successfully. But got Wrong argument type. Expected RubyQt6::QtCore::QObject. Recevied RubyQT6::QtCore::QObject exception when invoking QObject#meth (the method QObject#meth defined in the copy in qtcore.so).

void Init_qaction(Rice::Module rb_mQt6QtGui, rb_mQt6QtCore)
{
    define_class_under<QObject>(rb_mQt6QtCore, "QObject")
      .define_constructor<Constructor>(...);  # required, cuz `klass.undef_creation_funcs();`
      # no more method define.

    rb_cQAction = define_class_under<QAction, QObject>(rb_mQt6QtGui, "QAction")
        ...

---

Second Try (failed)

Try to DLL export Rice::detail::Registries::instance in qtcore.so, and DLL import Rice::detail::Registries::instance in qtgui.so. I think this makes Registries::instance only one copy.

#ifdef _WIN32
#  ifdef RUBYQT6_BUILD_QTCORE_LIB
#    define RICE_EXPORT __declspec(dllexport)
#    define RICE_LIBRARY_INIT 1
#  else
#    define RICE_EXPORT __declspec(dllimport)
#    define RICE_LIBRARY_INIT 0
#  endif
# ...
#endif

namespace Rice::detail
{
  class RICE_EXPORT Registries
  {
  public:
    static RICE_EXPORT Registries instance;
....

#if RICE_LIBRARY_INIT == 1
  //Initialize static variables here.
  inline Registries Registries::instance;
#endif
}

But it still doesn't work. Seems Data_Type<T>::check_is_bound() returns false because klass_ is nil. And it does not seem possible to DLL export Data_Type<T> because Rice automatically defines some raw types (e.g. Pointer<int>, Pointer<QObject>) when defining methods.

---

Any advice on how to get the rice to work with multiple modules on Windows ?

Thx.

[cfis]

That seems to me a good thing! The more likely scenario is two different extensions developed by two different authors that use Rice - you don't want them to step on each other. What if they both define the same class names? What if they use different versions of Rice? Etc.

I'm surprised it works on Linux. Are you sure? For example, each library should have its own Registries::instance. I wouldn't think the second library loaded would use the one from the first library, but that must be what's happening? But from your description, they are not sharing instantiated Data_Type<T> classes.

Anyway, I would say don't split up the extension into separate modules. I don't think its feasible. In fact (and sorry I know you won't like this) I think Rice should actively prevent this by compiling with -fvisibility=hidden on non-Windows OSes.

But let's say you really, really wanted to try this. You can work around the type registry issue by not throwing an error on unknown types.

Next you have to workaround Data_Type<T>::klass() issue. Each library is going to instantiate its own Data_Type<T> - that is unavoidable. I don't see a way to fix this in C++. But maybe via Ruby.

So let's use your example, Data_Type<QObject>. Assume it is setup in the first library via define_klass. But the second library will also instantiate the template Data_Type<QObject> - so it has its own copy. Of course, its static klass and ruby_data_type fields would be unset.

But maybe you could hack it by setting them manually. You would get the klass via Object.const_get (pass it the Ruby class name). From there, you could get the data_type via RTYPEDDATA(obj) and then grab the rb_data_type_t from the type field (well you would need an instance of the Ruby klass, so an object). Then set those static fields and in theory you would have a working Data_Type that is bound.

But you would have to hard-code that for every single class you are wrapping. You'd probably want a static `Data_Type::from_kalss(VALUE klass) method or some such. And the load order of the libraries becomes critical.

I think you are better off not doing this, but if you try, let me know if it works.

[souk4711]

I'm surprised it works on Linux. Are you sure? For example, each library should have its own Registries::instance. I wouldn't think the second library loaded would use the one from the first library, but that must be what's happening? But from your description, they are not sharing instantiated Data_Type classes.

On Linux, GCC build c++ extension with following ldflags -rdynamic -Wl,-export-dynamic -Wl,--no-as-needed (the flags defined in RbConfig::CONFIG), so every library will export all symbols.

If I have understand right. Tho each library has its own Registries::instance, but they have same symbols, like _ZGVN4Rice6detail10Registries8instanceE, so the dynamic linker will only use the first found version, and all Data_Type<T> will be registered in the same instance.

---

Anyway, I would say don't split up the extension into separate modules. I don't think its feasible. In fact (and sorry I know you won't like this) I think Rice should actively prevent this by compiling with -fvisibility=hidden on non-Windows OSes.

The qt lib is a fairly big, and some modules are optional, such as qtwebkitcore, and some are only available on specific platforms, such as qtdbus. I also plan to add some unofficial libraries, such as kdelibs. So it is not possible to keep all the stuffs in one rubygem.

---

But let's say you really, really wanted to try this. You can work around the type registry issue by not throwing an error on unknown types.
...

Support for the Windows is a very low priority. I mainly use Linux. :-)

Now that I think about it, I could use an extern global tracker to trace all registered klasses and hack the Data_Type<T>::bind function to map all c++ Data_Type<QObject> defined in different libraries to the same ruby QObject klass ?

---

What if they use different versions of Rice? Etc.

When using bundler, the extensions will be compiled with the same Rice version.

That seems to me a good thing! The more likely scenario is two different extensions developed by two different authors that use Rice - you don't want them to step on each other. What if they both define the same class names?

Suppose you have two math libs, and they both define the same class, such as std::vector<int>, it will not be a problem on Linux in most cases, cuz the underlying c++ Data_Type<> is always refers to the first version.

But on Windows, the lib authors must use different ruby class name. The following code will cause an issue on Windows ():

extern "C" void Init_libaaa() {
    Rice::define_global_function("libaaa_invoke_stl_vector", [](std::vector<int>) {});
};

extern "C" void Init_libbbb() {
    Rice::define_global_function("libbbb_invoke_stl_vector", [](std::vector<int>) {});
};

# raise exception: Wrong argument type. Expected Std::Vector<Int>, Got Std::Vector<Int>
libaaa_invoke_stl_vector(Std::Vector<Int>.new) 
libbbb_invoke_stl_vector(Std::Vector<Int>.new) 

It failed cuz Data_Type<std::vector<int>>() will invoke rb_define_class twice, and make ruby Std::Vector<Int> refers the later one. To fix this, we should use following code:

extern "C" void Init_libbbb() {
    Rice::define_vector<int>("BbbIntVector");
    Rice::define_global_function("libbb_invoke_stl_vector", [](std::vector<int>) {});
};

libaaa_invoke_stl_vector(Std::AaaVector<Int>.new) 
libbbb_invoke_stl_vector(Std::BbbVector<Int>.new) 

the rice/api.hpp also not work properly.

[cfis]

When using bundler, the extensions will be compiled with the same Rice version.

You can't make that assumption. You could have 2 different extension that use different versions of Rice (they have different versions specified in their gemspecs). You could then have a 3rd extension that simply copied the Rice headers into its source tree. So now you have 3 different versions of Rice in 3 different extensions. Bundler will do nothing to fix or avoid that.

Suppose you have two math libs, and they both define the same class, such as std::vector, it will not be a problem on Linux

Unless one was built against Rice 4.7 and the other one was built against 4.8 which added a new method to std::vector. If the extension that uses Rice 4.7 loads first, then you'll get an error in the second extension that uses 4.8 when it calls a non-existent method (although it would have been defined if it was loaded first).

I agree the shared classes (STL and API) on Windows are a problem due to the shared Std namespace. You could do what I wrote in a comment above to fix that (manually set klass and ruby_data_type on Data_Type<std::vector> or whatever) but that will be super brittle across Rice versions (what I wrote above).

I guess one way to avoid this is not have a standard Std namespace at all and have the extension writer specify a "default" namespace to add classes too it.

[souk4711]

You can't make that assumption. You could have 2 different extension that use different versions of Rice (they have different versions specified in their gemspecs). You could then have a 3rd extension that simply copied the Rice headers into its source tree. So now you have 3 different versions of Rice in 3 different extensions. Bundler will do nothing to fix or avoid that.
...
I guess one way to avoid this is not have a standard Std namespace at all and have the extension writer specify a "default" namespace to add classes too it.

Yes. Allow developer using a custom namespace could resolve conflicts between two extensions. Maybe intro a RICE_NAMESPACE macro ?

[souk4711]

I think you are better off not doing this, but if you try, let me know if it works.

1. Add Data_Type::declare_class_under

See PR#358

2. Export a global variable RubyQt6_RTypedData_Collector

#define RUBYQT6_DECL_DLLEXPORT __declspec(dllexport)
#define RUBYQT6_DECL_DLLIMPORT __declspec(dllimport)

#ifdef _WIN32
#  ifdef RUBYQT6_BUILD_QTCORE_LIB
#    define RUBYQT6_QTCORE_EXPORT RUBYQT6_DECL_DLLEXPORT
#  else
#    define RUBYQT6_QTCORE_EXPORT RUBYQT6_DECL_DLLIMPORT
#  endif
#else
#  define RUBYQT6_QTCORE_EXPORT
#endif

struct RUBYQT6_QTCORE_EXPORT RubyQt6_RTypedData_Collector
{
    RUBYQT6_QTCORE_EXPORT static RubyQt6_RTypedData_Collector *instance();
    static RubyQt6_RTypedData_Collector* instance_;

    std::map<VALUE, rb_data_type_t*> map_;
};

#ifdef RUBYQT6_BUILD_QTCORE_LIB
inline RubyQt6_RTypedData_Collector *RubyQt6_RTypedData_Collector::instance()
{
    if (instance_ == nullptr) { instance_ = new RubyQt6_RTypedData_Collector; }
    return instance_;
}
inline RubyQt6_RTypedData_Collector *RubyQt6_RTypedData_Collector::instance_ = nullptr;
#endif

3. Trace define_class_under, and add klass & RTypedData to RubyQt6_RTypedData_Collector

template<typename T, typename Base_T = void>
inline Data_Type<T> define_qlass_under(Object parent, Identifier id)
{
    Class superKlass = get_superklass<Base_T>();
    Data_Type<T> klass = define_class_under<T, Base_T>(parent, id, superKlass);

    auto ret = RubyQt6_RTypedData_Collector::instance()->map_.insert({ klass.object_id(), klass.ruby_data_type() });
    if (!ret.second && (*ret.first).second != klass.ruby_data_type()) {
        std::string msg = "klass: " + klass.to_s().str();
        Q_ASSERT_X(false, "define_qlass_under", msg.c_str());
    }

    return klass;
}

extern "C" void Init_qtcore()
{
    return Rice::detail::cpp_protect([] {
        Rice::Module rb_mQt6 = Rice::define_module("RubyQt6");
        Rice::Module rb_mQt6QtCore = define_module_under(rb_mQt6, "QtCore");

        // define_class ...
        Init_qobject(rb_mQt6QtCore);
        ....
    });
}

4. declare_class_under after DLL loading ASAP

template<typename T, typename Base_T = void>
inline Data_Type<T> declare_qlass_under(Object parent, Identifier id)
{
    Class klass = parent.const_get(id).value();

    auto ret = RubyQt6_RTypedData_Collector::instance()->map_.find(klass.object_id());
    if (ret == RubyQt6_RTypedData_Collector::instance()->map_.end()) {
        std::string msg = "klass: " + klass.to_s().str();
        Q_ASSERT_X(false, "declare_qlass_under", msg.c_str());
    }

    return declare_class_under<T, Base_T>(parent, id.c_str(), (*ret).second);
}

extern "C" void Init_qtgui()
{
    return Rice::detail::cpp_protect([] {
        Rice::Module rb_mQt6 = Rice::define_module("RubyQt6");
        Rice::Module rb_mQt6QtCore = Rice::define_module_under(rb_mQt6, "QtCore");
        Rice::Module rb_mQt6QtGui = define_module_under(rb_mQt6, "QtGui");

        // `declare_qlass/declare_qlass_under` after DLL loading ASAP
        declare_qlass_under<QObject>(rb_mQt6QtCore, "QObject");
        ...

        // define_class ...
        Init_qbrush(rb_mQt6QtGui);
        ...

        // DEV ONLY.
        // Print all undeclared class.
        // Rice::detail::Registries.types.validateType();
    });
}

---

Now, it should work without any Type is not defined or Type is not registered exception. (Only tested on small samples.)

[cfis]

Another approach is add a hidden attribute the Ruby Class, which would be the rb_data_type_t. Data_Type could then check for that in is defined.

But neither of these solutions solve the problem of libraries built with different versions of Rice (100% guaranteed to happen) or different versions of C++ library. That probably won't happen, but imagine one extension built with g++ and another with Clang. They have different C++ libraries underneath, which I assume means you cannot share things like std::vector across them.

[souk4711]

Another approach is add a hidden attribute the Ruby Class, which would be the rb_data_type_t. Data_Type could then check for that in is defined.

So, you prefer to use klass to trace RbDataType instead of RubyQt6_RTypedData_Collector, and automatically bind it in the DataType::bind method ?

I will try it later.

But neither of these solutions solve the problem of libraries built with different versions of Rice (100% guaranteed to happen) or different versions of C++ library. That probably won't happen, but imagine one extension built with g++ and another with Clang. They have different C++ libraries underneath, which I assume means you cannot share things like std::vector across them.

Plan to add a RICE_NAMESPACE macro to solve this. E.g.

#ifndef Rice__configmacros__hpp_
#define Rice__configmacros__hpp_

#define STR_EXPAND(tok) #tok
#define STR(tok) STR_EXPAND(tok)

#ifndef RICE_NAMESPACE
  #define RICE_NAMESPACE Rice
  #define RICE_DEFINE_MODULE_RICE define_module( STR(RICE_NAMESPACE) );
  #define RICE_DEFINE_MODULE_RICE_STL define_module( STR(RICE_NAMESPACE) ).define_module("Std");
#else
  #define RICE_DEFINE_MODULE_RICE define_module( STR(RICE_NAMESPACE) );
  #define RICE_DEFINE_MODULE_RICE_STL define_module("Std");
#endif

#define RICE_DETAIL_NAMESPACE RICE_NAMESPACE::detail
#define RICE_STL_NAMESPACE RICE_NAMESPACE::stl

#define RICE_PREPEND_NAMESPACE(name) ::RICE_NAMESPACE::name
#define RICE_USE_NAMESPACE using namespace ::RICE_NAMESPACE;
#define RICE_BEGIN_NAMESPACE namespace RICE_NAMESPACE {
#define RICE_END_NAMESPACE }

#define RICE_DETAIL_USE_NAMESPACE using namespace ::RICE_DETAIL_NAMESPACE;
#define RICE_DETAIL_BEGIN_NAMESPACE namespace RICE_DETAIL_NAMESPACE {
#define RICE_DETAIL_END_NAMESPACE }

#define RICE_STL_USE_NAMESPACE using namespace ::RICE_STL_NAMESPACE;
#define RICE_STL_BEGIN_NAMESPACE namespace RICE_STL_NAMESPACE {
#define RICE_STL_END_NAMESPACE }

#endif  // Rice__configmacros__hpp_

[cfis]

Ok, let's do this one step at a time. I agree with you that it probably makes most sense to have the ability to expose the type registry. By default it should be hidden and not accesible outside of an extension. However, in cases like yours we should make it possible to export it from one of the libraries and import it from the others.

I'd rather do this using a new Rice::Config class that is a constexpr and have various settings (export/import type registry, std ruby namespace, rice ruby namespace) then C macros. But I don't know if that is possible.

So 1 PR to add a Config class with the first setting controlling the visibility of the type registry. I believe you can just export the whole class versus having to do individual methods (although that would also work).

Once that works, then we can update Data_Type to search it when its enabled.

[souk4711]

Ok, let's do this one step at a time. I agree with you that it probably makes most sense to have the ability to expose the type registry. By default it should be hidden and not accesible outside of an extension. However, in cases like yours we should make it possible to export it from one of the libraries and import it from the others.

Yes, in most cases, it should be ok.

But in my case, hiding the symbol in Linux will result to force using the method export it from one of the libraries and import it from the others. Since i can modify make_rice_headers.rb to generate a custom rice version, it will not be a big problem for me.

I'd rather do this using a new Rice::Config class that is a constexpr and have various settings (export/import type registry, std ruby namespace, rice ruby namespace) then C macros. But I don't know if that is possible.

So 1 PR to add a Config class with the first setting controlling the visibility of the type registry. I believe you can just export the whole class versus having to do individual methods (although that would also work).

Once that works, then we can update Data_Type to search it when its enabled.

I agree that using the Rice::Config class to set the rice/std ruby namespace is better than using C macros. Not sure if we need to export the entire class. It should only be DLL only, IMO.

---

In summary, we need to do the following things:

1. Fix exported symbol/name conflict between the extensions built against two different Rice version

If we don't want to use the C macro to implement this feature ( PR #360 ). I think we can implenent it through the following steps:

• Add the cppflags -fvisibility=hidden or whatever to prevent the default export behavior on Linux.
• Add the C++ class Rice::Config to allow developer to set the rice/std namespace for the current extension.
• Update Module rb_mRice = define_moudule('rice') lines, using the name defined in Rice::Config

I don't have much experience with the C++ language. I'm not sure how to correctly set CPPFLAGS in Linux/MacOS, and use the constexpr. :-(

2. Provide a mechanism to allow Data_Type bind to (klass, ruby_data_type_t) defined in other DLL.

This feature allows us to split a library into multiple modules on Windows (Also on Linux built with -fvisibility=hidden cppflags). PR #358 just modifies Data_Type<T>::bind signature and expose it to public. With these small changes, the extension developers can import (klass, ruby_data_type_t) from other DLL, tho they still need to implement other stuffs.

If we want to implement the full mechanism in Rice lib, we need to:

• Provide a way to trace all (klass, ruby_data_type_t).
  • Use RubyQt6_RTypedData_Collector, but this requires extra compiler options, e.g. RUBYQT6_BUILD_QTCORE_LIB.
  • Or add a hidden attribute to the Ruby class, not sure how to define the underlying type.
• Update Data_Type<T>::bind or whatever, make it automatically register the ruby_data_type_t.
• Update Data_Type<T>::is_defined or whatever, make it automatically create a new Data_Type<T> that bound to the global ruby_data_type_t
  • If this is not possible, the extension developers need to manually call declare_class_under after the DLL has been loaded.
• Of course, we need a global flag defined in Rice::Config to turn on/off this behavoirs.

It will be much more developer-friendly than PR #358. Ideally, developers would only need to set a flag, to use this feature out-of-box. Tho it may not easy to implement, and make Rice lib a bit more complex.

[cfis]

I think that is a good summary of the needed steps - thanks for writing it up. I want to first get Rice 4.8 out the door, and get some changes upstreamed to Cmake. But I can hopefully give a try to #1 this weekend. Or if you want to give it a go I'm happy to review.