Many existing C++ codes use SWIG to automatically generate bindings for Python, but over the last years pybind11 has gained a lot of popularity. Most newer projects rely on pybind11 and even large exiting code bases, such as, tensorflow1, made the transition to pybind11. While SWIG is extremely powerful with its support for a large number of languages, it has its shortcomings w.r.t. supporting modern C++ standards. There seems to be little development in this direction and with the increasing complexity of C++ it is unlikely SWIG will ever catch up. The scope of pybind11 is much more focused: the goal is to enable the generation of Python bindings employing modern C++ itself. As such, pybind11 does not have to process and transform C++ code to generate the glue code for the Python bindings, but rather pybind11 provides a C++ library to enable the developer to generate this glue code easily, as part of regular C++ library.
While creating Python modules using pybind11 is surprisingly easy, things get tricky when interacting with libraries using SWIG to generate their Python bindings. As long as the modules generated via SWIG and pybind11 interact only on the Python level, the method the bindings were generated with does not matter. But as soon as a C++ library wrapped to Python via pybind11 tries to interact with a library wrapped to Python via SWIG by exchanging C++ objects via the Python layer, incompatibilities arise due to the different methods of wrapping the underlying C++ types.
Assume we use the following code to generate the Python module pybind_example
via pybind11:
#include "pybind11/pybind11.h"
namespace py = pybind11;
namespace pybind_example {
class A {
public:
auto get_answer() -> int { return 42; }
};
auto foo(A *a) -> int {
return a->get_answer();
};
PYBIND11_MODULE(pybind_example, m) {
py::class_<A>(m, "A").def(py::init<>()).def("get_answer", &A::get_answer);
m.def("foo", &foo);
}
} // namespace pybind_exampleThere we have the class A with its (default) constructor and the member
function get_answer exposed via the Python module pybind_example, along with
the function foo taking a pointer to an instance of A and calling the
get_answer member function. Using this trivial module in Python is then
straight forward:
import pybind_example as pe
a = pe.A()
pe.foo(a)All the boilerplate code required is automatically generated by pybind11 and
wrapping additional C++ classes and functions is easily done. Also interfacing
with certain builtin Python types (e.g., str or list) is directly supported
by pybind11, as well as support and, where applicable, conversion to C++
standard data structures, such as, std::string or std::vector (including
handling of numpy arrays!).
Arbitrary C++ libraries can be included for the implementation of the required functionality and, if necessary, the corresponding types can be exposed via the Python API by wrapping them using pybind11.
Now suppose we have an existing C++ library A wrapped to Python via SWIG, resulting in the Python module a_swig. In our C++ library B, which we wrap to Python using pybind11 (giving us the module b_pybind), we now want to use A and, to make matters worse, not only internally, but as part of the public API. This is a common use case for a framework library with a series of plugin components. Consider the following code snippets:
namespace a {
class Bar {
// [...]
};
}// include header provided by external library A
#include "Bar.hpp"
namespace b {
class Foo {
public:
explicit Foo(a::Bar* bar);
// [...]
};
}#include "pybind11/pybind11.h"
#include "Foo.hpp"
namespace py = pybind11;
namespace b {
PYBIND11_MODULE(b_pybind, m) {
py::class_<Foo>(m, "Foo").def(py::init<a::Bar*>());
}
}While the Python module compiles just fine, when trying to create a new instance
of Foo on the Python side via
import a_swig as a
import b_pybind as b
bar = a.Bar()
foo = b.Foo(bar)an exception similar to the one below is triggered due to incompatible types:
TypeError: __init__(): incompatible constructor arguments. The following argument types are supported:
1. b_pybind.Foo(arg0: Bar)
Invoked with: <a_swig.Bar; proxy of <Swig Object of type 'Bar *' at 0x7fd79e3f12a0> >
So while we have Python bindings for both libraries, it is not possible to pass
instances of the underlying C++ types between the two libraries as the two
wrapping types (SWIG and pybind11) generate incompatible types on the Python
side. One could be tempted to additionally wrap a::Bar using pybind11, but
this would not resolve the problem: Instances created in Python via the SWIG
bindings from the module a_swig would have a different type compared to
instances created for the same underlying class a::Bar but via the pybind11
bindings. But, as we shall see in the next section, there is a solution!
Note: It is the responsibility of the user to add the pybind11 headers to the header-search path of his build-system. swigbind11 is not shipped with a pybind11 clone for this purpose!
Interfacing of libraries wrapped via SWIG and pybind11 is quite easy using
swigbind11. It allows the automatic conversion between SWIG-wrapped objects
and the corresponding C++ types by means of using a custom pybind11 type
caster. This type caster transforms a Python object holding, e.g., a a::Bar
into a std::shared_ptr<a::Bar> and vice versa. A std::shared_ptr<> is required
to properly track the lifetime of the underlying object on the Python and C++ side
simultaneously.
#include "pybind11/pybind11.h"
#include "swigbind11/swigbind11.hpp"
#include "Foo.hpp"
namespace py = pybind11;
SWIGBIND11_TYPE_CASTER(a::Bar, "a_swig.Bar");
namespace b {
PYBIND11_MODULE(b_pybind, m) {
py::class_<Foo>(m, "Foo").def(py::init<std::shared_ptr<a::Bar>>(), py::arg("bar"));
}
}The only other required change is to switch the interface of b::Foo to accept
a std::shared_ptr<a::Bar> instead of a raw pointer. Using a smart pointer here
at the interface of C*+ and Python instead of a raw pointer is often a good idea
anyway, as this avoids hard to debug issues with interference between the
automatic Python garbage collection and the use of the underlying object on both
sides of the interface.
In addition to the helper macro to define the custom type casters, swigbind11 also provides some low-level functions to perform type conversions:
Conversion from Python object to underlying C++ object:
template<typename T>
std::shared_ptr<T> swig_py_cast(pybind11::handle obj, std::string_view python_type_name)Wrapping an existing C++ object using the externally defined SWIG bindings in a Python object:
template<typename T>
pybind11::handle py_swig_cast(std::shared_ptr<T> obj, std::string_view python_type_name)template<typename T>
pybind11::handle py_swig_cast(std::unique_ptr<T> obj, std::string_view python_type_name)In all cases the user has to provide the correct mapping between the C++ type and the corresponding Python type generated by SWIG. If these two types do not match up, there is no way for swigbind11 to detect this, and one will (as a best case scenario) encounter segmentation faults and similar errors.
A demonstration of the usage of swigbind11 for a real-world example (plugin for the Flow Simulator) can be found in the example directory. To make the example more self-contained, pybind11 in included as a third-party library.