Holder types¶

By default pybind11 uses a std::unique_ptr to manage references to an object in Python. The object is deallocated when the Python’s reference count for it reaches zero. If objects are shared the holder type can be changed to a std::shared_ptr with the following.

py::class_<basics::Doodad, std::shared_ptr<basics::Doodad>>(m, "Doodad")

To enable transparent conversion of functions that convert between std::shared_ptr<Doodad> and Python Doodad the following macro is defined.

PYBIND11_DECLARE_HOLDER_TYPE(T, std::shared_ptr<T>);

In the case of Doodad this was required because many functions return or accept shared_ptr<Doodad> and, only one type of shared pointer can be associated with a given class using pybind11.

Functions¶

Two helper functions require wrapping at module level. This is easy with pybind11.

m.def("compare", &basics::compare);
m.def("adjacent", &basics::adjacent);

In principle methods work exactly the same, except the .def applies to the module instead.

However, in this challenge there is also a set of methods that take / return a type not exposed to Python (WhatsIt).

void read(WhatsIt const & it);

WhatsIt write() const;

The wrapper should transform these from / to a Python tuple. Adding methods that perform some additional pre / post processing is trivially supported by pybind11 by binding to a lambda function.

.def("write", [](const basics::Doodad &d) { auto tmp = d.write(); return make_pair(tmp.a, tmp.b); })
.def("read", [](basics::Doodad &d, std::pair<std::string, int> p) { d.read(basics::WhatsIt{p.first, p.second}) ; });

The first argument to the lambda is always a reference to the instance (i.e. self in Python). Furthermore pybind11 automatically maps a two element Python tuple to a std::pair which means that we can stick to nice and standard C++ on this side of the fence.

Ideally, we’d like to provide custom converters for a type like WhatsIt that are automatically used whenever it is encountered in a function signature (like Swig’s typemaps). This is unquestionably possible with pybind11 (we can see the feature being used in how it handles STL types), but we have not fully investigated whether this feature has a documented public interface.

Constructors¶

Two constructors are provided (and no default constructor). One takes default, and named, arguments.

explicit Doodad(std::string const & name_, int value_=1);

This is easily wrapped with.

.def(py::init<const std::string &, int>(), py::arg("name"), py::arg("value") = 1)

The second takes a reference to a WhatsIt object.

explicit Doodad(WhatsIt const & it);

Again, the mapping from a tuple to a WhatsIt is done by binding a lambda function. This time to the constructor __init__ overload (pybind11 allows for function overloads which are tried in the declared order).

.def("__init__",
        [](basics::Doodad &instance, std::pair<std::string, int> p) {
            new (&instance) basics::Doodad(basics::WhatsIt{p.first, p.second});
        }
    )

So in Python the following are now equivalent:

challenge.basics.Doodad("bla", 1)
challenge.basics.Doodad("bla")
tmp = ("bla", 1)
challenge.basics.Doodad(tmp)

Const problems¶

The aforementioned get_const() static method returns a shared_ptr<const Doodad> (that can’t be modified). This is exposed to Python with.

.def_static("get_const", &basics::Doodad::get_const)

Which just works because we have changed the holder type for Doodad to shared_ptr<Doodad> as discussed above. Great!

But wait, it turns out that the objects returned by this method are modifiable. Bummer :( Alas pybind11 strips out const modifiers. This issue was bugreported but the developer indicated this will not be fixed because the concept of const is quite alien to Python and it would significantly complicate the library to support it.

Therefore this solution to the C++/Python bindings challenge does not support constness. Note that the Cython solution also had to have an ugly hack (different types to represent const and non-const objects).

Opaque types¶

The challenge also involves wrapping the Secret type as an opaque type in Python. Again this is easy with pybind11.

py::class_<basics::Secret>(m, "Secret");

On the C++ side Secret is a friend class of Doodad and has no publicly accessible constructors. Instead a reference to a Secret instance is returned by the get_secret() method of Doodad. Importantly, the object is owned by Doodad and it remains responsible for destroying the Secret instance.

This can be communicated to pybind11 by specifying a return value policy.

.def("get_secret", &basics::Doodad::get_secret, py::return_value_policy::reference_internal)

Note that while this works perfectly, I personally think objects that cannot outlive their creator are not very Pythonic and are best avoided.

Clone and unique_ptr¶

For the C++ method clone

virtual std::unique_ptr<Doodad> clone() const;

pybind11 allows this to be wrapped directly as you’d expect, even though the holder type is shared_ptr, not unique_ptr:

.def("clone", &basics::Doodad::clone)

This simply constructs a shared_ptr from a unique_ptr, so the opposite would not work.

Overload resolution¶

The add method is overloaded twice with different argument types.

void add(std::shared_ptr<Item> item);
void add(basics::WhatsIt const & it);

In order for the compiler to know which one to select, for the non-lambda binding of add, it is explicitly cast to a function pointer.

.def("add", (void (containers::DoodadSet::*)(std::shared_ptr<basics::Doodad>)) &containers::DoodadSet::add)

When called from Python the two bindings of add are tried in order.

Class attributes¶

The .attr method can be used at module level to set constants and at class level to set class attributes. In this case a class attribute is set for the typedef basics::Doodad Item. Because Doodad is part of the challenge.basics module in Python we instantiate a new module object with an import. When reffering to something in the same module this is not needed and the current module object m can be used instead.

c.attr("Item") = py::module::import("challenge.basics").attr("Doodad");

STL types¶

Like Cython, pybind11 has built in support mapping the basic stl types (e.g. vector, map, tuple, pair etc.) to standard Python types (e.g. list, dict and tuple). Support for tuple and pair is included in the default header, for the rest an extra header needs to be included.

#include <pybind11/stl.h>

We thus can simply expose methods that use such types:

std::vector<std::shared_ptr<Doodad>> as_vector() const;
std::map<std::string,std::shared_ptr<Doodad>> as_map() const;
void assign(std::vector<std::shared_ptr<Doodad>> const & items);

directly as.

.def("as_dict", &containers::DoodadSet::as_map)
.def("as_list", &containers::DoodadSet::as_vector)
.def("assign", &containers::DoodadSet::assign);

Iterators¶

In order to support standard iteration over this DoodadSet container (e.g. with for) the DoodadSet wrapper defines the __iter__ special function.

.def("__iter__", [](containers::DoodadSet &ds) { return py::make_iterator{ds.begin(), ds.end()}; }, py::keep_alive<0,1>())

This function returns a special Python iterator type that just moves the C++ iterators it holds. In addition it uses the keep_alive call policy to make sure that the DoodadSet container cannot be destroyed while an iterator pointing to it still exists.

On symbol visibility¶

As an interesting side-note, when using cross module types with pybind11 it is also important to set the symbol visibility to default (as opposed to hidden). This can be done with the compiler option -fvisibility=default but typically isn’t necessary (since it is the default), but some examples of pybind11 online explicitly set visibility to hidden (to get smaller binaries) which creates problems when using cross-module types.

We might be able to address this by adding visibility hints to our C++ code itself, rather than using compiler options to set visibility at the scale of a full source file.