| | Functions |
| | ######### |
| |
|
| | Before proceeding with this section, make sure that you are already familiar |
| | with the basics of binding functions and classes, as explained in :doc:`/basics` |
| | and :doc:`/classes`. The following guide is applicable to both free and member |
| | functions, i.e. *methods* in Python. |
| |
|
| | .. _return_value_policies: |
| |
|
| | Return value policies |
| | ===================== |
| |
|
| | Python and C++ use fundamentally different ways of managing the memory and |
| | lifetime of objects managed by them. This can lead to issues when creating |
| | bindings for functions that return a non-trivial type. Just by looking at the |
| | type information, it is not clear whether Python should take charge of the |
| | returned value and eventually free its resources, or if this is handled on the |
| | C++ side. For this reason, pybind11 provides a several *return value policy* |
| | annotations that can be passed to the :func:`module::def` and |
| | :func:`class_::def` functions. The default policy is |
| | :enum:`return_value_policy::automatic`. |
| |
|
| | Return value policies are tricky, and it's very important to get them right. |
| | Just to illustrate what can go wrong, consider the following simple example: |
| |
|
| | .. code-block:: cpp |
| |
|
| | /* Function declaration */ |
| | Data *get_data() { return _data; /* (pointer to a static data structure) */ } |
| | ... |
| |
|
| | /* Binding code */ |
| | m.def("get_data", &get_data); // <-- KABOOM, will cause crash when called from Python |
| |
|
| | What's going on here? When ``get_data()`` is called from Python, the return |
| | value (a native C++ type) must be wrapped to turn it into a usable Python type. |
| | In this case, the default return value policy (:enum:`return_value_policy::automatic`) |
| | causes pybind11 to assume ownership of the static ``_data`` instance. |
| | |
| | When Python's garbage collector eventually deletes the Python |
| | wrapper, pybind11 will also attempt to delete the C++ instance (via ``operator |
| | delete()``) due to the implied ownership. At this point, the entire application |
| | will come crashing down, though errors could also be more subtle and involve |
| | silent data corruption. |
| |
|
| | In the above example, the policy :enum:`return_value_policy::reference` should have |
| | been specified so that the global data instance is only *referenced* without any |
| | implied transfer of ownership, i.e.: |
| |
|
| | .. code-block:: cpp |
| |
|
| | m.def("get_data", &get_data, return_value_policy::reference); |
| |
|
| | On the other hand, this is not the right policy for many other situations, |
| | where ignoring ownership could lead to resource leaks. |
| | As a developer using pybind11, it's important to be familiar with the different |
| | return value policies, including which situation calls for which one of them. |
| | The following table provides an overview of available policies: |
| |
|
| | .. tabularcolumns:: |p{0.5\textwidth}|p{0.45\textwidth}| |
| |
|
| | +--------------------------------------------------+----------------------------------------------------------------------------+ |
| | | Return value policy | Description | |
| | +==================================================+============================================================================+ |
| | | :enum:`return_value_policy::take_ownership` | Reference an existing object (i.e. do not create a new copy) and take | |
| | | | ownership. Python will call the destructor and delete operator when the | |
| | | | object's reference count reaches zero. Undefined behavior ensues when the | |
| | | | C++ side does the same, or when the data was not dynamically allocated. | |
| | +--------------------------------------------------+----------------------------------------------------------------------------+ |
| | | :enum:`return_value_policy::copy` | Create a new copy of the returned object, which will be owned by Python. | |
| | | | This policy is comparably safe because the lifetimes of the two instances | |
| | | | are decoupled. | |
| | +--------------------------------------------------+----------------------------------------------------------------------------+ |
| | | :enum:`return_value_policy::move` | Use ``std::move`` to move the return value contents into a new instance | |
| | | | that will be owned by Python. This policy is comparably safe because the | |
| | | | lifetimes of the two instances (move source and destination) are decoupled.| |
| | +--------------------------------------------------+----------------------------------------------------------------------------+ |
| | | :enum:`return_value_policy::reference` | Reference an existing object, but do not take ownership. The C++ side is | |
| | | | responsible for managing the object's lifetime and deallocating it when | |
| | | | it is no longer used. Warning: undefined behavior will ensue when the C++ | |
| | | | side deletes an object that is still referenced and used by Python. | |
| | +--------------------------------------------------+----------------------------------------------------------------------------+ |
| | | :enum:`return_value_policy::reference_internal` | Indicates that the lifetime of the return value is tied to the lifetime | |
| | | | of a parent object, namely the implicit ``this``, or ``self`` argument of | |
| | | | the called method or property. Internally, this policy works just like | |
| | | | :enum:`return_value_policy::reference` but additionally applies a | |
| | | | ``keep_alive<0, 1>`` *call policy* (described in the next section) that | |
| | | | prevents the parent object from being garbage collected as long as the | |
| | | | return value is referenced by Python. This is the default policy for | |
| | | | property getters created via ``def_property``, ``def_readwrite``, etc. | |
| | +--------------------------------------------------+----------------------------------------------------------------------------+ |
| | | :enum:`return_value_policy::automatic` | **Default policy.** This policy falls back to the policy | |
| | | | :enum:`return_value_policy::take_ownership` when the return value is a | |
| | | | pointer. Otherwise, it uses :enum:`return_value_policy::move` or | |
| | | | :enum:`return_value_policy::copy` for rvalue and lvalue references, | |
| | | | respectively. See above for a description of what all of these different | |
| | | | policies do. | |
| | +--------------------------------------------------+----------------------------------------------------------------------------+ |
| | | :enum:`return_value_policy::automatic_reference` | As above, but use policy :enum:`return_value_policy::reference` when the | |
| | | | return value is a pointer. This is the default conversion policy for | |
| | | | function arguments when calling Python functions manually from C++ code | |
| | | | (i.e. via handle::operator()). You probably won't need to use this. | |
| | +--------------------------------------------------+----------------------------------------------------------------------------+ |
| |
|
| | Return value policies can also be applied to properties: |
| |
|
| | .. code-block:: cpp |
| |
|
| | class_<MyClass>(m, "MyClass") |
| | .def_property("data", &MyClass::getData, &MyClass::setData, |
| | py::return_value_policy::copy); |
| |
|
| | Technically, the code above applies the policy to both the getter and the |
| | setter function, however, the setter doesn't really care about *return* |
| | value policies which makes this a convenient terse syntax. Alternatively, |
| | targeted arguments can be passed through the :class:`cpp_function` constructor: |
| |
|
| | .. code-block:: cpp |
| |
|
| | class_<MyClass>(m, "MyClass") |
| | .def_property("data" |
| | py::cpp_function(&MyClass::getData, py::return_value_policy::copy), |
| | py::cpp_function(&MyClass::setData) |
| | ); |
| |
|
| | .. warning:: |
| |
|
| | Code with invalid return value policies might access uninitialized memory or |
| | free data structures multiple times, which can lead to hard-to-debug |
| | non-determinism and segmentation faults, hence it is worth spending the |
| | time to understand all the different options in the table above. |
| |
|
| | .. note:: |
| |
|
| | One important aspect of the above policies is that they only apply to |
| | instances which pybind11 has *not* seen before, in which case the policy |
| | clarifies essential questions about the return value's lifetime and |
| | ownership. When pybind11 knows the instance already (as identified by its |
| | type and address in memory), it will return the existing Python object |
| | wrapper rather than creating a new copy. |
| |
|
| | .. note:: |
| |
|
| | The next section on :ref:`call_policies` discusses *call policies* that can be |
| | specified *in addition* to a return value policy from the list above. Call |
| | policies indicate reference relationships that can involve both return values |
| | and parameters of functions. |
| |
|
| | .. note:: |
| |
|
| | As an alternative to elaborate call policies and lifetime management logic, |
| | consider using smart pointers (see the section on :ref:`smart_pointers` for |
| | details). Smart pointers can tell whether an object is still referenced from |
| | C++ or Python, which generally eliminates the kinds of inconsistencies that |
| | can lead to crashes or undefined behavior. For functions returning smart |
| | pointers, it is not necessary to specify a return value policy. |
| |
|
| | .. _call_policies: |
| |
|
| | Additional call policies |
| | ======================== |
| |
|
| | In addition to the above return value policies, further *call policies* can be |
| | specified to indicate dependencies between parameters or ensure a certain state |
| | for the function call. |
| |
|
| | Keep alive |
| | ---------- |
| |
|
| | In general, this policy is required when the C++ object is any kind of container |
| | and another object is being added to the container. ``keep_alive<Nurse, Patient>`` |
| | indicates that the argument with index ``Patient`` should be kept alive at least |
| | until the argument with index ``Nurse`` is freed by the garbage collector. Argument |
| | indices start at one, while zero refers to the return value. For methods, index |
| | ``1`` refers to the implicit ``this`` pointer, while regular arguments begin at |
| | index ``2``. Arbitrarily many call policies can be specified. When a ``Nurse`` |
| | with value ``None`` is detected at runtime, the call policy does nothing. |
| |
|
| | When the nurse is not a pybind11-registered type, the implementation internally |
| | relies on the ability to create a *weak reference* to the nurse object. When |
| | the nurse object is not a pybind11-registered type and does not support weak |
| | references, an exception will be thrown. |
| |
|
| | Consider the following example: here, the binding code for a list append |
| | operation ties the lifetime of the newly added element to the underlying |
| | container: |
| |
|
| | .. code-block:: cpp |
| |
|
| | py::class_<List>(m, "List") |
| | .def("append", &List::append, py::keep_alive<1, 2>()); |
| |
|
| | For consistency, the argument indexing is identical for constructors. Index |
| | ``1`` still refers to the implicit ``this`` pointer, i.e. the object which is |
| | being constructed. Index ``0`` refers to the return type which is presumed to |
| | be ``void`` when a constructor is viewed like a function. The following example |
| | ties the lifetime of the constructor element to the constructed object: |
| |
|
| | .. code-block:: cpp |
| |
|
| | py::class_<Nurse>(m, "Nurse") |
| | .def(py::init<Patient &>(), py::keep_alive<1, 2>()); |
| |
|
| | .. note:: |
| |
|
| | ``keep_alive`` is analogous to the ``with_custodian_and_ward`` (if Nurse, |
| | Patient != 0) and ``with_custodian_and_ward_postcall`` (if Nurse/Patient == |
| | 0) policies from Boost.Python. |
| |
|
| | Call guard |
| | ---------- |
| |
|
| | The ``call_guard<T>`` policy allows any scope guard type ``T`` to be placed |
| | around the function call. For example, this definition: |
| |
|
| | .. code-block:: cpp |
| |
|
| | m.def("foo", foo, py::call_guard<T>()); |
| |
|
| | is equivalent to the following pseudocode: |
| |
|
| | .. code-block:: cpp |
| |
|
| | m.def("foo", [](args...) { |
| | T scope_guard; |
| | return foo(args...); // forwarded arguments |
| | }); |
| |
|
| | The only requirement is that ``T`` is default-constructible, but otherwise any |
| | scope guard will work. This is very useful in combination with `gil_scoped_release`. |
| | See :ref:`gil`. |
| |
|
| | Multiple guards can also be specified as ``py::call_guard<T1, T2, T3...>``. The |
| | constructor order is left to right and destruction happens in reverse. |
| |
|
| | .. seealso:: |
| |
|
| | The file :file:`tests/test_call_policies.cpp` contains a complete example |
| | that demonstrates using `keep_alive` and `call_guard` in more detail. |
| |
|
| | .. _python_objects_as_args: |
| |
|
| | Python objects as arguments |
| | =========================== |
| |
|
| | pybind11 exposes all major Python types using thin C++ wrapper classes. These |
| | wrapper classes can also be used as parameters of functions in bindings, which |
| | makes it possible to directly work with native Python types on the C++ side. |
| | For instance, the following statement iterates over a Python ``dict``: |
| |
|
| | .. code-block:: cpp |
| |
|
| | void print_dict(py::dict dict) { |
| | /* Easily interact with Python types */ |
| | for (auto item : dict) |
| | std::cout << "key=" << std::string(py::str(item.first)) << ", " |
| | << "value=" << std::string(py::str(item.second)) << std::endl; |
| | } |
| |
|
| | It can be exported: |
| |
|
| | .. code-block:: cpp |
| |
|
| | m.def("print_dict", &print_dict); |
| |
|
| | And used in Python as usual: |
| |
|
| | .. code-block:: pycon |
| |
|
| | >>> print_dict({'foo': 123, 'bar': 'hello'}) |
| | key=foo, value=123 |
| | key=bar, value=hello |
| |
|
| | For more information on using Python objects in C++, see :doc:`/advanced/pycpp/index`. |
| |
|
| | Accepting \*args and \*\*kwargs |
| | =============================== |
| |
|
| | Python provides a useful mechanism to define functions that accept arbitrary |
| | numbers of arguments and keyword arguments: |
| |
|
| | .. code-block:: python |
| |
|
| | def generic(*args, **kwargs): |
| | ... # do something with args and kwargs |
| |
|
| | Such functions can also be created using pybind11: |
| |
|
| | .. code-block:: cpp |
| |
|
| | void generic(py::args args, py::kwargs kwargs) { |
| | /// .. do something with args |
| | if (kwargs) |
| | /// .. do something with kwargs |
| | } |
| |
|
| | /// Binding code |
| | m.def("generic", &generic); |
| |
|
| | The class ``py::args`` derives from ``py::tuple`` and ``py::kwargs`` derives |
| | from ``py::dict``. |
| |
|
| | You may also use just one or the other, and may combine these with other |
| | arguments as long as the ``py::args`` and ``py::kwargs`` arguments are the last |
| | arguments accepted by the function. |
| |
|
| | Please refer to the other examples for details on how to iterate over these, |
| | and on how to cast their entries into C++ objects. A demonstration is also |
| | available in ``tests/test_kwargs_and_defaults.cpp``. |
| |
|
| | .. note:: |
| |
|
| | When combining \*args or \*\*kwargs with :ref:`keyword_args` you should |
| | *not* include ``py::arg`` tags for the ``py::args`` and ``py::kwargs`` |
| | arguments. |
| |
|
| | Default arguments revisited |
| | =========================== |
| |
|
| | The section on :ref:`default_args` previously discussed basic usage of default |
| | arguments using pybind11. One noteworthy aspect of their implementation is that |
| | default arguments are converted to Python objects right at declaration time. |
| | Consider the following example: |
| |
|
| | .. code-block:: cpp |
| |
|
| | py::class_<MyClass>("MyClass") |
| | .def("myFunction", py::arg("arg") = SomeType(123)); |
| |
|
| | In this case, pybind11 must already be set up to deal with values of the type |
| | ``SomeType`` (via a prior instantiation of ``py::class_<SomeType>``), or an |
| | exception will be thrown. |
| |
|
| | Another aspect worth highlighting is that the "preview" of the default argument |
| | in the function signature is generated using the object's ``__repr__`` method. |
| | If not available, the signature may not be very helpful, e.g.: |
| |
|
| | .. code-block:: pycon |
| |
|
| | FUNCTIONS |
| | ... |
| | | myFunction(...) |
| | | Signature : (MyClass, arg : SomeType = <SomeType object at 0x101b7b080>) -> NoneType |
| | ... |
| |
|
| | The first way of addressing this is by defining ``SomeType.__repr__``. |
| | Alternatively, it is possible to specify the human-readable preview of the |
| | default argument manually using the ``arg_v`` notation: |
| |
|
| | .. code-block:: cpp |
| |
|
| | py::class_<MyClass>("MyClass") |
| | .def("myFunction", py::arg_v("arg", SomeType(123), "SomeType(123)")); |
| |
|
| | Sometimes it may be necessary to pass a null pointer value as a default |
| | argument. In this case, remember to cast it to the underlying type in question, |
| | like so: |
| |
|
| | .. code-block:: cpp |
| |
|
| | py::class_<MyClass>("MyClass") |
| | .def("myFunction", py::arg("arg") = (SomeType *) nullptr); |
| |
|
| | Keyword-only arguments |
| | ====================== |
| |
|
| | Python 3 introduced keyword-only arguments by specifying an unnamed ``*`` |
| | argument in a function definition: |
| |
|
| | .. code-block:: python |
| |
|
| | def f(a, *, b): # a can be positional or via keyword; b must be via keyword |
| | pass |
| |
|
| | f(a=1, b=2) # good |
| | f(b=2, a=1) # good |
| | f(1, b=2) # good |
| | f(1, 2) # TypeError: f() takes 1 positional argument but 2 were given |
| |
|
| | Pybind11 provides a ``py::kwonly`` object that allows you to implement |
| | the same behaviour by specifying the object between positional and keyword-only |
| | argument annotations when registering the function: |
| |
|
| | .. code-block:: cpp |
| |
|
| | m.def("f", [](int a, int b) { /* ... */ }, |
| | py::arg("a"), py::kwonly(), py::arg("b")); |
| |
|
| | Note that, as in Python, you cannot combine this with a ``py::args`` argument. |
| | This feature does *not* require Python 3 to work. |
| |
|
| | .. versionadded:: 2.6 |
| |
|
| | .. _nonconverting_arguments: |
| |
|
| | Non-converting arguments |
| | ======================== |
| |
|
| | Certain argument types may support conversion from one type to another. Some |
| | examples of conversions are: |
| |
|
| | * :ref:`implicit_conversions` declared using ``py::implicitly_convertible<A,B>()`` |
| | * Calling a method accepting a double with an integer argument |
| | * Calling a ``std::complex<float>`` argument with a non-complex python type |
| | (for example, with a float). (Requires the optional ``pybind11/complex.h`` |
| | header). |
| | * Calling a function taking an Eigen matrix reference with a numpy array of the |
| | wrong type or of an incompatible data layout. (Requires the optional |
| | ``pybind11/eigen.h`` header). |
| |
|
| | This behaviour is sometimes undesirable: the binding code may prefer to raise |
| | an error rather than convert the argument. This behaviour can be obtained |
| | through ``py::arg`` by calling the ``.noconvert()`` method of the ``py::arg`` |
| | object, such as: |
| |
|
| | .. code-block:: cpp |
| |
|
| | m.def("floats_only", [](double f) { return 0.5 * f; }, py::arg("f").noconvert()); |
| | m.def("floats_preferred", [](double f) { return 0.5 * f; }, py::arg("f")); |
| |
|
| | Attempting the call the second function (the one without ``.noconvert()``) with |
| | an integer will succeed, but attempting to call the ``.noconvert()`` version |
| | will fail with a ``TypeError``: |
| |
|
| | .. code-block:: pycon |
| |
|
| | >>> floats_preferred(4) |
| | 2.0 |
| | >>> floats_only(4) |
| | Traceback (most recent call last): |
| | File "<stdin>", line 1, in <module> |
| | TypeError: floats_only(): incompatible function arguments. The following argument types are supported: |
| | 1. (f: float) -> float |
| |
|
| | Invoked with: 4 |
| |
|
| | You may, of course, combine this with the :var:`_a` shorthand notation (see |
| | :ref:`keyword_args`) and/or :ref:`default_args`. It is also permitted to omit |
| | the argument name by using the ``py::arg()`` constructor without an argument |
| | name, i.e. by specifying ``py::arg().noconvert()``. |
| |
|
| | .. note:: |
| |
|
| | When specifying ``py::arg`` options it is necessary to provide the same |
| | number of options as the bound function has arguments. Thus if you want to |
| | enable no-convert behaviour for just one of several arguments, you will |
| | need to specify a ``py::arg()`` annotation for each argument with the |
| | no-convert argument modified to ``py::arg().noconvert()``. |
| |
|
| | .. _none_arguments: |
| |
|
| | Allow/Prohibiting None arguments |
| | ================================ |
| |
|
| | When a C++ type registered with :class:`py::class_` is passed as an argument to |
| | a function taking the instance as pointer or shared holder (e.g. ``shared_ptr`` |
| | or a custom, copyable holder as described in :ref:`smart_pointers`), pybind |
| | allows ``None`` to be passed from Python which results in calling the C++ |
| | function with ``nullptr`` (or an empty holder) for the argument. |
| |
|
| | To explicitly enable or disable this behaviour, using the |
| | ``.none`` method of the :class:`py::arg` object: |
| |
|
| | .. code-block:: cpp |
| |
|
| | py::class_<Dog>(m, "Dog").def(py::init<>()); |
| | py::class_<Cat>(m, "Cat").def(py::init<>()); |
| | m.def("bark", [](Dog *dog) -> std::string { |
| | if (dog) return "woof!"; /* Called with a Dog instance */ |
| | else return "(no dog)"; /* Called with None, dog == nullptr */ |
| | }, py::arg("dog").none(true)); |
| | m.def("meow", [](Cat *cat) -> std::string { |
| | // Can't be called with None argument |
| | return "meow"; |
| | }, py::arg("cat").none(false)); |
| |
|
| | With the above, the Python call ``bark(None)`` will return the string ``"(no |
| | dog)"``, while attempting to call ``meow(None)`` will raise a ``TypeError``: |
| |
|
| | .. code-block:: pycon |
| |
|
| | >>> from animals import Dog, Cat, bark, meow |
| | >>> bark(Dog()) |
| | 'woof!' |
| | >>> meow(Cat()) |
| | 'meow' |
| | >>> bark(None) |
| | '(no dog)' |
| | >>> meow(None) |
| | Traceback (most recent call last): |
| | File "<stdin>", line 1, in <module> |
| | TypeError: meow(): incompatible function arguments. The following argument types are supported: |
| | 1. (cat: animals.Cat) -> str |
| |
|
| | Invoked with: None |
| |
|
| | The default behaviour when the tag is unspecified is to allow ``None``. |
| |
|
| | .. note:: |
| |
|
| | Even when ``.none(true)`` is specified for an argument, ``None`` will be converted to a |
| | ``nullptr`` *only* for custom and :ref:`opaque <opaque>` types. Pointers to built-in types |
| | (``double *``, ``int *``, ...) and STL types (``std::vector<T> *``, ...; if ``pybind11/stl.h`` |
| | is included) are copied when converted to C++ (see :doc:`/advanced/cast/overview`) and will |
| | not allow ``None`` as argument. To pass optional argument of these copied types consider |
| | using ``std::optional<T>`` |
| |
|
| | Overload resolution order |
| | ========================= |
| |
|
| | When a function or method with multiple overloads is called from Python, |
| | pybind11 determines which overload to call in two passes. The first pass |
| | attempts to call each overload without allowing argument conversion (as if |
| | every argument had been specified as ``py::arg().noconvert()`` as described |
| | above). |
| |
|
| | If no overload succeeds in the no-conversion first pass, a second pass is |
| | attempted in which argument conversion is allowed (except where prohibited via |
| | an explicit ``py::arg().noconvert()`` attribute in the function definition). |
| |
|
| | If the second pass also fails a ``TypeError`` is raised. |
| |
|
| | Within each pass, overloads are tried in the order they were registered with |
| | pybind11. |
| |
|
| | What this means in practice is that pybind11 will prefer any overload that does |
| | not require conversion of arguments to an overload that does, but otherwise prefers |
| | earlier-defined overloads to later-defined ones. |
| |
|
| | .. note:: |
| |
|
| | pybind11 does *not* further prioritize based on the number/pattern of |
| | overloaded arguments. That is, pybind11 does not prioritize a function |
| | requiring one conversion over one requiring three, but only prioritizes |
| | overloads requiring no conversion at all to overloads that require |
| | conversion of at least one argument. |
| |
|
