Boost.Python

C++ & Python Integration

MotivationLeveraging of strong sides in both languages:

1. C++: a. performance, b. effective resource management

2. Python: a. high coding performance, b. variety of available solutionsc. easy to start

Why Boost.Python?Python C-API: requires too much manual workSWIG (aka generators): usually requires manual adjusting - take time to understand “someones” code.Boost.Python (middleware lib): most flexible - expose what you need in the way you like.

Getting Boost readyGetting Boost ready:

get your copy of boost and build it:./bootstrap.sh --prefix=../../ --with-python-version=3.4

--with-python-version=X.Y --with-python=/path/to/bin/python--with-python-root=/path/to

./b2 -j4 address-model=64 --with-python stagebuilds python 3.4 lib (only) for 64 bit os, all binaries are in

stage folder

… pick what to wrap Simple C API to wrap into Python

int LZ4_compress(const char* source, char* dest, int sourceSize);

int LZ4_decompress_safe (const char* source, char* dest, int compressedSize, int maxDecompressedSize);

basic API of lz4 compression library

Step #1: declare Python module#include <boost/python.hpp>#include <lz4.h>using namespace boost::python;

BOOST_PYTHON_MODULE(lz4py) // make sure that your output binary will have same name// lz4py.so (not liblz4py.so ) or lz4py.dll

{// place for our wrappers}

basic wrappers...

def("dump", // name of function on Python sideLZ4_compress/*, return_value_policy<>() - optional */ );

def("load", LZ4_decompress_safe);

… butdef() will wrap functions as they are …and call on Python side will not be so comfortable:

lz4py.dump(input, # data to compressoutput, # ALLOCATED OUTPUT BUFFER (!?)size) # size of input data (!?)

and function will return size of used bytes in output bufferit is too much C style - this is not Python...

Step #2: make it more Pythonadd tiny wrapper in C++ code:

const std::string compress(const std::string& d);const std::string decompress(const std::string&

d);and wrap it for Python:

def("dump", compress );def("load", decompress);

and … new trouble

… string and bytesIn Python 3 all string are UNICODE:

1. can’t return binary data as string - will faila. could work for python 2 - but this is hidden

problem in future2. need a way to return binary data

a. convert output to bytes()

Step 4: type conversionDeclare own type:

class buffer {public: typedef boost::shared_ptr<char> data_ptr_t;public: buffer(data_ptr_t& data, int size): data_(data), size_(size){} int len() const { return size_; } char* const get_data() const { return data_.get(); }private: data_ptr_t data_; int size_;

};

Converter to PythonConverter interface declaration:

template< typename T >struct type_into_python {

static PyObject* convert( T const &);};

Converter implementation:template<> PyObject* type_into_python<buffer>::convert(

buffer const& data){return PyBytes_FromStringAndSize(data.get_data(), data.len());

}

Converter from Python: declarationtemplate< typename T >struct type_from_python {

type_from_python(){ converter::registry::push_back( convertible,

construct, type_id<T>() );}static void* convertible( PyObject* );static void construct(

PyObject*, converter::rvalue_from_python_stage1_data* );

};

Convert from Python: implementtemplate<> void* type_from_python<buffer>::convertible( PyObject* obj){ return PyBytes_Check(obj) ? obj : nullptr;}template<> void type_from_python<buffer>::construct( PyObject* obj, converter::rvalue_from_python_stage1_data* data){

auto storage = reinterpret_cast< converter::rvalue_from_python_storage<buffer>* >( data )->storage.bytes;

int size = PyBytes_Size(obj);buffer::data_ptr_t buffer_data = buffer::data_ptr_t(new char[size]);memcpy(buffer_data.get(), PyBytes_AsString(obj), size);new(storage) buffer(buffer_data, size);data->convertible = storage;

}

New APINew C++ functions:

const buffer compress(buffer const& src);const buffer compress(std::string const& src);

const buffer decompress(buffer const& src);

Python exports:def("dump", static_cast<const buffer (*)(buffer const&)>( compress ));def("dump", static_cast<const buffer (*)(std::string const&)>( compress ));def("load", decompress);

Wrapp interface:class lz4_c {

class bytes_ref{ PyObject* bytes_;public:

int len() const { return PyBytes_Size(bytes_);} char const* data() const { return PyBytes_AsString(bytes_); }};class buffer{ private: lz4_c& owner_; };

Step #5: class

public: // initialize & use pool of preallocated blocks lz4_c(int block_max_size, int count); const buffer compress(bytes_ref const& src); const buffer compress(std::string const& src); const buffer decompress(bytes_ref const& src); void release(data_block_t&);};

Boost.Python class wrapperDeclare Python class for lz4_c:

class_<lz4_c>("lz4",init<int, int>( args("block_max_size", "count"))) //.def(init<int, int>( args("block_max_size", "count")))

● “lz4” - class name on Python side;● second argument - constructor, can be skipped if class have default one;● other constructors could be added using .def() like functions;● args - you can give names to your arguments and use named args in

python;

Class methodsextend our class by methods:

.def("dump", static_cast<const lz4_c::buffer (lz4_c::*)(lz4_c::bytes_ref const&)>(&lz4_c::compress)).def("dump", static_cast<const lz4_c::buffer (lz4_c::*)

(std::string const&)>( &lz4_c::compress)) .def("load", &lz4_c::decompress)similar to regular functions.

Class propertiescan be exposed to python code:

// read only.add_property("ratio", &lz4_c::ratio)// read / write.add_property("digit", make_getter(&lz4_c::digit),

make_setter(&lz4_c::digit))

Exceptionsclass my_ex: public std::exception {};...void translate_error( error const& my_ex);...//boost::python::register_exception_translator<T,F>(F)boost::python::register_exception_translator<my_ex>(

translate_error)

And moreSTL type conversion:

● std::map <-> dict(); ● std::vector <-> list(), tuple()

Define Python operators● __str__, __repr__, __ call__ ● __add__, __lt__:

o .def(self + self)o .def(self < self)

Inheritance:● make available in Python● declare C++ relation

Default args:● wrapper to allow default args:

Python function pointers:● typedef boost::function<void

(string s) > funcptr;

Return Value Policy● with_custodian_and_ward: Ties lifetimes of the arguments● with_custodian_and_ward_postcall: Ties lifetimes of the arguments and

results● return_internal_reference: Ties lifetime of one argument to that of result● return_value_policy<T> with T one of:

o reference_existing_object: naive (dangerous) approacho copy_const_reference: Boost.Python v1 approacho copy_non_const_reference:o manage_new_object: Adopt a pointer and hold the instance

Thank You !

Andriy.Ohorodnyk@globallogic.com