boost.python: c++ and python integration
Post on 06-Dec-2014
516 Views
Preview:
DESCRIPTION
TRANSCRIPT
Boost.Python
C++ & Python Integration
MotivationLeveraging of strong sides in both languages:
1. C++: a. performance, b. effective resource management
2. Python: a. high coding performance, b. variety of available solutionsc. easy to start
Why Boost.Python?Python C-API: requires too much manual workSWIG (aka generators): usually requires manual adjusting - take time to understand “someones” code.Boost.Python (middleware lib): most flexible - expose what you need in the way you like.
Getting Boost readyGetting Boost ready:
get your copy of boost and build it:./bootstrap.sh --prefix=../../ --with-python-version=3.4
--with-python-version=X.Y --with-python=/path/to/bin/python--with-python-root=/path/to
./b2 -j4 address-model=64 --with-python stagebuilds python 3.4 lib (only) for 64 bit os, all binaries are in
stage folder
… pick what to wrap Simple C API to wrap into Python
int LZ4_compress(const char* source, char* dest, int sourceSize);
int LZ4_decompress_safe (const char* source, char* dest, int compressedSize, int maxDecompressedSize);
basic API of lz4 compression library
Step #1: declare Python module#include <boost/python.hpp>#include <lz4.h>using namespace boost::python;
BOOST_PYTHON_MODULE(lz4py) // make sure that your output binary will have same name// lz4py.so (not liblz4py.so ) or lz4py.dll
{// place for our wrappers}
basic wrappers...
def("dump", // name of function on Python sideLZ4_compress/*, return_value_policy<>() - optional */ );
def("load", LZ4_decompress_safe);
… butdef() will wrap functions as they are …and call on Python side will not be so comfortable:
lz4py.dump(input, # data to compressoutput, # ALLOCATED OUTPUT BUFFER (!?)size) # size of input data (!?)
and function will return size of used bytes in output bufferit is too much C style - this is not Python...
Step #2: make it more Pythonadd tiny wrapper in C++ code:
const std::string compress(const std::string& d);const std::string decompress(const std::string&
d);and wrap it for Python:
def("dump", compress );def("load", decompress);
and … new trouble
… string and bytesIn Python 3 all string are UNICODE:
1. can’t return binary data as string - will faila. could work for python 2 - but this is hidden
problem in future2. need a way to return binary data
a. convert output to bytes()
Step 4: type conversionDeclare own type:
class buffer {public: typedef boost::shared_ptr<char> data_ptr_t;public: buffer(data_ptr_t& data, int size): data_(data), size_(size){} int len() const { return size_; } char* const get_data() const { return data_.get(); }private: data_ptr_t data_; int size_;
};
Converter to PythonConverter interface declaration:
template< typename T >struct type_into_python {
static PyObject* convert( T const &);};
Converter implementation:template<> PyObject* type_into_python<buffer>::convert(
buffer const& data){return PyBytes_FromStringAndSize(data.get_data(), data.len());
}
Converter from Python: declarationtemplate< typename T >struct type_from_python {
type_from_python(){ converter::registry::push_back( convertible,
construct, type_id<T>() );}static void* convertible( PyObject* );static void construct(
PyObject*, converter::rvalue_from_python_stage1_data* );
};
Convert from Python: implementtemplate<> void* type_from_python<buffer>::convertible( PyObject* obj){ return PyBytes_Check(obj) ? obj : nullptr;}template<> void type_from_python<buffer>::construct( PyObject* obj, converter::rvalue_from_python_stage1_data* data){
auto storage = reinterpret_cast< converter::rvalue_from_python_storage<buffer>* >( data )->storage.bytes;
int size = PyBytes_Size(obj);buffer::data_ptr_t buffer_data = buffer::data_ptr_t(new char[size]);memcpy(buffer_data.get(), PyBytes_AsString(obj), size);new(storage) buffer(buffer_data, size);data->convertible = storage;
}
New APINew C++ functions:
const buffer compress(buffer const& src);const buffer compress(std::string const& src);
const buffer decompress(buffer const& src);
Python exports:def("dump", static_cast<const buffer (*)(buffer const&)>( compress ));def("dump", static_cast<const buffer (*)(std::string const&)>( compress ));def("load", decompress);
Wrapp interface:class lz4_c {
class bytes_ref{ PyObject* bytes_;public:
int len() const { return PyBytes_Size(bytes_);} char const* data() const { return PyBytes_AsString(bytes_); }};class buffer{ private: lz4_c& owner_; };
Step #5: class
public: // initialize & use pool of preallocated blocks lz4_c(int block_max_size, int count); const buffer compress(bytes_ref const& src); const buffer compress(std::string const& src); const buffer decompress(bytes_ref const& src); void release(data_block_t&);};
Boost.Python class wrapperDeclare Python class for lz4_c:
class_<lz4_c>("lz4",init<int, int>( args("block_max_size", "count"))) //.def(init<int, int>( args("block_max_size", "count")))
● “lz4” - class name on Python side;● second argument - constructor, can be skipped if class have default one;● other constructors could be added using .def() like functions;● args - you can give names to your arguments and use named args in
python;
Class methodsextend our class by methods:
.def("dump", static_cast<const lz4_c::buffer (lz4_c::*)(lz4_c::bytes_ref const&)>(&lz4_c::compress)).def("dump", static_cast<const lz4_c::buffer (lz4_c::*)
(std::string const&)>( &lz4_c::compress)) .def("load", &lz4_c::decompress)similar to regular functions.
Class propertiescan be exposed to python code:
// read only.add_property("ratio", &lz4_c::ratio)// read / write.add_property("digit", make_getter(&lz4_c::digit),
make_setter(&lz4_c::digit))
Exceptionsclass my_ex: public std::exception {};...void translate_error( error const& my_ex);...//boost::python::register_exception_translator<T,F>(F)boost::python::register_exception_translator<my_ex>(
translate_error)
And moreSTL type conversion:
● std::map <-> dict(); ● std::vector <-> list(), tuple()
Define Python operators● __str__, __repr__, __ call__ ● __add__, __lt__:
o .def(self + self)o .def(self < self)
Inheritance:● make available in Python● declare C++ relation
Default args:● wrapper to allow default args:
Python function pointers:● typedef boost::function<void
(string s) > funcptr;
Return Value Policy● with_custodian_and_ward: Ties lifetimes of the arguments● with_custodian_and_ward_postcall: Ties lifetimes of the arguments and
results● return_internal_reference: Ties lifetime of one argument to that of result● return_value_policy<T> with T one of:
o reference_existing_object: naive (dangerous) approacho copy_const_reference: Boost.Python v1 approacho copy_non_const_reference:o manage_new_object: Adopt a pointer and hold the instance
Thank You !
Andriy.Ohorodnyk@globallogic.com
top related