mixed language programming

7/31/2019 Mixed Language Programming

1/43

Mixed Language

Programming


2/43

Types of Languages Mixings

Embedding Python Using Python code

from code in a different language.

Extending Python Using code in adifferent language from Python.

We are going to focus only on extending

Python in this presentation.


3/43

Why Mixing Python with Another

Language? Extending\Embedding Python:

To spare the development cycle of a code that wasalready written once (code reuse).

Extending Python: To improve Python code performance.

True multithreading support.

Type-safe language -> Less bugs (?).

Other reasons: To save testing time by testing low-level code using

Python high-level tests.

Other languages are more mature: Less bugs in the language.

Mission-specific tools that ease development.


4/43

Python-C Integration

Using ctypes


5/43

ctypes

Functionality:

Gives access to the OS specific DLLs via

Python.Allows calling C language functions in user-

defined DLLs\shared libraries from Python.

Pros:

No need to recompile the DLL (wrap librariesin pure Python).

Availability: External module up to V2.4. FromV2.5 part of the Python standard library.


6/43

Loading a DLL cdecl calling convention (most of the

times):

dll = ctypes.cdll.LoadLibrary()

stdcallcalling convention (on Windows):dll =

ctypes.windll.LoadLibrary()

On Windows, some DLLs are beingloaded automatically:

Kernel32.dll is at ctypes.windll.kernel32

Msvcrt.dll is at ctypes.cdll.msvcrt


7/43

Fundamental Data Types

Python typeC typectypes type

1 character

string

charc_char

int/longintc_int

int/longunsignedlongc_ulong

floatfloatc_float

floatdoublec_double

string or Nonechar * (NULL

terminated)

c_char_p

int\long or Nonevoid *c_void_p


8/43

Fundamental Data Types (II)

An int data type:

i = ctypes.c_int(3) # or i=ctypes.c_int(), to get

an init val of 0. i.value = -99

A string data type:

s = "Hello, World"c_s = ctypes.c_char_p(s) # or c_s =

ctypes.c_char_p(), to an init val of NULL.

c_s.value = "Hi, there"


9/43

Strings and Buffers

Assigning a new value to instances of thepointer types c_char_p and c_void_pchanges the memory locationthey pointto, not the contentsof the memory block.

If you need mutable memory blocks, usecreate_string_buffer():

buff = ctypes.create_string_buffer("Hello", 10)

# create a 10 byte bufferbuff.value = "Hi"

Use the raw attribute to get the entirebuffer contents.

ctypes.sizeof() == sizeof() in C


10/43

Accessing Global Variables

Suppose that in your DLL, Test.dll, youhave the following exported global variabledeclaration:

__declspec(dllexport) double some_var;

To access it from ctypes, you must call thein_dll() method of the variables properctype, with the variables containing DLL

and name: test_dll = ctypes.CDLL(Test.dll)

test_dll_var = ctypes.c_double.in_dll(test_dll,"some_var")

test_dll_var.value = 3.14


11/43

Calling Functions You can call every exported function (i.e.,

__declspec(dllexport)-ed function on Windows)in your DLL.

All Python types except integers, strings, andunicode strings have to be wrapped in theircorresponding ctypes type: libc = ctypes.cdll.msvcrt

printf = libc.printf

printf("An int %d, a double %f\n", 1234,c_double(3.14))

None is used as the NULL pointer: libc.time(None) # System time in seconds since

the UNIX epoch


12/43

Windows Specific Issues

You must specify (for example)GetModuleHandleA or GetModuleHandleWexplicitly, and then call it with normal or unicodestrings respectively.

ctypes tries to protect you from calling functionswith the wrong number of arguments or thewrong calling convention (raises ValueError). It does this by examining the stack after the function

returns, so although an error is raised the functionhasbeen called.

ctypes uses SEH to prevent crashes fromgeneral protection faults due to invalid argumentvalues (raises WindowsError): >>> windll.kernel32.GetModuleHandleA(32)


13/43

Specifying Functions Prototypes

Specifying the required argument types can be

done by the argtypes attribute - a sequence of

ctypes types:

strchr = libc.strchr strchr.argtypes = [ctypes.c_char_p, ctypes.c_char]

By default, functions are assumed to return the

C int type.

Other return types can be specified by setting

the restype attribute:

strchr.restype = ctypes.c_char_p

strchr("abcdef", "d") # Would now return def, and not


14/43

Passing Parameters By Reference

The byref() function is used to pass

parameters by reference:

f = ctypes.c_float() # f = 0.0

s = ctypes.create_string_buffer(10)

# create a 10 byte empty buffer.

libc.sscanf("Input", "%f %s", ctypes.byref(f), s) pointer() does a lot more work, so it is

faster to use byref() if you don't need the

pointer object in Python itself.


15/43

Structures and Unions Structures and unions must derive from the

Structure and Union base classes.

They must contain a _fields_ attribute: a list of2-

tuples, containing a field nameand a field type. The field type must be a ctypes type.

>>> class POINT(ctypes.Structure):

... _fields_ = [("x", ctypes.c_int), ("y",ctypes.c_int)]

>>> point = POINT(10, 20)

>>> print point.x, point.y


16/43

Bit Fields in Structures and Unions

Bit fields are only possible for integer fields.

The bit width is specified as the third item in

the _fields_ tuples.

In C:struct {int word1 :16;

int word2 : 16;} DoubleWord;

In Python:>>> class DoubleWord(ctypes.Structure):

... _fields_ = [(word1", ctypes.c_int, 16),

... (word2", ctypes.c_int, 16)]


17/43

Arrays and Sizes

The way to create array types is by multiplying a

data type with the array length.

>>> TenIntegers = ctypes.c_int * 10>>> ii = TenIntegers(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

# You can also write: ii = TenIntegers(), and

get an array of zeroes.

>>> for i in ii:

print i


18/43

Pointers Pointer instances are created by calling the

pointer() function on a ctypes type: i = ctypes.c_int(42)

pi = ctypes.pointer(i)

pi have a contents attribute which returns theobject to which the pointer points (the i object,above): i2 = ctypes.c_int(99)

pi.contents = i2 It is also possible to use indexes to change the

pointed value, but you must know what you'rechanging, just as in C: pi[0] = 22 # i2.value is now 22.


19/43

Pointer Types and NULL Pointers

A pointer type is being created via the

POINTER() factory function:

PI = ctypes.POINTER(ctypes.c_int)

pi = PI(ctypes.c_int(42))

Calling the pointer type without an

argument creates a NULL pointer:

null_ptr = PI()

As already mentioned, to set a POINTER

type field to NULL, you can assign None.


20/43

Type Checking and Casting Usually, only instances of the specified type are

accepted.

cast() takes two parameters, a ctypes object thatcan be converted to a pointer of some kind, and

a ctypes pointer type. It returns an instance of the second argument,

which references the same memory block as thefirst argument:

a = (ctypes.c_byte * 4)() # Creates a zeroed 4-byte array.

pi = ctypes.cast(a, PI) # Converts thearray to an int ptr.

Array instances are auto-casted to the

compatible pointer types (ii to PI, for example).


21/43

Forward Declarations We can write a linked list on C like this:

struct node; // forward declaration.

struct {

void *data;struct node *next;

} node;

In ctypes, we can define the cell class and set the

_fields_ attribute after the class statement:>>> class node(ctypes.Structure):

... pass

>>> node._fields_ = [(data", ctypes.c_void_p),

("next", ctypes.POINTER(node))]


22/43

Callback Functions

You can pass Python functions as C callbacks.

The CFUNCTYPE factory function creates types

for callback functions using the cdecl callingconvention.

On Windows, the WINFUNCTYPE factoryfunction creates types for callback functions

using the stdcall calling convention. Both of these factory functions are called with

the result type as first argument, and thecallback functions expected argument types as

the remaining arguments.


23/43

Callback Functions An example>>> from ctypes import c_int, POINTER>>> IntArray5 = c_int * 5

>>> ia = IntArray5(5, 1, 7, 33, 99)

>>> qsort = libc.qsort

>>> qsort.restype = None # A void return value.>>> CMPFUNC = CFUNCTYPE(c_int,

POINTER(c_int), POINTER(c_int))

>>> def py_cmp_func(a, b):

... return a.contents b.contents

>>> qsort(ia, len(ia), ctypes.sizeof(ctypes.c_int),CMPFUNC(py_cmp_func))

# ia is now sorted


24/43

Callback Functions Warnings

Make sure you keep references toCFUNCTYPE objects as long as they areused from C code.

Otherwise, they may be garbage collected,crashing your program when a callback ismade.

The result type of callback functions is

limited to the predefined fundamentaltypes.

ctypes 1.0.1 (included in Python 2.5) behaves

strangely if you use other types.


25/43

Function Pointers In Python In-order to call a function pointer from Python,

you need to wrap it with the appropriate callbacktype.

If a function pointer is defined as: typedef int (*MyFuncPtr)(void);

Then the equivalent ctypes type is:MyFuncPtr = ctypes.CFUNCTYPE(ctypes.c_int)

Use WINFUNCTYPE, instead of CFUNCTYPEfor the stdcall pointer: typedef int (__stdcall *MyStdFuncPtr)(void);

You can also use this type as a struct field, function

return type, etc.


26/43

Auto-Generated Wrapper Code

You can try the ctypeslib code-generator

that parses the header file of the DLL and

generate the wrapper code from it(currently unstable).


27/43

Alternatives For ctypes

SWIG

Boost.Python

Many moreUsing the Python\C API directly

Pyrex

Weave\PyInline

and many more:

SIP

Cython

PyCXX

Etc.


28/43

Python-.NET

Integration UsingPython For .NET


29/43

Why Extending Python with .NET

Code? Managed code -> no resource leaks.

An extensive runtime library (BCL) Windows Forms

ASP .NET ADO .NET

GUI developing is simpler on the Visual Studio .NETDesigner.

Cross-Platform: .NET Framework

Mono

DotGNU

Availability:

.NET 3.0 is installed on Windows Vista and Windows Server2008.


30/43


31/43

Assembly Loading

Before you can access a namespace in anassembly, you have to load it.

Loading an assembly: import clr

clr.AddReference("System.Windows.Forms") # No.dll prefix required

Assemblies are loaded from: The application directory.

The PYTHONPATH (= sys.path).

The GAC.

-> To ensure that you can load an assembly, putthe directory containing the assembly insys.path.


32/43

Accessing CLR Namespaces

Once their assembly was loaded ,CLR

namespaces are treated as Python packages:

from System.Windows.Forms import Form

You must import the clr module BEFOREaccessing the CLR namespaces, even if their

containing assembly is already\automatically

loaded.

import clr

from System import String, Int32

from System.Collections import *


33/43

Using Classes Python for .NET allows you to use any non-

private classes, structs, interfaces, enums ordelegates from Python.

To create an instance of a managed class, youcall one of its public constructors:

from System.Collections import Hashtable table = Hashtable()

table["key 1"] = "value 1

You can get and set fields and properties of CLRobjects just as if they were regular attributes.

You can subclass managed classes in Python,though members of the Python subclass are notvisible to .NET code: import System.Windows.Forms as WinForms

class HelloApp(WinForms.Form):

# Subclass Implementation


34/43

Using Methods

All public and protected methods of CLRobjects are accessible to Python:

from System import Environmentenv_vars =

Environment.GetEnvironmentVariables()

for env_var in env_vars:

print env_var.Key, env_var.Value

Static methods may be called eitherthrough the class or through an instance of

the class (As above, for Environment).


35/43

Type Conversions

Elemental Python types (string, int, long, etc.)

convert automatically to compatible managed

equivalents (String, Int32, etc.) and vice-versa. All strings returned from the CLR are returned as

Unicode.

Types that do not have a Pythonic equivalent

are exposed as instances of managed classesor structs (System.Decimal, for example).


36/43

Using Generics When running under.NET runtime 2.0+, you can

use generic types.

A generic type must be bound to create aconcrete type before it can be instantiated:

from System.Collections.Generic import Dictionarymy_dict = Dictionary[String, Int32]()

my_dict[a] = 1

You can also pass a subset of Python types thatdirectly correspond to .NET types:my_dict = Dictionary[str, int]()

This also works when explicitly selecting genericmethods or specific versions of overloadedmethods and constructors.


37/43

Overloaded and Generic Methods

There are cases where it is desirable to select a

particular method overload explicitly.

Methods of CLR objects have an

"__overloads__" attribute that can be used forthis purpose:

from System import Console

Console.WriteLine.__overloads__[bool](true) Generic methods may be bound at runtime

using this syntax directly on the method:

someobject.SomeGenericMethod[str]("10")


38/43

Other Features If a managed object implements one or more indexers,

you can call the indexer using standard Python indexingsyntax.

You can raise and catch managed exceptions just thesame as you would pure-Python exceptions.

Managed arrays behave like standard Python sequence.

Multidimensional arrays support indexing one would usein C#: arr = System.Array.CreateInstance(String, 3, 2, 4) # Creating a 3

dimensions 3*2*4 array of strings. arr[1, 1, 1] = abc

Managed objects that implement the IEnumerableinterface (=Collections) can be iterated over using thestandard iteration Python idioms.


39/43

Getting Help

The docstring of a CLR method (__doc__)

can be used to view the signature of the

method, including overloads:print Environment.GetFolderPath.__doc__

You can also use the Python help method

to inspect a managed class:help(Environment)


40/43

(Explicit) Delegates And Events A delegate type can be instantiated and passed a

callable Python object to get a delegate instance:

>>> def my_handler(source, args):

print 'my_handler called!'

>>> from System import AssemblyLoadEventHandler,AppDomain

>>> deleg = AssemblyLoadEventHandler(my_handler)

>>> AppDomain.CurrentDomain.AssemblyLoad += deleg

# event+=deleg -> use deleg as an event handler.>>> from System.Drawing import Point #

The delegate would be called. You can also call itdirectly: deleg(None, None).

You can add callable objects to a delegate instanceusing: deleg+=method.


41/43

(Implicit) Delegates And Events

You do not have to pass an explicitly instantiateddelegate instance to an event:

>>> class SimpleApp(WinForms.Form):

def __init__(self): self.button = WinForms.Button()

self.button.Click += self.button_Click# register the event handler (unregister it with -=)

self.Controls.Add(self.button) def button_Click(self, sender, args):

WinForms.MessageBox.Show(Thisbutton was clicked-on!)

>>> WinForms.Application.Run(SimpleApp())


42/43

The Boxing Problem An example:

points = System.Array.CreateInstance(Point, 3)

points[0].X = 1 # Wont work as you expect!

In C#, the compiler knows that Point is a valuetype and change the value in place.

On Python, the setattr (.X = 1) changes the stateof the boxed value, not the original unboxedvalue It is still 0!

Handling the boxing problem:

point = points[0] point.X = 1

points[0] = point

The rule in Python is: "the result of any attribute

or item access is a boxed value.


43/43

The Alternative: IronPython

An implementation of Python running on .NET. Same syntax as Python For .NET.

Pros:

Managed code -> no resource leaks.

Better .NET interoperability.

Has (unofficial) support from Microsoft.

Cons:

Cant use neither Python modules written in C (likectypes), nor P/Invoke (directly).

Not every Python module is implemented (os) or

behave the same (re)

mixed language programming

Documents