introduction to python · swig connects programs written in c/c++ with a variety of high-level...

Introduction to Python

Part 3: Advanced Topics

Michael Kraus ([email protected])

Max-Planck-Institut fur Plasmaphysik, Garching

7. March 2012

Some Advanced Topics

calling and embedding C and Fortran code:

weave: inline C/C++ code, translation of Python code to C++SWIG: wrap C/C++ codef2py: wrap Fortran codectypes, cython: call functions in C libraries

GUI programming with PyQt and PySide

parallelisation:

threadingmultiprocessingparallel python (PP)mpi4py

symbolic computing with Sage

Calling and Embedding C and Fortran Code in Python

Python is very fast when writing code, but not necessarily so fast whenexecuting code (especially for numerical applications)

� implement time-consuming parts of your program in C/C++/Fortran

lots of existing library routines in C/C++/Fortran

� reuse in Python by wrapping the corresponding libraries or source files,making them appear as Python modules

couple or extend existing codes with Python

� combine different modules or codes to larger programs

� use Python as (graphical) user interface, for visualisation or debugging

Sample Problem: Laplace Equation

solving the 2D Laplace equation using an iterative finite differencescheme (four point averaging, Gauss-Seidel or Gauss-Jordan)

� solve for some unknown function u(x , y) such that ∇2u = 0 with someboundary condition specified

� discretise the domain into an (nx × ny ) grid of points

� the function u can be represented as a two-dimensional array u(nx , ny )

� the values of u along the sides of the domain are given (and stay fixed)

� the solution can be obtained by iterating in the following manner:

for i in range(1, nx -1):

for j in range(1, ny -1):

u[i,j] = ( (u[i-1, j] + u[i+1, j])*dy**2 + \

(u[i, j-1] + u[i, j+1])*dx**2 \

) / (2.0*( dx**2 + dy **2))

� in pure Python this is REALLY slow

Sample Problem: Laplace Equation in NumPy

the for loop of the Laplace solver can be readily expressed by a muchsimpler NumPy expression:

u[1:-1, 1:-1] = ( (u[0:-2, 1:-1] + u[2:, 1: -1])*dy**2 + \

(u[1:-1, 0:-2] + u[1:-1, 2:])*dx**2 \

) / (2.0*( dx**2 + dy **2))

� the advantage of this expression is that it is completely done in C

� speedup of a factor of 50x over the pure Python loop(another factor of 5 or so if you link NumPy with Intel MKL or ATLAS)

� (slight) drawback: this expression uses temporary arrays� during one iteration, the computed values at an already computed

location will not be used� in the original for loop, once the value of u[1,1] is computed, the next

value for u[1,2] will use the newly computed u[1,1] and not the old one� since the NumPy expression uses temporary arrays internally, only the old

value of u[1,1] will be used� the algorithm will still converge but in twice as much time� reduction of the benefit by a factor of 2

Weave

Weave is a subpackage of SciPy and has two modes of operation

weave.blitz accelerates Python code by translating it to C++ code whichit compiles into a Python moduleweave.inline allows to embed C/C++ code directly into Python code

� mainly used to speed up calculations on arrays

� fast/efficient: directly operates on NumPy arrays (no temporary copies)

� the first time you run a blitz or inline function, it gets compiled into aPython module, the next time it is called, it will run immediately

References:http://www.scipy.org/Weave

http://www.scipy.org/Cookbook/Weave

http://www.scipy.org/Weave

http://www.scipy.org/Cookbook/Weave

Weave: Laplace Equation in weave.blitz

to use weave.blitz, the accelerated code has to be put into a stringwhich is passed to the weave.blitz function:

from scipy import weave

expr = """

u[1:-1, 1:-1] = ( (u[0:-2, 1:-1] + u[2:, 1: -1])*dy**2 + \

(u[1:-1, 0:-2] + u[1:-1, 2:])*dx**2 \

) / (2.0*( dx**2 + dy **2))

"""

weave.blitz(expr , check_size =0)

� the first time the code is called, weave.blitz converts the NumPyexpression into C++ code, builds a Python module, and invokes it

� for the array expressions, weave.blitz uses Blitz++

� speedup of 100-200x over the Python loop

� weave.blitz does not use temporary arrays for the computation (thecomputed values are re-used immediately) and therefore behaves morelike the original for loop

Weave: Laplace Equation in weave.inline

in weave.inline the C/C++ code has to be put into a string which ispassed to the weave.inline function, together with the variables used:

from scipy.weave import converters , inline

code = """

for (int i=1; i<nx -1; ++i) {

for (int j=1; j<ny -1; ++j) {

u(i,j) = ( (u(i-1,j) + u(i+1,j))*dy*dy +

(u(i,j-1) + u(i,j+1))*dx*dx

) / (2.0*( dx*dx + dy*dy));

}

}

"""

inline(code , [’u’, ’dx’, ’dy’, ’nx’, ’ny’],

type_converters=converters.blitz , compiler = ’gcc’)

� here we use Blitz++ arrays (speedup 250-500x over the Python loop)

� with pointer arithmetic, you can get an additional speedup of a factor 2

� weave.inline does not use temporary arrays for the computation

SWIG

connects programs written in C/C++ with a variety of high-levelprogramming languages (e.g. Python)

parses C/C++ interfaces, takes the declarations found in header files,and uses them to generate wrapper code required for Python to callinto the C/C++ code

� control of C/C++ applications (GUI, visualisation)

� testing and debugging

� glue together different C/C++ modules or codes

� reuse existing functions or libraries in Python(numerical methods, data analysis)

References:http://www.swig.org/

http://docs.scipy.org/doc/numpy/reference/swig.html

http://www.scipy.org/Cookbook/SWIG_NumPy_examples

http://www.swig.org/

http://docs.scipy.org/doc/numpy/reference/swig.html

http://www.scipy.org/Cookbook/SWIG_NumPy_examples

SWIG

example: square function in C

swig example.h

#ifndef _SWIG_EXAMPLE_H_

#define _SWIG_EXAMPLE_H_

int square(int);

#endif // _SWIG_EXAMPLE_H_

swig example.c

#include "swig_example.h"

int square(int x) {

return x*x;

}

� in order to call the functions in swig_example.c from Python, you needto write an interface file which is the input to SWIG

� SWIG generates a wrapper that looks like a normal Python module

SWIG

SWIG interface file for the example:

swig example.i

%module swig_example %{

#include "swig_example.h"

%}

%include "swig_example.h"

� contains C/C++ declarations and special SWIG directives

� %module defines a module name

� code between %{ and %} is copied verbatim to the resulting wrapper file

� used to include header files and other declarations required to makethe generated wrapper code compile

� below the module, declarations of C/C++ functions to be included inthe Python module are listed

� declarations included in a SWIG input file to not automatically appearin the generated wrapper code

SWIG

call SWIG to generate wrapper code (swig_example_wrap.c) and Pythonmodule (swig_example.py):

> swig -python swig_example.i

compile the wrapper code into a shared library:

gcc ‘python -config --cflags ‘ -c swig_example_wrap.c \

swig_example.c

gcc ‘python -config --ldflags ‘ -shared -o _swig_example.so \

swig_example_wrap.o swig_example.o

� important: the library name has to start with _

import the module from Python and call its functions:

>>> import swig_example as se

>>> se.square (2)

4

SWIG

NumPy arrays need some more care

� we need typemaps to convert between C arrays and NumPy arrays

� they generate additional code that takes care of correctly translatingbetween C and Python objects

� NumPy provides all you need in a SWIG include file (numpy.i)

Laplace solver in C:

laplace.h

#ifndef _LAPLACE_H_

#define _LAPLACE_H_

void laplace(double* u, int nx, int ny, \

double dx, double dy);

#endif // _LAPLACE_H_

SWIG

Laplace solver in C:

laplace.c

#include "laplace.h"

void laplace(double* u, int nx, int ny, \

double dx, double dy) {

double dx2 = dx*dx;

double dy2 = dy*dy;

double d2inv = 0.5 / (dx2 + dy2);

for (int i=1; i<nx -1; i++) {

for (int j=1; j<ny -1; j++) {

*(u+i*nx+j) = (

( *(u+(i-1)*nx+j) + *(u+(i+1)*nx+j) ) * dy2

+ ( *(u+i*nx+(j-1)) + *(u+i*nx+(j+1)) ) * dx2

) * d2inv;

}

}

}

SWIG

SWIG interface file for the Laplace solver:

laplace.i

%module laplace_swig %{

#define SWIG_FILE_WITH_INIT

#include "laplace.h"

%}

%include "numpy.i"

%init %{

import_array ();

%}

%apply (double* INPLACE_ARRAY2 , int DIM1 , int DIM2)

{( double* u, int nx, int ny)};

%include "laplace.h"

� call SWIG to generate wrapper code (laplace_wrap.c, laplace.py):

> swig -I$(NUMPY_SWIG_DIR) -python laplace.i

� you have to specify the directory containing numpy.i (NUMPY SWIG DIR)

SWIG

compile the wrapper code into a shared library:

gcc ‘python -config --cflags ‘ -I$(NUMPY_INCLUDE_DIR) \

-c laplace_wrap.c laplace.c

gcc ‘python -config --ldflags ‘ -shared -o _laplace_swig.so \

laplace_wrap.o laplace.o

� you have to add the NumPy include directory (NUMPY INCLUDE DIR)

import the module from Python and call its functions:

>>> import laplace_swig

>>> dx = dy = 0.1

>>> u = np.zeros( (100 ,100) )

>>> u[0] = 1.

>>> laplace_swig.laplace(u, dx, dy)

(the python27/numpy/1.6.1 module on the cluster provides environmentvariables NUMPY INCLUDE DIR and NUMPY SWIG DIR)

f2py

f2py is a NumPy module that lets you easily call Fortran functionsfrom Python

� the f2py command builds a shared library and creates Python wrappercode that makes the Fortran routine look like a native Python module:

> f2py -c laplace.f90 -m laplace_fortran

� in Python you only have to import it like every other Python module:

>>> import laplace_fortran

>>> laplace_fortran.laplace (...)

� when passing arrays, f2py automatically takes care of the right layout,i.e. row-major in Python and C vs. column-major in Fortran

References:http://cens.ioc.ee/projects/f2py2e/

http://www.f2py.com/

http://www.scipy.org/Cookbook/F2Py

http://cens.ioc.ee/projects/f2py2e/

http://www.f2py.com/

http://www.scipy.org/Cookbook/F2Py

f2py

in the Fortran routine you have to include some additional directivestelling f2py the intent of the parameters:

subroutine laplace(u, n, m, dx , dy)

real*8, dimension (1:n,1:m) :: u

real*8 :: dx, dy

integer :: n, m, i, j

!f2py intent(in,out) :: u

!f2py intent(in) :: dx ,dy

!f2py intent(hide) :: n,m

...

end subroutine

� the dimensions of the array are passed implicitly by Python:

>>> dx = dy = 0.1

>>> u = np.zeros( (100 ,100) )

>>> u[0] = 1.

>>> laplace_fortran.laplace(u, dx, dy)

Sample Problem: Laplace Solver with f2py

laplace.f90

subroutine laplace(u, nx , ny , dx , dy)

real*8, dimension (1:nx ,1:ny) :: u

real*8 :: dx, dy

integer :: nx , ny , i, j

!f2py intent(in,out) :: u

!f2py intent(in) :: dx , dy

!f2py intent(hide) :: nx , ny

do i=2, nx -1

do j=2, ny -1

u(i,j) = ( (u(i-1,j) + u(i+1,j))*dy*dy +

(u(i,j-1) + u(i,j+1))*dx*dx

) / (2.0*( dx*dx + dy*dy))

enddo

enddo

end subroutine

External Libraries: ctypes

the ctypes module of the Python Standard Library provides Ccompatible data types and allows calling functions in shared libraries

� ctypes can be used to wrap these libraries in pure Python

� to use ctypes to access C code you need to know some details aboutthe underlying C library (names, calling arguments, types, etc.), butyou do not have to write C extension wrapper code or compileanything with a C compiler (like in Cython)

� simple example: libc.rand()

>>> import ctypes

>>> libc = ctypes.CDLL("/usr/lib/libc.so")

>>> libc.rand()

16807

� ctypes provides functionality to take care of correct datatype handling,automatic type casting, passing values by reference, pointers, etc.

Reference: http://docs.python.org/library/ctypes.html

http://docs.python.org/library/ctypes.html

External Libraries: Cython

Cython provides declarations for many functions from the standard Clibrary, e.g. for the C math library:

cython cmath.py

from libc.math cimport sin

cdef double f(double x):

return sin(x*x)

� calling C’s sin() functions is substantially faster than Python’smath.sin() function as there’s no wrapping of arguments, etc.

� the math library is not linked by default

� in addition to cimporting the declarations, you must configure yourbuild system to link against the shared library m, e.g. in setup.py:

ext_modules = [Extension("cython_cmath",

["cython_cmath.pyx"],

libraries =["m"])]

External Libraries: Cython

if you want to access C code for which Cython does not provide aready to use declaration, you must declare them yourself:

cdef extern from "math.h":

double sin(double)

� this declares the sin() function in a way that makes it available toCython code and instructs Cython to generate C code that includes themath.h header file

� the C compiler will see the original declaration in math.h at compiletime, but Cython does not parse math.h and thus requires a separatedefinition

� you can declare and call into any C library as long as the module thatCython generates is properly linked against the shared or static library

Performance Python Benchmark Reloaded

2D Laplace Solver: 500x500 grid, 100 iterations

Type of Solution Time GNU (ms) Time Intel (ms)

Numpy 895 907

Weave (Blitz) 286 -Weave (Inline) 291 -

Cython 289 196Cython (fast) 287 194Cython (2 threads) 196 100Cython (4 threads) 147 52

SWIG - 167ctypes 195 143f2py 409 353

Pure C 228 136Pure Fortran 200 136

(Benchmarked on an Intel Xeon E5440 @ 2.83GHz)

GUI Programming

lots of options:

� Tkinter: Python’s default GUI toolkit included in the Standard Library

� wxPython: Python wrapper for wxWidgetshttp://www.wxpython.org/

� PyGTK: Python wrapper for GTKhttp://www.pygtk.org/

� PyQt and PySide: Python wrappers for Qt (not just a GUI library!)http://www.riverbankcomputing.co.uk/software/pyqt

http://www.pyside.org/

� Traits and TraitsUI: development model that comes with automaticallycreated user interfaceshttp://code.enthought.com/projects/traits/

http://code.enthought.com/projects/traits_ui/

http://www.wxpython.org/

http://www.pygtk.org/

http://www.riverbankcomputing.co.uk/software/pyqt

http://www.pyside.org/

http://code.enthought.com/projects/traits/

http://code.enthought.com/projects/traits_ui/

PyQt and PySide

PyQt and PySide are Python bindings for the Qt application framework

� run on all platforms supported by Qt (Linux, MacOSX, Windows)

� the interface of both modules is almost identical(PySide is slightly cleaner, see http://developer.qt.nokia.com/

wiki/Differences_Between_PySide_and_PyQt)

� main difference: License (PyQt: GPL and commercial, PySide: LGPL)and PySide is supported by Nokia (who develops Qt)

� generate Python code from Qt Designer

� add new GUI controls written in Python to Qt Designer

Documentation:http://www.riverbankcomputing.co.uk/static/Docs/PyQt4/html/index.html

http://www.riverbankcomputing.co.uk/static/Docs/PyQt4/html/classes.html

http://zetcode.com/tutorials/pyqt4/

http://doc.qt.nokia.com/

http://qt.nokia.com/learning

http://developer.qt.nokia.com/wiki/Differences_Between_PySide_and_PyQt

http://developer.qt.nokia.com/wiki/Differences_Between_PySide_and_PyQt

http://www.riverbankcomputing.co.uk/static/Docs/PyQt4/html/index.html

http://www.riverbankcomputing.co.uk/static/Docs/PyQt4/html/classes.html


http://doc.qt.nokia.com/

http://qt.nokia.com/learning

PyQt and PySide: Simple Example

simple example: only shows a small window

necessary imports: basic GUI widgets are located in the QtGui module

from PySide import QtGui

every PySide application must create an application object, thesys.argv parameter is a list of arguments from the command line:

app = QtGui.QApplication(sys.argv)

QtGui.QWidget is the base class of all user interface objects in PySide,the default constructor has no parent and creates a window:

w = QtGui.QWidget ()

resize the window, move it around on the screen, and set a title:

w.resize (250, 150)

w.move (300, 300)

w.setWindowTitle(’Simple Example ’)


make the window visible:

w.show()

finally, enter the main loop of the application:

sys.exit(app.exec_ ())

� the event handling starts from this point: the main loop receives eventsfrom the window system and dispatches them to the applicationwidgets

� the main loop ends, if we call the exit() method or the main widget isdestroyed (e.g. by clicking the little x on top of the window)

� the sys.exit() method ensures a clean exit


pyside simple example.py

import sys

from PySide import QtGui

def main ():


w = QtGui.QWidget ()

w.resize (250, 150)

w.move (300, 300)

w.setWindowTitle(’Simple Example ’)

w.show()


if __name__ == ’__main__ ’:

main()

� we can do a lot with this window: resize it, maximise it, minimise it


you could also set an application icon:

w.setWindowIcon(QtGui.QIcon(’my_app.png’))

� the QtGui.QIcon is initialised by providing it with a (path and) filename

you can move and resize at once:

w.setGeometry (300, 300, 250, 150)

� the first two parameters are the x and y positions of the window

� the latter two parameters are the width and height of the window

PyQt and PySide: Widgets

this was a procedural example, but in PySide you’re writing objects:

create a new class called Example that inherits from QtGui.QWidget:

class Example(QtGui.QWidget ):

we must call two constructors: for the Example class and for theinherited class:

def __init__(self):

super(Example , self). __init__ ()

self.initUI ()

� the super() method returns the parent object of the Example class

� the constructor method is always called __init__() in Python

� the creation of the GUI is delegated to the initUI() method:


pyside object example.py

class Example(QtGui.QWidget ):

def __init__(self):

super(Example , self). __init__ ()

self.initUI ()

def initUI(self):

self.setGeometry (300, 300, 250, 150)

self.setWindowTitle(’PySide Object Example ’)

self.setWindowIcon(QtGui.QIcon(’my_app.png’))

self.show()

def main ():


ex = Example ()


if __name__ == ’__main__ ’:

main()


our Example class inherits lots of methods from the QtGui.QWidget class:


self.setWindowTitle(’PySide Object Example ’)


PyQt and PySide: Buttons and Tooltips

add a button and some tooltips:

QtGui.QToolTip.setFont(QtGui.QFont(’SansSerif ’, 10))

self.setToolTip(’This is a QWidget widget ’)

btn = QtGui.QPushButton(’Button ’, self)

btn.setToolTip(’This is a QPushButton widget ’)

btn.resize(btn.sizeHint ())

btn.move(50, 50)

PyQt and PySide: Buttons and Tooltips

QtGui provides static methods to set default properties like fonts:

QtGui.QToolTip.setFont(QtGui.QFont(’SansSerif ’, 10))

set a tooltip for our Example class:

self.setToolTip(’This is a QWidget widget ’)

create a button which is placed within our Example class’ main widget:

btn = QtGui.QPushButton(’Button ’, self)

set a tooltip for the button, resize it and move it somewhere:

btn.setToolTip(’This is a QPushButton widget ’)

btn.resize(btn.sizeHint ())

btn.move(50, 50)

� GUI elements can give a size hint corresponding to their content(e.g. button text, picture size)

PyQt and PySide: Signals and Slots

bring the button to life by connecting it to a slot:

btn = QtGui.QPushButton(’Quit’, self)

btn.clicked.connect(QtCore.QCoreApplication.instance (). quit)

btn.resize(qbtn.sizeHint ())

btn.move(50, 50)

� now we can close our window programatically (not only by clicking x)

� you have to import QtCore for this to work:

from PyQt4 import QtCore

� the event processing system in PySide uses the signal & slot mechanism

� if we click on the button, the signal clicked is emitted

� it can be connected to any Qt slot or any Python function

� QtCore.QCoreApplication is created with the QtGui.QApplication

� it contains the main event loop and processes and dispatches all events

� it’s instance() method returns its current instance

� the quit() method terminates the application

PyQt and PySide: QLineEdit and QMessageBox

add a QtGui.QLineEdit where the user can enter some text that isdisplayed in a popup window when he clicks the OK button:

def initUI(self):

...

self.inputle = QtGui.QLineEdit(self)

self.inputle.resize (120 ,20)

self.inputle.move (10 ,50)

okbtn = QtGui.QPushButton(’OK’, self)

okbtn.clicked.connect(self.showMessage)

okbtn.resize(okbtn.sizeHint ())

okbtn.move (150, 50)

...

we also have to define a function that serves as slot:

def showMessage(self):

QtGui.QMessageBox.information(self , "Information",

self.inputle.text ())


the QtGui.QLineEdit has to be an element of the class so that the slotcan access it


the OK button is connected to the method showMessage:


showMessage reads the content of the QtGui.QLineEdit via it’s text()

method and creates a QtGui.QMessageBox:

QtGui.QMessageBox.information(self , "Information",

self.inputle.text ())

� the title of the QtGui.QMessageBox is set to "Information"

� the message it displays is the content of the QtGui.QLineEdit


that’s how our QLineEdit example looks like:

� the left window looks a little messy...

...let’s add some intelligent layout management!

PyQt and PySide: Layout Management

absolute positioning, i.e. specifying the position and size of eachwidget in pixels, is not very practical

� the size and the position of a widget do not change, if you resize a window� changing fonts in your application might spoil the layout� if you decide to change your layout, you must completely redo your layout

� use layout classes instead:

QtGui.QHBoxLayout, QtGui.QVBoxLayout, QtGui.QGridLayout

� line up widgets horizontally, vertically or in a grid

� add stretches that adapt when the window is resized

example: place two buttons in the right bottom corner

� use one horizontal box, one vertical box, and stretch factors


create two push buttons:

okButton = QtGui.QPushButton("OK")

cancelButton = QtGui.QPushButton("Cancel")

create a horizontal box layout, add a stretch factor and the buttons:

hbox = QtGui.QHBoxLayout ()

hbox.addStretch (1)

hbox.addWidget(okButton)

hbox.addWidget(cancelButton)

put the horizontal layout into a vertical one

vbox = QtGui.QVBoxLayout ()

vbox.addStretch (1)

vbox.addLayout(hbox)

set the main layout of the window

self.setLayout(vbox)


def initUI(self):


self.setWindowTitle(’PySide Layout Example ’)


okButton = QtGui.QPushButton("OK")

cancelButton = QtGui.QPushButton("Cancel")

hbox = QtGui.QHBoxLayout ()

hbox.addStretch (1)

hbox.addWidget(okButton)

hbox.addWidget(cancelButton)

vbox = QtGui.QVBoxLayout ()

vbox.addStretch (1)

vbox.addLayout(hbox)

self.setLayout(vbox)

self.show()


for the QLineEdit example we use a grid layout:

create widgets:



qtbtn = QtGui.QPushButton(’Quit’, self)

create grid layout:

grid = QtGui.QGridLayout ()

add widgets with addWidget(widget, row, column):

grid.addWidget(self.inputle , 0, 1)

grid.addWidget(okbtn , 0, 2)

grid.addWidget(qtbtn , 2, 2)

set the stretch factor of the second row to 1

grid.setRowStretch (1, 1)


def initUI(self):

...


qtbtn = QtGui.QPushButton(’Quit’, self)


app = QtCore.QCoreApplication.instance ()

qtbtn.clicked.connect(app.quit)


grid = QtGui.QGridLayout ()

grid.addWidget(self.inputle , 0, 1)

grid.addWidget(okbtn , 0, 2)

grid.addWidget(qtbtn , 2, 2)

grid.setRowStretch (1, 1)

self.setLayout(grid)

self.show()

PyQt and PySide: Matplotlib

we want to embed a Matplotlib Figure into a QtWidget

the Figure object is the backend-independent representation of our plot:

from matplotlib.figure import Figure

the FigureCanvasQTAgg object is the backend-dependent figure canvas:

from matplotlib.backends.backend_qt4agg \

import FigureCanvasQTAgg as FigureCanvas

� renders the Figure we’re about to draw to the Qt4 backend

� the FigureCanvasQTAgg is a Matplotlib class as well as a QWidget

� we can create a new widget by deriving from it:

class Qt4MplCanvas(FigureCanvas ):

...

� this class will render our Matplotlib plot


the __init__ method contains the code to draw the graph:

def __init__(self):

self.x = np.arange (0.0, 3.0, 0.01)

self.y = np.cos (2*np.pi*self.x)

self.fig = Figure ()

self.axes = self.fig.add_subplot (111)

self.axes.plot(self.x, self.y)

initialise the FigureCanvas (renders the Matplotlib Figure in a QtWidget):

FigureCanvas.__init__(self , self.fig)

the plot canvas is instantiated in the main() function:

def main ():


mpl = Qt4MplCanvas ()

mpl.show()



import sys

import numpy as np

from PySide import QtCore , QtGui

from matplotlib.figure import Figure

from matplotlib.backends.backend_qt4agg \

import FigureCanvasQTAgg as FigureCanvas

class Qt4MplCanvas(FigureCanvas ):

def __init__(self):

self.x = np.arange (0.0, 3.0, 0.01)

self.y = np.cos(2*np.pi*self.x)

self.fig = Figure ()

self.axes = self.fig.add_subplot (111)

self.axes.plot(self.x, self.y)

FigureCanvas.__init__(self , self.fig)

def main ():


mpl = Qt4MplCanvas ()

mpl.show()


if __name__ == ’__main__ ’:

main()

PyQt and PySide: Designer

Qt tool for designing and building graphical user interfaces

� design widgets, dialogs or complete main windows using on-screenforms and a simple drag-and-drop interface

Qt Designer uses XML .ui files to store designs and does not generateany code itself

� Qt’s uic utility generates the C++ code that creates the user interface

� Qt’s QUiLoader class allows an application to load a .ui file and createthe corresponding user interface dynamically

PySide and PyQt include the uic Python module

� like QUiLoader it can load .ui files to create a user interface dynamically

� like the uic utility it can also generate the Python code that will createthe user interface

� the pyuic4 utility is a command line interface to the uic module


we just created a graphical user interface! (example.ui)

� generate Python code with pyuic4:

pyuic4 example.ui -o example.py

� the code is contained in a single class Ui_LeWidget that is derived fromthe Python object

� he name of the class is the name of the toplevel object set in Designerwith Ui_ prepended

the class contains a method called setupUi()

� this takes a single argument which is the widget in which the userinterface is created (typically QWidget, QDialog or QMainWindow)

window = QWidget ()

ui = Ui_LeWidget ()

ui.setupUi(window)

� the generated code can then be used in a number of ways


create a simple application to create the dialog:

import sys

from PyQt4.QtGui import QApplication , QWidget

from example import Ui_LeWidget

app = QApplication(sys.argv)

window = QWidget ()

ui = Ui_LeWidget ()

ui.setupUi(window)

window.show()



single inheritance approach: subclass QWidget and set up the userinterface in the __init__() method

import sys

from PyQt4.QtGui import QApplication , QWidget , QMessageBox



class LeWidget(QWidget ):

def __init__(self):

QWidget.__init__(self)

self.ui = Ui_LeWidget ()

self.ui.setupUi(self)


self.ui.qtbtn.clicked.connect(app.quit)

self.ui.okbtn.clicked.connect(self.showMessage)

self.show()


text = self.ui.inputle.text()

QMessageBox.information(self , ’Information ’, text)


multiple inheritance approach: subclass QWidget and Ui_LeWidget

import sys

from PyQt4.QtGui import QApplication , QWidget , QMessageBox



class LeWidget(QWidget , Ui_LeWidget ):

def __init__(self):

QWidget.__init__(self)

self.setupUi(self)


self.qtbtn.clicked.connect(app.quit)

self.okbtn.clicked.connect(self.showMessage)

self.show()


text = self.inputle.text()

QMessageBox.information(self , ’Information ’, text)

PyQt and PySide

this just gives you a taste of how PySide and PyQt work

� there are quite a few other important basic topics:

widgets: combo box, check box, slider, progress bar, tables, ...menus, toolbars, statusbarsignals & slots:

emit signals yourselfevent modelscatch events and ask to ignore or accept them(“Do you really want to quit?”)

dialogs: input, select file, printdrag’n’drop, using the clipboardimages, graphics, drawingcustom widgetsthe model, view, controller paradigm

� the aforementioned tutorial is a good place to start:http://zetcode.com/tutorials/pyqt4/


Parallel Programming

Python includes a multithreading and a multiprocessing package

multithreading is seriously limited by the Global Interpreter Lock, whichallows only one thread to be interacting with the interpreter at a time

� this restricts Python programs to run on a single processor regardlessof how many CPU cores you have and how many threads you create

multiprocessing allows spawning subprocesses which may run ondifferent cores but are completely independent entities

� communication is only possible by message passing which makesparallelisation an effort that is probably not justified by the gain

however, you can

� compile NumPy and SciPy with threaded libraries like ATLAS or MKL� use Cython’s prange for very simple parallelisation of loops via OpenMP� use Parallel Python (job based parallelisation)� use mpi4py (message passing, standard in HPC with C/C++/Fortran)

Parallel Python (PP)

the PP module provides a mechanism for parallel execution of pythoncode on systems with multiple cores and clusters connected via network

� simple to implement job-based parallelisation technique

� internally PP uses processes and Inter Process Communication (IPC)to organise parallel computations

� all the details and complexity are hidden from you and yourapplication, it just submits jobs and retrieves their results

� very simple way to write parallel Python applications

� cross-platform portability (Linux, MacOSX, Windows), interoperability,dynamic load-balancing

� software written with PP works in parallel also on many computersconnected via local network or internet even if the run differentoperating systems (it’s pure Python!)

Reference: http://www.parallelpython.com/

http://www.parallelpython.com/


import the pp module:

import pp

start pp execution server with the number of workers set to the numberof processors in the system:

job_server = pp.Server ()

submit all the tasks for parallel execution:

f1 = job_server.submit(func1 , args1 , depfuncs1 , modules1)



...

retrieve the results as needed:

r1 = f1()

r2 = f2()

r3 = f3()

...


import math

import pp

def sum_primes(nstart , nfinish ):

sum = 0

for n in xrange(nstart , nfinish +1):

if isprime(n): # checks if n is a prime

sum += n

return sum

nprimes = 100001

job_server = pp.Server ()

ncpus = job_server.get_ncpus ()

np_cpu , np_add = divmod(nprimes , ncpus)

ranges = [ (i*np_cpu+1, (i+1)* np_cpu) for i in range(0,ncpus)]

ranges[ncpus -1] = (ranges[ncpus -1][0] , ranges[ncpus -1][1]+ np_add)

sum = 0

jobs = [( job_server.submit(sum_primes , input , (isprime ,),

("math" ,))) for input in ranges]

for job in jobs:

sum += job()


the task object, returned by a submit call, has an argument finished

which indicates the status of the task and can be used to check if ithas been completed:

task = job_server.submit(f1, (a,b,c))

...

if task.finished:

print("The task is done!")

else

print("Still working on it ...")

you can perform an action at the time of completion of each individualtask by setting the callback argument of the submit method:

sum = 0

def add_sum(n):

sum += n

...

task = job_server.submit(sum_primes , (nstart , nend),

callback=add_sum)

mpi4py

MPI for Python provides full-featured bindings of the Message PassingInterface standard for the Python programming language

point-to-point (sends, receives) and collective (broadcasts, scatters,gathers) communications of any picklable Python object (via thepickle module) as well as buffer-providing objects (e.g. NumPy arrays)

(pickling: conversion of a Python object hierarchy into a byte stream)

provides an object oriented interface which closely follows the MPI-2C++ bindings and works with most of the MPI implementations

� any user of the standard C/C++ MPI bindings should be able to usethis module without the need of learning a new interface

Reference: http://mpi4py.scipy.org/

http://mpi4py.scipy.org/

mpi4py

allows any Python program to exploit multiple processors

allows wrapping of C/C++ and Fortran code that uses MPI withCython, SWIG and f2py

� you can run almost any MPI based C/C++/Fortran code from Python

integration with IPython

� enables MPI applications to be used interactively

Cython has bindings for mpi4py (mpi4py itself is written in Cython)

from mpi4py import MPI

from mpi4py cimport MPI

mpi4py: Hello World

simple example:

helloworld.py


comm = MPI.COMM_WORLD

rank = comm.Get_rank ()

size = comm.Get_size ()

print("Hello , World! I am process %d of %d." % (rank ,size))

execute with mpiexec:

> openmpiexec -n 4 python helloworld.py

Hello , World! I am process 2 of 4.




mpi4py: Point-to-Point Communication of Python Objects

point-to-point: transmission of data between a pair of processes

send() and recv() communicate typed data (Python objects)

ptp pyobj.py




if rank == 0:

data = {’a’: 7, ’b’: 3.14}

comm.send(data , dest=1, tag =11)

print("I am process %d. Sent data:" % (rank) )

print(data)

elif rank == 1:

data = comm.recv(source=0, tag =11)

print("I am process %d. Recieved data:" % (rank) )

print(data)

else:

print("I am process %d. I am bored." % (rank) )

mpi4py: Point-to-Point Communication of Python Objects

� the type information enables the conversion of data representationfrom one architecture to another

� allows for non-contiguous data layouts and user-defined datatypes

� the tag information allows selectivity of messages at the receiving end

output:

> openmpiexec -n 4 python ptp_pyobj.py

I am process 0 of 4. Sent data:

{’a’: 7, ’b’: 3.14}

I am process 1 of 4. Recieved data:

{’a’: 7, ’b’: 3.14}

I am process 2 of 4. I am bored.

I am process 3 of 4. I am bored.

mpi4py: Point-to-Point Communication of NumPy Arrays

Send() and Recv() communicate memory buffers (e.g. NumPy arrays)

pass explicit MPI datatypes:


import numpy as np



if rank == 0:

data = np.arange (100, dtype=np.float64)

comm.Send([data , MPI.DOUBLE], dest=1, tag =77)

elif rank == 1:


comm.Recv([data , MPI.DOUBLE], source=0, tag =77)

mpi4py: Point-to-Point Communication of NumPy Arrays

Send() and Recv() communicate memory buffers (e.g. NumPy arrays)

automatic MPI datatype discovery:


import numpy as np



if rank == 0:


comm.Send(data , dest=1, tag =77)

elif rank == 1:

data = np.empty (100, dtype=np.float64)

comm.Recv(data , source=0, tag =77)

mpi4py: Blocking and Nonblocking Communications

send() and recv() as well as Send() and Recv() are blocking functions

� block the caller until the data buffers involved in the communicationcan be safely reused by the application program

often, performance can be significantly increased by overlappingcommunication and computations

� nonblocking send and receive functions

� always come in two parts

posting functions: begin the requested operationtest-for-completion functions: discover whether the requested operationhas completed

� Isend() and Irecv() initiate a send and receive operation, respectively,and return a Request instance, uniquely identifying the started operation

� completion can be managed using the Test(), Wait(), and Cancel()

methods of the Request class

mpi4py: Collective Communications

simultaneous transmission of data between multiple processes

commonly used collective communication operations:global communication functions:

broadcast data from one node to all nodesscatter data from one node to all nodesgather data from all nodes to one node

barrier synchronisation across all nodesglobal reduction operations such as sum, maximum, minimum, etc.

collective functions communicate typed data or memory buffers

� bcast(), scatter(), gather(), allgather() and alltoall()

communicate generic Python object� Bcast(), Scatter(), Gather(), Allgather() and Alltoall()

communicate memory buffers

collective functions come in blocking versions only

collective messages are not paired with an associated tag

� selectivity of messages is implied in the calling order

mpi4py: Broadcasting

broadcast a Python list from one process to all processes:




if rank == 0:

data = [(i+1)**2 for i in range(size)]

else:

data = None

data = comm.bcast(data , root =0)

� each process receives the complete list

mpi4py: Scattering

scatter a Python list from one process to all processes:





if rank == 0:

data = [(i+1)**2 for i in range(size)]

else:

data = None

data = comm.scatter(data , root =0)

assert data == (rank +1)**2

� each process receives one array element (data[rank+1])

mpi4py: Gathering

gather together values from all processes to one process:





data = (rank +1)**2

data = comm.gather(data , root =0)

if rank == 0:

for i in range(size):

assert data[i] == (i+1)**2

else:

assert data is None

� the root process gathers a list whose entries are the data values of allprocesses

� allgather(sendbuf, recvbuf) gathers data from all processes anddistributes it to all other processes

mpi4py: Cython

mpi4py works in Cython just the same way it does in Python:

hello world.pyxdef hello(comm):



print("Hello , World! I am process %d of %d." % (rank ,size))

call from Python:

hello cython.pyfrom mpi4py import MPI

from helloworld_cython import hello


hello(comm)


> openmpiexec -n 4 python hello_cython.py





mpi4py: Cython

do some array calculations in Cython:

calc cython.pyimport numpy as np


from calc import square




if rank == 0:

data = np.arange (25* size)

data = np.split(data , size)

else:

data = None

data = comm.scatter(data , root =0)

squ = square(data)

squ = comm.gather(squ , root =0)

if rank == 0:

squ = np.concatenate(squ)

print(squ)

mpi4py: Cython

within Cython you can of course use the fast cdef functions:

calc.pyx

cimport numpy as np

cdef fast_square(np.ndarray[long , ndim =1] x):

return x*x

def square(np.ndarray[long , ndim =1] x):

return fast_square(x)


> openmpiexec -n 4 python calc_cython.py

[ 0 1 4 9 16 25 36 49 64 81 100 121

144 169 196 225 256 289 324 361 400 441 484 529

576 625 676 729 784 841 900 961 1024 1089 1156 1225

1296 1369 1444 1521 1600 1681 1764 1849 1936 2025 2116 2209

2304 2401 2500 2601 2704 2809 2916 3025 3136 3249 3364 3481

3600 3721 3844 3969 4096 4225 4356 4489 4624 4761 4900 5041

5184 5329 5476 5625 5776 5929 6084 6241 6400 6561 6724 6889

7056 7225 7396 7569 7744 7921 8100 8281 8464 8649 8836 9025

9216 9409 9604 9801]

mpi4py: Wrapping with SWIG

MPI hello world in C:

hello.h

#ifndef _HELLO_H_

#define _HELLO_H_

#include <mpi.h>

void sayhello(MPI_Comm );

#endif // _HELLO_H_

hello.c

#include "hello.h"

void sayhello(MPI_Comm comm) {

int size , rank;

MPI_Comm_size(comm , &size);

MPI_Comm_rank(comm , &rank);

printf("Hello , World! I am process %d of %d.\n",

rank , size);

}

SWIG interface:

hello.i

%module helloworld %{

#include "hello.h"

}%

%include mpi4py/mpi4py.i

%mpi4py_typemap(Comm , MPI_Comm );

%include <hello.h>;

create wrapper and compile library:

> swig -I$(MPI4PY_SWIG_DIR) -python hello.i

> openmpicc ‘python -config --cflags ‘ -I$(MPI4PY_INCLUDE_DIR) \

-c hello_wrap.c hello.c

> openmpicc ‘python -config --ldflags ‘ -shared -o _hello_swig.so \

hello_wrap.o hello.o


call from Python:

hello swig.py

>>> from mpi4py import MPI

>>> import hello

>>> hello.sayhello(MPI.COMM_WORLD)

� the SWIG typemap takes care of translating the mpi4py communicatorinto a C object, so we can pass it directly


> openmpiexec -n 4 python hello_swig.py





mpi4py: Wrapping with f2py

MPI hello world in Fortran:

helloworld.f90

subroutine sayhello(comm)

use mpi

implicit none

integer :: comm , rank , size , ierr

call MPI_Comm_size(comm , size , ierr)

call MPI_Comm_rank(comm , rank , ierr)

print *,’Hello , World! I am process ’,rank ,’ of ’,size ,’.’

end subroutine sayhello

compile with f2py (almost) as usual:

> f2py --fcompiler=gnu95 --f90exec=openmpif90 \

-m helloworld -c helloworld.f90

� you only have to specify the MPI Fortran compiler with --f90exec

mpi4py: Wrapping with f2py

call from Python:

hello f2py.py


import helloworld


helloworld.sayhello(comm.py2f ())

� the py2f() function provides a Fortran communicator


> openmpiexec -n 4 python hello_f2py.py

Hello , World! I am process 2 of 4 .




Symbolic Computing with Sage

Sage is an open-source mathematics software system based on Python

� combines nearly 100 packages under a unified interface

� includes a huge range of mathematics, including basic algebra,calculus, elementary to very advanced number theory, cryptography,numerical computation, commutative algebra, group theory,combinatorics, graph theory, exact linear algebra and much more

� the user interface is a notebook in a web browser or the command line

� it’s a viable, free alternative to Maple, Mathematica, and MATLAB

References: http://www.sagemath.org/

Sage: Beginner’s Guide (2011)Craig Finch

http://www.sagemath.org/

introduction to python · swig connects programs written in c/c++ with a variety of high-level...

Documents