python speleology

Getting deep in python features

Python Speleology

1. The Zen of Python

The Zen of Python

○ Beautiful is better than ugly.○ Explicit is better than implicit.○ Simple is better than complex.○ Complex is better than complicated.○ Flat is better than nested.○ Sparse is better than dense.○ Readability counts.○ Special cases aren't special enough to break the rules.○ Although practicality beats purity.○ Errors should never pass silently.○ Unless explicitly silenced.○ In the face of ambiguity, refuse the temptation to guess.○ There should be one -and preferably only one- obvious way to do it.○ Although that way may not be obvious at first unless you're Dutch.○ Now is better than never.○ Although never is often better than right now.○ If the implementation is hard to explain, it's a bad idea.○ If the implementation is easy to explain, it may be a good idea.○ Namespaces are one honking great idea -let's do more of those!

2. Types and objects

Types and objects: To be or not to be

a = 256

b = 256

print (a is b)

a = 257

b = 257

print (a is b)


a = 256

b = 256

print (a is b)

a = 257

b = 257

print (a is b)

True

False


a = 256

b = 256

print (a is b)

a = 257

b = 257

print (a is b)

True

False

a = 256

b = 256

print id(a), id(b)

a = 257

b = 257

print id(a), id(b)

22036112 22036112

22363568 22363640


a = 256

b = 256

print (a is b)

a = 257

b = 257

print (a is b)

True

False

a = 256

b = 256

print id(a), id(b)

a = 257

b = 257

print id(a), id(b)


a = 256

b = 256

print (a is b)

a = 257

b = 257

print (a is b)

True

False

a = 256

b = 256

print id(a), id(b)

a = 257

b = 257

print id(a), id(b)

22036112 22036112

22363568 22363640

Types and objects: functions are objects

def x(f):

return f

def y(f):

return x

print x(y)(8)(0)


def x(f):

return f

def y(f):

return x

print x(y)(8)(0)

0


def x(i):

if x.enabled:

return i

else:

return "disabled"

x.enabled = True

print x(8)


def x(i):

if x.enabled:

return i

else:

return "disabled"

x.enabled = True

print x(8)

What happened if not set x.enabled?

8

Types and objects: star operator

args = (3 , 6)

print range(*args)

args = { "name": "example" }

def f(name):

print name

f(**args)

def f(*args, **kwargs):

print args

print kwargs

f(1,2,3, name="example")

Types and objects: star operator

args = (3 , 6)

print range(*args)

args = { "name": "example" }

def f(name):

print name

f(**args)

def f(*args, **kwargs):

print args

print kwargs

f(1,2,3, name="example")

[3, 4, 5]

example

(1, 2, 3)

{'name': 'example'}

3. Reflection

Reflection: get & set

class Example(object):

num = 10

x = Example

dir(x)

hasattr(x, "num") == True

getattr(x, "num", 0) == 10

setattr(x, "num", 20)

Reflection: local & global

globals()

Return a dictionary representing the current global symbol table. This is always the dictionary of the current module (inside a function or method, this is the module where it is defined, not the module from which it is called).

locals()

Update and return a dictionary representing the current local symbol table. Free variables are returned by locals() when it is called in function blocks, but not in class blocks.

Reflection: __magic__

__name__

This is the name of the function. This only have a meaningful value is the function is defined with “def”.

__class__

This is a reference to the class a method belongs to.

__code__

This is a reference to the code object used in the implementation of python.

Reflection: inspector

from inspect import getcomments

# This is a comment

def f(x):

print x

print getcomments(f)


from inspect import getcomments

# This is a comment

def f(x):

print x

print getcomments(f)

# This is a comment


from inspect import getsource

# This is a comment

def f(x):

print x

print getsource(f)


from inspect import getsource

# This is a comment

def f(x):

print x

print getsource(f)

def f(x):

print x

Reflection: let's more tricky

def f(x):

print x

print f.__code__.co_code


def f(x):

print x


'd\x01\x00GHd\x00\x00S'


def f(x):

print x


'd\x01\x00GHd\x00\x00S'

YEEES!!! THE BYTECODE!!!

4. Context manager

Context manager: in the beginning...

item = Item()

try:

item.open()

item.do()

finally:

item.close()

Context manager: nowadays...

with Item() as item:

item.do

class Item(object):

def __enter__(self):

self.open()

return self

def __exit__(self,exc_type,exc_value,exc_t):

self.close()

...

Context manager: the real world

with file("/tmp/test", "w") as f:

f.write("hello world")

with lock():

# do some concurrent

with sudo("root"):

# do some as root

5. Decorations

Decorations: bold & italic

def makebold(fn):

def wrapped():

return "" + fn() + ""

return wrapped

def makeitalic(fn):

def wrapped():

return "" + fn() + ""

return wrapped

@makebold

@makeitalic

def hello():

return "hello world"

print hello()

Decorations: bold & italic

def makebold(fn):

def wrapped():

return "" + fn() + ""

return wrapped

def makeitalic(fn):

def wrapped():

return "" + fn() + ""

return wrapped

@makebold

@makeitalic

def hello():


print hello()

hello world

Decorations: complex decor

def makestyle(arg):

def decorator(f):

def wrapper(*args, **kw):

return "<" + arg + ">" + f() + "</" + arg + ">"

return wrapper

return decorator

@makestyle("b")

def hello():


print hello()

Decorations: syntax sugar

def makebold(fn):

def wrapped():

return "" + fn() + ""

return wrapped

def makestyle(arg):

def decorator(f):

def wrapper(*args, **kw):

return "<" + arg + ">" + f() + "</" + arg + ">"

return wrapper

return decorator

makebold(hello)

makestyle("b")(hello)

6. Iterations

Iterations: comprehesions

squares = []

for x in range(10):

squares.append(x**2)

squares = [x**2 for x in range(10)]

[(x, y) for x in [1,2,3] for y in [3,1,4] if x != y]

{ (k,v) for k,v in [(1,2)] }


squares = []

for x in range(10):




{ (k,v) for k,v in [(1,2)] }

SET NOT DICT!


squares = []

for x in range(10):




{ (k,v) for k,v in [(1,2)] }

SET NOT DICT!

{ k:v for k,v in [(1,2)] }

Iterations: co-routine

def countdown(n):

print "Counting down from", n

while n > 0:

yield n

n -= 1

print "Done counting down"

for i in countdown(10):

print i


def countdown(n):


while n > 0:

yield n

n -= 1


for i in countdown(10):

print i

Counting down from 10

10

9

8

7

6

5

4

3

2

1

Done counting down

def countdown(n):


while n > 0:

yield n

n -= 1


print countdown(10)

<generator object at 0x4035874c>

7. Overloading

Overloading: binary operations

__add__(self, other) x + y

__sub__(self, other) x - y

__mul__(self, other) x * y

__div__(self, other) x / y

__pow__(self, other) x ** y


__radd__(self, other) y + x

__rsub__(self, other) y - x

__rmul__(self, other) y * x

__rdiv__(self, other) y / x

__rpow__(self, other) y ** x


__radd__(self, other) 1 + x

__rsub__(self, other) 1 - x

__rmul__(self, other) 1 * x

__rdiv__(self, other) 1 / x

__rpow__(self, other) 1 ** x


__iadd__(self, other) x+=y

__isub__(self, other) x-=y

__imul__(self, other) x*=y

__idiv__(self, other) x/=y

__ipow__(self, other) x**=y

Overloading: unary operations

__neg__(self) -x

__pos__(self) +x

__abs__(self) abs(x)

__invert__(self) ~x

Overloading: conversion operations

__int__(self) int(x)

__float__(self) float(x)

__complex__(self) complex(x)

__str__(self) str(x)

__nonzero__(self) bool(x)

__unicode__(self) unicode(x)

Overloading: comparison operations

__eq__(self, other) x == y

__lt__(self, other) x < y

__le__(self, other) x <= y

__gt__(self, other) x > y

__ge__(self, other) x >= y

__ne__(self, other) x != y

Overloading: containers

__contains__(self, other) y in x

__getitem__(self, other) x[y]

__setitem__(self, other,value) x[y] = z

__delitem__(self, other) del x[y]

__len__(self) len(x)

__reversed__(self) reversed(x)

__iter__(self) iter(x)

8. The Class Factory

The Class Factory: class & objects


attribute = "this is a class attribute"

def __init__(self):

self.attribute = "this is an obj attr override class one"

self.another = "this is another obj attr, no class"

print Example.attribute

print Example().attribute

print Example().another

print Example.another

The Class Factory: class & objects


attribute = "this is a class attribute"

def __init__(self):

self.attribute = "this is an obj attr override class one"

self.another = "this is another obj attr, no class"

print Example.attribute

print Example().attribute

print Example().another


this is a class attribute

this is an object attribute and override class one

this is another object attribute, no class

Traceback (most recent call last):

Line 11, in <module>


AttributeError: type object 'Example' has no attribute 'another'

http://codepad.org/H73aJvdJ#line-11

http://codepad.org/H73aJvdJ#line-11

The Class Factory: set & get


def __init__(self):

self._name = x

@property

def name(self):

return self._name

@name.setter

def name(self, value):

self._name = value

The Class Factory: @property abuse


def __init__(self, host, port):

self.host = host

self.port = port

@property

def connect(self):

lib.connect(self.host, self.port)

The Class Factory: @property abuse


def __init__(self, host, port):

self.host = host

self.port = port

@property

def connect(self):

lib.connect(self.host, self.port)

NEVER TYPE METHODS AS PROPERTIES

The Class Factory: methods and more methods

@staticmethod

Nothing more than a function defined inside a class. It is callable without instantiating the class first. It’s definition is immutable via inheritance.

@classmethod

Also callable without instantiating the class, but its definition follows Sub class, not Parent class, via inheritance. That’s because the first argument for @classmethod function must always be cls (class).


@staticmethod

Nothing more than a function defined inside a class. It is callable without instantiating the class first. It’s definition is immutable via inheritance.

@classmethod

Also callable without instantiating the class, but its definition follows Sub class, not Parent class, via inheritance. That’s because the first argument for @classmethod function must always be cls (class).

@staticmethod

def static_method():

print "I do not receive nothing :("

@classmethod

def class_method(cls):

print "I'm a class %s" % str(cls)



def __init__(self, name):

self.name = name

@classmethod

def class_method(cls, name):

return cls(name)

x = Example.class_method("example")

print x

self.name = name

@classmethod

def class_method(cls, name):

return cls(name)

x = Example.class_method("example")

print x

<__main__.Example object at 0x40358b2c>

The Class Factory: children and parents



self.name = name

def do_something(self):

raise NotImplementedError()

class ChildExample(Example):


print self.name

The Class Factory: children and parents

class ExampleA(object):


self.name = name



class ExampleB(object):

def do_otherthing(self):


class ChildExample(ExampleB, ExampleA):


print self.name

def do_otherthing(self):

print self.name

The Class Factory: τὰ μετὰ τὰ κλάση


pass

x = Example()

y = Example

print x

print y

pass

x = Example()

y = Example

print x

print y

<__main__.Example object at 0x4035894c>

<class '__main__.Example'>


def example():


pass

return Example

x = example()

y = x()

print x

print y

def example():


pass

return Example

x = example()

y = x()

print x

print y


<__main__.Example object at 0x403589ec>



pass

print type(Example)

print type(Example())

pass

print type(Example)

print type(Example())

<type 'type'>



pass

x = Example

y = type('Example', (object,), {})

print x

print y

print (x == y)



pass

x = Example

y = type('Example', (object,), {})

print x

print y

print (x == y)



False

The Class Factory: __magic__

__new__(cls, *args, **kwargs)

Is the first method to get called in an object's instantiation, is a @classmethod, and must return a new instance of type cls.

__init__(self, *args, **kwargs)

Is the initializer for the instance. It gets passed whatever the primary constructor was called with.

__del__(self)

Is the destructor of the instance, will be invoked before clean the reference to the instance.

The Class Factory: __metaclass__

def upper_attr(f_class_name, f_class_parents, f_class_attr):

attrs = ((name, value) \

for name, value in f_class_attr.items() \

if not name.startswith('__'))

uppercase_attr = dict((name.upper(), value) \

for name, value in attrs)

return type(f_class_name, f_class_parents, uppercase_attr)

class Foo(object):

__metaclass__ = upper_attr

bar = 'bip'

print hasattr(Foo, 'bar')

print hasattr(Foo, 'BAR')

The Class Factory: __metaclass__

def upper_attr(f_class_name, f_class_parents, f_class_attr):

attrs = ((name, value) \

for name, value in f_class_attr.items() \

if not name.startswith('__'))

uppercase_attr = dict((name.upper(), value) \

for name, value in attrs)

return type(f_class_name, f_class_parents, uppercase_attr)

class Foo(object):

__metaclass__ = upper_attr

bar = 'bip'

print hasattr(Foo, 'bar')

print hasattr(Foo, 'BAR')

False

True

9. Monkey Patching

Monkey patching: first try

class Person(object):

def speak(self):

print "hello"

def monkey(foo):

print "uh uh uh"

Person.speak = monkey

x = Person()

x.speak()

Monkey patching: be evil }:-)

import os

def monkey(*args, **kwargs):

print "no no no"

os.system = monkey

os.system("ls /tmp")

Monkey patching: be evil }:-)

import os

def monkey(*args, **kwargs):

print "no no no"

os.system = monkey

os.system("ls /tmp")

no no no

10. Profiling

Profiling: hand made

import time

class Timer(object):

def __init__(self, verbose=False):

self.verbose = verbose


self.start = time.time()

return self

def __exit__(self, *args):

self.end = time.time()

self.secs = self.end - self.start

self.msecs = self.secs * 1000 # millisecs

if self.verbose:

print 'elapsed time: %f ms' % self.msecs

Profiling: hand made

import time

class Timer(object):

def __init__(self, verbose=False):

self.verbose = verbose


self.start = time.time()

return self

def __exit__(self, *args):

self.end = time.time()

self.secs = self.end - self.start

self.msecs = self.secs * 1000 # millisecs

if self.verbose:

print 'elapsed time: %f ms' % self.msecs

from redis import Redis

rdb = Redis()

with Timer() as t:

rdb.lpush("foo", "bar")

print "=> elasped lpush: %s s" % t.

secs

with Timer as t:

rdb.lpop("foo")

print "=> elasped lpop: %s s" % t.secs

Profiling: profile & cProfile

try:

import cProfile as profile

except ImportError:

import profile

def fib(n):

if n == 0:

return 0

elif n == 1:

return 1

else:

return fib(n-1) + fib(n-2)

def fib_seq(n):

seq = [ ]

if n > 0:

seq.extend(fib_seq(n-1))

seq.append(fib(n))

return seq

profile.run('print fib_seq(6); print')

Profiling: profile & cProfile

try:

import cProfile as profile

except ImportError:

import profile

def fib(n):

if n == 0:

return 0

elif n == 1:

return 1

else:

return fib(n-1) + fib(n-2)

def fib_seq(n):

seq = [ ]

if n > 0:

seq.extend(fib_seq(n-1))

seq.append(fib(n))

return seq

profile.run('print fib_seq(6); print')

[0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181, 6765]

57356 function calls (66 primitive calls) in 0.746 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)

21 0.000 0.000 0.000 0.000 :0(append)

20 0.000 0.000 0.000 0.000 :0(extend)

1 0.001 0.001 0.001 0.001 :0(setprofile)

1 0.000 0.000 0.744 0.744 <string>:1(<module>)

1 0.000 0.000 0.746 0.746 profile:0(print fib_seq(20); print)

0 0.000 0.000 profile:0(profiler)

57291/21 0.743 0.000 0.743 0.035 profile_fibonacci_raw.py:13(fib)

21/1 0.001 0.000 0.744 0.744 profile_fibonacci_raw.py:22(fib_seq)

11. Documentation

Documentation: Zen

Don't create documentation for your code.Code your documentation.

def elements(n):

"""Return a list of n numbers from 0 to n-

1.

"""

return range(0,n)

Documentation: everything is an object

def elements(n):


1.

"""

return range(0,n)

print elements.__doc__

Documentation: everything is an object

def elements(n):


1.

"""

return range(0,n)

print elements.__doc__

Return a list of n numbers from 0 to n-1

Documentation: style is important

def elements(n):

"""Return a list of n numbers from 0 to n-1.

:type n: int

:param n: the limit upper for the elements to be created (not

included).

:return: a class:`list` with the items.

"""

return range(0,n)


"""This is the documentation of the class.

Usually you do not need it :)

"""

def __init__(self, param):

"""This is the documentation of the instance type.

"""

Documentation: do the work

$ sphinx-quickstart

Documentation: Bonus... abuse the doc

class Command(object):

"""Undocumented command

"""

class ListCommand(Command):

"""List command.

"""

def show_help(command):

cmd = getattr(globals()[__name__], "%sCommand" % command,

None)

if cmd is None:

return "Invalid command"

else:

return cmd.__doc__

print show_help("List")

12. Future

Future: pypy

Just in time compiler FAST!

With sandboxing

Best concurrency support

Future: python3

def hello(name: str, age: int) -> str:

return name

print hello.__anotations__

Future: python3

def hello(name: str, age: int) -> str:

return name

print hello.__anotations__

'return':<class'int'>,'name':<class'str'>,'age':<class'int'>}{'return':<class'int'>,'name':<class'str'>,'age':<class'int'>}

Future: python3

from functools import lru_cache

@lru_cache(maxsize=None)

def fib(n):

if n < 2:

return n

return fib(n-1)+fib(n-2)

Applauses & questionsNot necessarily in that order.

python speleology

Technology

return wrappedmakeboldmakeitalicdef

binary operations

return wrappeddef

print xprint

print xprint

print xprint

arg

print elapsed