pofeaa and sqlalchemy

PofEAA andSQLAlchemy

INADA Naoki@methane

Public Activity

MessagePack-PythonJSON like binary protocol

wsaccelWebSocket accelerator for Tornado, ws4py

MyKazePyMySQL in Tornado (WIP)

Python for PHPerQuick guide of Python for PHPer (Japanese)

Online Game Developer

R&D and ArchitectTuning server side middleware and application

KLab Inc.

SQLAlchemy

Great O/R Mapperfor Python

Patterns ofEnterprise Application

Architecture

Design patternsgood to know

for Web Programmers

SQLAlchemyQuickly

(from tutorial)

● Dialects -- Various DB-API wrapper

● Engine -- Connection management

● Schema and Types

● SQL expression

● and O/R Mapper

SQLAlchemy features

Create Enginefrom sqlalchemy import create_engine

engine = create_engine('sqlite:///:memory:',

echo=True)

# Using engine without O/R mapper

con = engine.connect()

with con.begin() as trx:

con.execute('SELECT 1+1')

con.close()

Define Object and Schemafrom sqlalchemy.ext.declarative import declarative_base

from sqlalchemy import Column, Integer, String

Base = declarative_base()

class User(Base):

__tablename__ = 'users'

id = Column(Integer, primary_key=True)

name = Column(String)

email = Column(String, nullable=False, unique=True)

password = Column(String, nullable=False)

Create table on DB

Base.metadata.create_all(engine)

Output:CREATE TABLE users (

id INTEGER NOT NULL,

name VARCHAR,

email VARCHAR NOT NULL,

password VARCHAR NOT NULL,

PRIMARY KEY (id),

UNIQUE (email)

)

Create an instance>>> ed_user = User(name='ed',

... email='[email protected]',

... password='edspassword')

...

>>> ed_user.password

'edspassword'

>>> str(ed_user.id)

'None'

Save itfrom sqlalchemy.orm import sessionmaker

Session = sessionmaker(bind=engine)

session = Session()

session.add(ed_user)

session.commit()

Too complicated?

Why not just

engine.save(user)

What's session?

Session is Unit of Work

Unit of Work isa Pattern

in PofEAA

Part 1. The Narratives3. Mapping to RDB

Part 2. The Patterns10. Data Source Architectural Patterns11. O/R Behavioral Patterns12. O/R Structural Patterns13. O/R Metadata Mapping Patterns

O/R chapters in the P of EAA

11. O/R Behavioral Patterns

● Unit of Work

● Identity map

● Lazy load

Unit of Work

Target: Maintains set of objects to save in transaction.

How:1. Wrap transaction.2. Keep track of new, modified and deleted objects.3. Save objects when commit

Why Unit of Work

Remove boilerplate saving code from domain logic.

Avoid objects saved too often.

Unit of Work in SQLAlchemy# Create an Unit of Work

session = Session()

# Get connection and start transaction

ed_user = session.query(User).\

filter(User.name=='ed').one()

ed_user.name='E.D.'

session.commit() # save ed_user and commit

Object states

user = User(...)

# Transient (not saved)

session.add(user)

# pending (saved)

session.flush()

# persistent (saved if modified)

session.expunge(user)

# Detached (not saved)

Flushing

Execute insert / update /delete query.

Session flushes automatically when queryingto avoid inconsistent result. (autoflush)

You can stop autoflush.You can manually flush.

expire -- cause reload

session.expire(user)print(user.name) # Cause reloading

commit() # expires all objects for consistency.

You can avoid this by expire_on_commit=False.

Contextual Sessionfrom sqlalchemy.orm import (sessionmaker,

scoped_session)

Session = scoped_session(

sessionmaker(bind=engine))

Session.add(ed_user)

Session.commit()

...

Session.remove()


● Unit of Work

● Identity map

● Lazy load

Target:Avoid two objects for one record.

How:Keep mapping of (table, pk) -> object.Check the map when querying.

Bonus: It acts like a easy cache

Identity map

Identity map in SQLAlchemy

Session has identity map.

# Looks up id map after queryingsession.query(User).filter(User.id==3).one()

# Looks up id map before and after querying.session.query(User).get(3)

NOTE: Id map is weak reference by default.


● Unit of Work

● Identity map

● Lazy load

Lazy Load

Load relation or heavy column on demand.

Lazy loading relationship

http://docs.sqlalchemy.org/en/rel_0_8/orm/loading.html

SQLAlchemy uses lazy loading by default for relationship.

You can choose eager loading strategy.




Lazy loading big column

http://docs.sqlalchemy.org/en/rel_0_8/orm/mapper_config.html#deferred-column-loading

SQLAlchemy supports deferred column tolazy loading big BLOB/TEXT.





O/R Structural Pattern

12. O/R Structural Pattern

● Identity Field● Foreign Key Mapping● Association Table Mapping● Dependent Mapping● Embedded Value● Serialized LOB● Single Table Inheritance● Class Table Inheritance● Concrete Table Inheritance● Inheritance Mappers

Object have PK in database as a field.

class Player(Base):

id = Column(INTEGER, primary=True)

Identity Field

Map one to one or one to many relation toObject reference.

Not:session.query(Address).get(user.address_id)

Yes:user.address

Foreign Key Mapping

With propertyfrom werkzeug.utils.cached_property

class User(Base):

...

@cached_property

def address(self):

session.query(Address).\

get(self.address_id)

This is enough to achieve ``user.address``.

But no SQLA support.

relationshipclass User(Base):

...

addresses = relationship('Address',

backref='user')

class Address(Base):

...

user_id = Column(Integer,

ForeignKey('user.id'))

Association Table Mapping

Mapping many-to-many association table toobject reference.

In SQLAlchemy:Pass “secondary” argument to relationship()http://docs.sqlalchemy.org/en/rel_0_8/orm/tutorial.html#building-a-many-to-many-relationship

http://docs.sqlalchemy.org/en/rel_0_8/orm/tutorial.html#building-a-many-to-many-relationship



Dependent Mapping

O/R map ownership without identity.

PofEAA says:

I don’t recommend Dependent Mapping if you’re using Unit of Work.

Embedded Value

Map an object to some columns in a row.

class Duration: start_date end_date

class Account: duration

CREATE TABLE account ( … start_date DATE, end_date DATE, …);

Embedded Value by property@property

def duration(self):

return Duration(

self.start_date, self.end_date)

@duration.setter

def duration(self, duration):

self.start_date = duration.start

self.end_date = duration.end

Embedded Value by Compositehttp://docs.sqlalchemy.org/en/rel_0_8/orm/mapper_config.html#mapper-composite

class Account(Base):

...

start_date = Column(DATE)

end_date = Column(DATE)

duration = composite(Duration, start_date, end_date)

class Duration(namedtuple(‘Duration’, ‘start end’)):

def __composite_value__(self):

return self

http://docs.sqlalchemy.org/en/rel_0_8/orm/mapper_config.html#mapper-composite



Serialized LOB

Serialize objects and save to XLOB column.

Using property

_info = Column('info', BLOB)

@property

def info(self):

return json.loads(self._info)

@info.setter

def _set_info(self, info):

self._info = json.dumps(self)

Custom Type

SQLAlchemy provides PickleType for serialized LOB.

You can define custom type via TypeDecorator:http://docs.sqlalchemy.org/en/rel_0_8/core/types.html#marshal-json-strings

http://docs.sqlalchemy.org/en/rel_0_8/core/types.html#marshal-json-strings



Single Table Inheritance

Player

SoccerPlayer BaseballPlayer

CREATE TABLE player( id INTEGER PRIMARY KEY, type INTEGER NOT NULL, position INTEGER,)

All classes saved into one table.

Single Table Inheritance in SQLAclass Player(Base):

__tablename__ = ‘player’

id = Column(INTEGER, primary_key=True)

position = Column(INTEGER)

type = Column(INTEGER, nullable=False)

__mapper_args__ = {‘polymorphic_on’: type}

SOCCER_PLAYER = 1

BASEBALL_PLAYER = 2

class SoccerPlayer(Player):

__mapper_args__ = {

‘polymorhic_identity’: Player.SOCCER_PLAYER}

Defining columns in subclass

Defining columns in subclass may cause conflict.

SQLA provides way to avoid it. See below.http://docs.sqlalchemy.org/en/latest/orm/extensions/declarative.html#resolving-column-conflicts

http://docs.sqlalchemy.org/en/latest/orm/extensions/declarative.html#resolving-column-conflicts




Class Table Inheritance

Player


CREATE TABLE player( id INTEGER PRIMARY KEY, type INTEGER NOT NULL)CREATE TABLE soccer_player( id INTEGER PRIMARY KEY, position INTEGER NOT NULL)CREATE TABLE baseball_player( id INTEGER PRIMARY KEY, position INTEGER NOT NULL)

Tables for each classes.

Class table inheritance in SQLA

SQLA call it “Joined table inheritance”http://docs.sqlalchemy.org/en/latest/orm/extensions/declarative.html#joined-table-inheritance

http://docs.sqlalchemy.org/en/latest/orm/extensions/declarative.html#joined-table-inheritance



class Player(Base):

__tablename__ = 'player'


type = Column(INTEGER, nullable=False)

SOCCER_PLAYER = 1

BASEBALL_PLAYER = 2

__mapper_args__ = {'polymorphic_on': type}

class SoccerPlayer(Player):

__tablename__ = 'soccer_player'

id = Column(ForeignKey('player.id'), primary_key=True)

position = Column(INTEGER, nullable=False)

__mapper_args__ = {

'polymorhic_identity': Player.SOCCER_PLAYER}

Concrete Table Inheritance

Player


CREATE TABLE soccer_player( id INTEGER PRIMARY KEY, name VARCHAR(32) NOT NULL, position INTEGER NOT NULL)

CREATE TABLE baseball_player( id INTEGER PRIMARY KEY, name VARCHAR(32) NOT NULL, position INTEGER NOT NULL)

Tables for each concrete classes

Mix-inclass BasePlayer(object):


name = Column(VARCHAR(32), nullable=False)

class SoccerPlayer(BasePlayer, Base):

__tablename__ = ‘soccer_player’

position = Column(INTEGER, nullable=False)

Pros) Simple.Cons) You can’t get support from SQLAlchemy

Concrete Table Inheritance in SQLA

http://docs.sqlalchemy.org/en/latest/orm/inheritance.html#concrete-table-inheritance




10. Data Source Architectural Patterns

● Table Data Gateway

● Row Data Gateway

● Active Record

● Data Mapper

● Architectural Pattern and SQLAlchemy

Table Data Gateway

Target: Split querying from your domain logicHow: Create gateway for each table.

class PlayerGateway:

def find_by_id(self, id)

def find_by_name(self, name)

def update(self, id, name=None, age=None)

def insert(self, id, name, age)

Row Data Gateway

Target: Split querying from your domain logicHow: Create gateway for each record.

class PlayerGateway:

@classmethod

def find_by_id(cls, id):

… return cls(id, name, age)

… def insert(self)

def update(self)

Active Record

Row Data Gateway with Domain Logic

class Player:

@classmethod

def find_by_id(cls, id)

...

def birthday(self):

self.age += 1

Data Mapper

Target: Split out column/attribute mapping code from your domain logic.

class PersonMapper(BaseMapper):

def map(self, row):

return Person(id=row.id,

name=row.name,

age=row.age)

Architectural Patternsand SQLAlchemy

My personal consideration

Querying in SQLAlchemy

Unit of Work handles most of update and insert queries.

Easy query can be written very easy.(Building complex query is hard)

Table Gateway?

Most tables doesn’t require Table Gateway.

When there are some complex queries,Table Gateway is good to have.

Row Data Gateway?Active Record?

Separating model object and row data gateway cause additional complexity.For example, does your model need identity map?

Person

Person

PersonRow

Active Record drawbacks

Problem: Good table design is not good class design always.My answer: Use structural patterns like Embedded Value.

P: Mixing database access and domain logic is bad idea.A: Use Table Gateway to separate them.

Fat Model (™)

Person.register(...) # class methodperson.unregister()person.request_friend(other_person_id)...

Simple ActiveRecord & Serviceclass AccountService:

def register(self, name, email, age,...)

def unregister(self, person_id)

class FriendService:

def request(self,

from_person_id, to_person_id)

Finder in ServiceWhen you use service classes, you may be able to put finders there and don’t use Table Gateway.

class PersonService:

...

def find_by_name(self, name):

return Session.query(Person). \

filter_by(name=name).all()

Anemic Domain Model

http://www.martinfowler.com/bliki/AnemicDomainModel.html

P: Implementing all domain logic in service is not a OOP.

A: Simple Active Record + Medium sized classes + Service




When you design large application or framework,

PofEAA helps you lot.

pofeaa and sqlalchemy

Technology