ArchitectRelease

May 01, 2017

Contents

1 Features 3

2 Dependencies 5

3 Contacts and Support 7

4 Donations and Sponsorship 9

5 Thanks 11

6 Copyright and License 13

7 Table of contents 157.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

7.1.1 PyPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157.1.2 GitHub . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

7.2 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167.2.1 Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167.2.2 Table partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177.2.3 Custom features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

7.3 Exceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227.4 License . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237.5 Changelog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

7.5.1 0.5.6 (2017-05-01) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247.5.2 0.5.5 (2016-10-08) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247.5.3 0.5.4 (2016-07-04) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247.5.4 0.5.3 (2015-11-08) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247.5.5 0.5.2 (2015-07-31) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247.5.6 0.5.1 (2015-06-08) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247.5.7 0.5.0 (2015-05-08) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257.5.8 0.4.0 (2015-04-18) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257.5.9 0.3.0 (2015-04-05) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257.5.10 0.2.0 (2014-07-19) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257.5.11 0.1.0 (2014-07-13) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Python Module Index 27

i

ii

Architect, Release

Architect is a set of tools which enhances ORMs written in Python with more features and simplifies different databaserelated complex tasks.

Contents 1

Architect, Release

2 Contents

CHAPTER 1

Features

• Supported ORMs

– Django >= 1.4

– Peewee >= 2.2.0

– Pony >= 0.5.0

– SQLAlchemy >= 0.8.0

– SQLObject >= 1.5.0

• Supported DBs

– PostgreSQL >= 8.0

– MySQL >= 5.5

• Supports Python 2.6 - 3.6

• Extensively documented

3

Architect, Release

4 Chapter 1. Features

CHAPTER 2

Dependencies

Architect has no required external dependencies except for the ORM it will be used with.

5

Architect, Release

6 Chapter 2. Dependencies

CHAPTER 3

Contacts and Support

I will be glad to get your feedback, pull requests, issues, whatever. Feel free to contact me for any questions.

7

Architect, Release

8 Chapter 3. Contacts and Support

CHAPTER 4

Donations and Sponsorship

If you like this project and want to support it you have several options:

1. Just give this project a star at the GitHub repository.

2. Become a sponsor. Contact me via tepkeev at gmail dot com if you are interested in becoming asponsor and we will discuss the terms and conditions.

9

Architect, Release

10 Chapter 4. Donations and Sponsorship

CHAPTER 5

Thanks

PremiumSoft for supporting this project by providing Open Source License of Navicat.

11

Architect, Release

12 Chapter 5. Thanks

CHAPTER 6

Copyright and License

Architect is licensed under Apache 2.0 license. Check the License for details.

13

Architect, Release

14 Chapter 6. Copyright and License

CHAPTER 7

Table of contents

Installation

PyPI

The recommended way to install is from Python Package Index (PyPI) with pip:

$ pip install architect

or with easy_install:

$ easy_install architect

GitHub

Architect is actively developed on GitHub. If you want to get latest development sources you have to clone therepository:

$ git clone git://github.com/maxtepkeev/architect.git

Once you have the sources, you can install it into your site-packages:

$ python setup.py install

You can also install latest stable development version via pip:

$ pip install git+https://github.com/maxtepkeev/architect.git@master

15

Architect, Release

Features

Architect has a concept of “features”. A feature in Architect is something that can somehow enhance the ORMArchitect is used with, e.g. table partitioning. Architect provides a single interface to all of its features despite of theORM being used.

All features in Architect are installed into models using install decorator. An install decorator in it’s generalform can be written as the following:

import architect

@architect.install(feature, **options)class Model(object):

pass

where feature is the feature name as a string which should be installed into the model class and options are theoptions that this feature takes (if any) in the form of keyword arguments. If a model wants to use several features, thiscan easily be done with applying several decorators at once, like this:

import architect

@architect.install(feature1, **options1)@architect.install(feature2, **options2)@architect.install(feature3, **options3)class Model(object):

pass

All installed features can be found inside architect namespace inside the model, e.g.:

>>> print(dir(Model.architect))['__class__', '__dict__', 'feature1', 'feature2', 'feature3']

This provides 100% non-conflicting behaviour with all ORMs and other 3rd party packages.

Note: Architect tries its best to automatically determine what ORM it is being used with, however there can besituations when it can fail to properly do that. To help Architect, one can pass an orm option to the installdecorator and set it to the needed ORM.

Under some circumstances a model might not need one or all of the installed features anymore, one example can be amodel inheritance, where the parent model has a feature installed and a child model doesn’t want to use it. A featurecan be easily uninstalled from a model using uninstall decorator:

import architect

@architect.uninstall(feature)class ChildModel(object):

pass

where a feature is a feature name as a string which was installed before.

Operations

Sometimes there is a need to execute raw SQL statements, unfortunately different ORMs provide different APIs towork with raw SQL. This feature creates an abstraction layer to execute raw SQL statements which will work with any

16 Chapter 7. Table of contents

Architect, Release

supported ORM. It is used by all other Architect features internally and usually doesn’t need to be installed separately.However, if this is the only feature that will be used in the model it can be installed like this:

import architect

@architect.install('operation')class Model(object):

pass

After the installation operation feature can be accessed via Model.architect.operation. It provides thefollowing methods:

execute(sql, autocommit=True)Executes raw SQL for write operations.

Parameters

• sql (string) – (required). SQL statement to execute.

• autocommit (boolean) – (optional). Turns autocommit on/off.

select_one(sql)Executes raw SQL for read operations and returns a single result.

Parameters sql (string) – (required). SQL statement to execute.

select_all(sql, as_dict=False)Executes raw SQL for read operations and returns all results.

Parameters

• sql (string) – (required). SQL statement to execute.

• as_dict (boolean) – (optional). Whether to return result as dict or as a list of tuples.

Table partitioning

Table partitioning is a division of one table into several tables, called partitions, which still represent original table.This is usually done for manageability, performance or availability reasons. If you are unsure whether you needpartitioning or not, then you almost certainly don’t need it.

To use this feature, define the model as usual and create a table for it in the database using the usual tools that yourORM provides. After the table was created, this feature can be installed into the model as the following:

import architect

@architect.install('partition', **options)class Model(object):

pass

where options are:

• type (required). Partition type, e.g. range, list etc

• subtype (required). Partition subtype, e.g. date, integer etc

• constraint (required). What data fits into partition, e.g. day, 5 (every 5 items) etc

• column (required). Column, which value determines which partition record belongs to

• db (optional). Currently used with:

7.2. Features 17

Architect, Release

– Django - only for specifying other database name instead of default. Also if custom routers are used,Architect tries it’s best to figure out the database name from them automatically.

– SQLAlchemy - required if model’s metadata is not bound to any engine, should be set in the form ofDSN, e.g. dialect://user:pass@host/database.

Above options can take different values depending on the database type because different databases support differentpartition types, subtypes etc. To find out which values can be set for the above options choose the database type whichyou currently use from the list below:

PostgreSQL

PostgreSQL’s partitioning implementation in Architect is done purely at the database level. That means that Architectcreates several triggers and functions and inserts them directly into the database, so even if you issue direct insertstatement from database console and not from the ORM, everything will work as expected and record will be insertedinto the correct partition, if partition doesn’t exist, it will be created for you automatically. Also partitions may becreated in any order and not only from lower to higher.

Supported types

range

Range partitioning maps data to partitions based on ranges of values of the partitioning column. It is the most commontype of partitioning and is often used with dates or integers, but it can be used with other types as well. Architectsupports several subtypes for range partitioning type.

date

Date subtype is used to partition table by date ranges. For example, the y2005m01 partition would contain rows withpartitioning column values from 2005-01-01 to 2005-01-31. This subtype has the following constraints:

• day - a new partition will be created every day

• week - a new partition will be created every week

• month - a new partition will be created every month

• year - a new partition will be created every year

import architect

@architect.install('partition', type='range', subtype='date', constraint='month',→˓column='columnname')class Model(object):

pass

integer

New in version 0.4.0.

Integer subtype is used to partition table by integer ranges, for example, one may want to create a new partition forevery 100 rows of data, i.e. rows with id 37 and id 68 will go to partition which collects ids from 1 to 100. Integersubtype also supports negative and zero values, so it can be used not only with ids but with any integer column, signedor unsigned.

18 Chapter 7. Table of contents

Architect, Release

import architect

@architect.install('partition', type='range', subtype='integer', constraint='100',→˓column='columnname')class Model(object):

pass

string_firstchars

New in version 0.4.0.

String firstchars subtype is used to partition table by selecting first N characters from a string, for example, we wantto have partitions based on the first 5 characters of a string, i.e. strings “foobar” and “foobarbaz” will go to partitionnamed “fooba”, but string “foo” will go to its own partition “foo” because it doesn’t have enough characters, string“yadayada” will go to partition “yaday” etc.

import architect

@architect.install('partition', type='range', subtype='string_firstchars', constraint=→˓'5', column='columnname')class Model(object):

pass

string_lastchars

New in version 0.4.0.

This subtype is the same as string_firstchars except that it counts from the end of string and not from thebeginning, i.e. string “foobar” will go to partition “oobar”, string “foobarbaz” to partition “arbaz” etc.

import architect

@architect.install('partition', type='range', subtype='string_lastchars', constraint=→˓'5', column='columnname')class Model(object):

pass

Limitations

• Not all partitioning types are supported. New ones will be added in next releases of Architect.

MySQL

MySQL’s partitioning implementation in Architect is done in a mixed way, half at the python level and half at thedatabase level. Unfortunately MySQL doesn’t support dynamic sql in triggers or functions / procedures that are calledwithin triggers, so the only way to create partitions automatically is to calculate everything at the python level, then tocreate needed sql statements based on calculations and issue that statements into the database.

7.2. Features 19

Architect, Release

Supported types

range

Range partitioning maps data to partitions based on ranges of values of the partitioning column. It is the most commontype of partitioning and is often used with dates or integers, but it can be used with other types as well. Architectsupports several subtypes for range partitioning type.

date

Date subtype is used to partition table by date ranges. For example, the y2005m01 partition would contain rows withpartitioning column values from 2005-01-01 to 2005-01-31. This subtype has the following constraints:

• day - a new partition will be created every day

• week - a new partition will be created every week

• month - a new partition will be created every month

• year - a new partition will be created every year

import architect

@architect.install('partition', type='range', subtype='date', constraint='month',→˓column='columnname')class Model(object):

pass

Limitations

• Not all partitioning types are supported. New ones will be added in next releases of Architect.

• New partitions can be created only from lower to higher, you can overcome this with MySQL’s special commandREORGANIZE PARTITION which you have to issue from the database console. You can read more about it atthe MySQL’s documentation. We plan to remove this limitation in one of the future releases of Architect.

Note: Using this feature with a Django ORM, run the following command before moving to next step, substitutingmysite.settings to the real path of the Django settings module:

$ export DJANGO_SETTINGS_MODULE=mysite.settings

After the feature has been installed into the model, run the following console command:

$ architect partition --module path.to.the.model.module

Note: You can optionally setup an AppConfig in Django to automatically run architect partition command after anymigrations:

from architect.commands import partition

from django.apps import AppConfigfrom django.db import ProgrammingError

20 Chapter 7. Table of contents

Architect, Release

from django.db.models.signals import post_migrate

def create_partitions(sender, **kwargs):"""After running migrations, go through each of the modelsin the app and ensure the partitions have been setup"""paths = {model.__module__ for model in sender.get_models()}for path in paths:

try:partition.run(dict(module=path))

except ProgrammingError:# Possibly because models were just un-migrated or# fields have been changed that effect Architectprint "Unable to apply partitions for module '{}'".format(path)

else:print "Applied partitions for module '{}'".format(path)

class DemoConfig(AppConfig):name = 'apps.demo'

def ready(self):super(DemoConfig, self).ready()# Hook up Architect to the post migrations signalpost_migrate.connect(create_partitions, sender=self)

That’s it. Now, when a new record will be inserted, a value from column, specified in the column option will be usedto determine into what partition the data should be saved. Keep in mind that if new partitioned models are added orany settings are changed in existing partitioned models, the partition command should be rerun, otherwise the databasewon’t know about this changes.

After the installation partition feature can be accessed via Model.architect.partition, though it usuallydoesn’t need to be accessed directly because everything is done automatically. It provides the following methods:

model_metaReturns dictionary of model meta attributes needed for partitioning under common names.

get_partition()Returns partition type object to work with depending on the given partition options.

This object provides the following methods:

prepare()Prepares everything that is needed to initialize partitioning.

create()Creates new partition.

exists()Checks if partition exists.

Custom features

New in version 0.5.0.

Architect supports custom features which can be used to add completely new functionality to a model or to redefinesome built-in feature’s behaviour. A completely new feature can be created and installed as the following:

7.2. Features 21

Architect, Release

from architect.orms.bases import BaseFeature

class CustomFeature(BaseFeature):orm = 'django'name = 'my_feature'decorate = ('bar',)

def foo(self):"""Does something with a model"""pass

@staticmethoddef _decorate_bar(method):

"""Decorates model's bar method"""def wrapper(instance, *args, **kwargs):

# Do something before method invocationmethod(instance, *args, **kwargs)# Do something after method invocation

return wrapper

@architect.install('my_feature')class Model(object):

pass

To be considered a feature class, a class should inherit from a BaseFeature or any other class that itself inheritsfrom a BaseFeature and implement orm and name attributes. orm attribute is used to identify to which ORM thisfeature belongs to and name attribute is used by Architect to identify this feature in an internal feature registry. If afeature class should be considered a base for other feature classes simply don’t set an orm attribute and it will defaultsto None which tells Architect that this is a base feature class.

BaseFeature class provides the following methods and attributes:

orm = Nonewhich orm this feature belongs to

name = Nonename that will be used to access this feature

decorate = ()model methods that should be decorated by feature decorators

dependencies = ()features that this feature depends on

__init__(model_obj, model_cls, **options)

Parameters

• model_obj (object) – (required). Model instance object to work with.

• model_cls (class) – (required). Model class to work with.

• options (dictionary) – (optional). Feature options if any.

Exceptions

Architect tries it’s best to provide human readable errors in all situations. This is a list of all exceptions that Architectcan throw.

22 Chapter 7. Table of contents

Architect, Release

exception architect.exceptions.BaseArchitectError(message, **kw)Base exception class. All Architect exceptions should inherit from it.

exception architect.exceptions.CommandNotProvidedError(**kw)Command not provided.

exception architect.exceptions.CommandError(**kw)Unrecognized command.

exception architect.exceptions.CommandArgumentError(**kw)Unrecognized command argument.

exception architect.exceptions.ImportProblemError(message, **kw)Wrapper for ImportError.

exception architect.exceptions.BaseDatabaseError(message, **kw)Base exception class for all database exceptions.

exception architect.exceptions.ORMError(**kw)Unsupported ORM.

exception architect.exceptions.FeatureInstallError(**kw)Unsupported feature.

exception architect.exceptions.FeatureUninstallError(**kw)Can’t uninstall feature that isn’t installed.

exception architect.exceptions.MethodAutoDecorateError(**kw)Unable to autodecorate method.

exception architect.exceptions.DatabaseError(**kw)Unsupported database.

exception architect.exceptions.OptionNotSetError(**kw)Option not set.

exception architect.exceptions.OptionValueError(**kw)Invalid value provided for option.

exception architect.exceptions.PartitionColumnError(**kw)Unrecognized partition column.

exception architect.exceptions.PartitionTypeError(**kw)Unsupported partition type.

exception architect.exceptions.PartitionConstraintError(**kw)Unsupported partition constraint.

exception architect.exceptions.PartitionRangeSubtypeError(**kw)Unsupported partition range subtype.

exception architect.exceptions.PartitionFunctionError(**kw)Unsupported partition function.

License

Copyright 2017 Max Tepkeev

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance withthe License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

7.4. License 23

Architect, Release

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an“AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See theLicense for the specific language governing permissions and limitations under the License.

Changelog

0.5.6 (2017-05-01)

• Added: Support for testing all databases in one go using DB=all environmental variable

• Fixed: Django: Unable to partition tables in multi database configuration without default database

• Fixed: PostgreSQL: Failed to partition a table if a reserved keyword was used as a column name (turned out thiswasn’t fixed properly in v0.5.4) (thanks to ra2er)

0.5.5 (2016-10-08)

• Fixed: Django: Error handling with 1.10.x branch

0.5.4 (2016-07-04)

• Added: Django: Issue #30 (Support for using database routers to determine correct database for a model in amulti database configuration)

• Fixed: Peewee: Table partitioning was broken in Peewee >= 2.7.4 due to Peewee API changes

• Fixed: PostgreSQL: Failed to partition a table if a reserved keyword was used as a column name

0.5.3 (2015-11-08)

• Added: Tests are now built-in into source package distributed via PyPI

• Fixed: Django: Issue #21 (Unable to partition a model with non-lazy translations)

0.5.2 (2015-07-31)

• Fixed: PostgreSQL: Issue #14 (Error when trying to save a record into a table if partitioned column value hadspecial characters inside, using string_firstchars and string_lastchars partition subtypes)

• Fixed: PostgreSQL: Issue #11 (Error when trying to save a record into a table that was partitioned by a columnthat can be NULL, now if a column, that was used for partitioning, has NULL value it will be inserted into aspecial partition with _null postfix)

• Fixed: SQLObject: Error when trying to partition a model with a field that has a default attribute set to somevalue

0.5.1 (2015-06-08)

• Fixed: Issue #13 (MySQL support was broken in v0.5.0 released to PyPI)

24 Chapter 7. Table of contents

Architect, Release

0.5.0 (2015-05-08)

• Added: Django: Issue #9 (Support for multiple databases)

• Added: Support for custom features, see docs for details

• Changed: dsn partition option renamed to db to cover more use cases

• Changed: DsnParseError exception renamed to OptionValueError to cover more use cases

• Fixed: Django: Error when trying to partition a model with Django <= 1.5 in debug mode

• Fixed: “No module named modulename.py” error when trying to specify model’s module with .py extension atthe end in partition command

0.4.0 (2015-04-18)

• Added: wheel support

• Added: SQLObject ORM support

• Added: PostgreSQL: New integer (thanks to Nikolay Yarovoy), string_firstchars (thanks to DmitryBrytkov) and string_lastchars range partition subtypes, see docs for details

• Changed: range partition option renamed to constraint to better suit new partition subtypes

• Changed: PostgreSQL: Triggers refactoring and speedups, don’t forget to rerun partition command to applynew refactored triggers to the database

• Fixed: architect.uninstall decorator wasn’t able to restore modified model methods under Python 3

0.3.0 (2015-04-05)

• Added: Documentation rewritten from scratch

• Added: Introduced completely new API (almost 80% of Architect code was rewritten from scratch). All func-tionality is now provided by an architect.install decorator which dynamically injects requested feature,e.g. partition, into a model under the architect namespace, e.g. model.architect.partition. Nomore mixins, inheritance and nested classes with settings that pollute model’s namespace. More information isavailable in the docs

• Added: New operation feature which provide an abstraction layer to execute raw SQL statements whichwill work with any supported ORM, see docs

• Fixed: Issue #8 (cannot import name string_literal error with PonyORM and PyMySQL ifpymysql.install_as_MySQLdb() was used)

• Fixed: Issue #7 (SQLite dummy backend was completely broken)

• Fixed: Issue #4 (autocommit cannot be used inside a transaction error with Django if amodel was used inside with transaction.atomic() block)

• Fixed: Issue #2 (partition command was unable to find module with models to partition)

• Fixed: Issue #1 (relation already exists error when trying to insert data into non-existent partitionsimultaneously from several queries) (thanks to Daniel Kontsek)

0.2.0 (2014-07-19)

• Added: MySQL: Range partitioning support

7.5. Changelog 25

Architect, Release

0.1.0 (2014-07-13)

• Initial release

26 Chapter 7. Table of contents

Python Module Index

aarchitect.exceptions, 22

27

Architect, Release

28 Python Module Index

Index

Symbols__init__(), 22

Aarchitect.exceptions (module), 22

BBaseArchitectError, 22BaseDatabaseError, 23

CCommandArgumentError, 23CommandError, 23CommandNotProvidedError, 23create(), 21

DDatabaseError, 23decorate, 22dependencies, 22

Eexecute(), 17exists(), 21

FFeatureInstallError, 23FeatureUninstallError, 23

Gget_partition(), 21

IImportProblemError, 23

MMethodAutoDecorateError, 23model_meta, 21

Nname, 22

OOptionNotSetError, 23OptionValueError, 23orm, 22ORMError, 23

PPartitionColumnError, 23PartitionConstraintError, 23PartitionFunctionError, 23PartitionRangeSubtypeError, 23PartitionTypeError, 23prepare(), 21

Sselect_all(), 17select_one(), 17

29