Balázs Zsoldos

Modularized Persistence

History of NoSQL by Mark Madsen, Picture published by Edd Dumbill

Blueprint+

JPA+

JSF

DeclarativeServices

+ModularizedPersistence

+JSF

ECM+

ModularizedPersistence

+JSF

ECM+

ModularizedPersistence

+Modularized

Web

DeclarativeServices

+JPA

+JSF

Stefan Seifert – Apache Sling & Friends Tech Meetup, Berlin 2012

JDBC Driver(DataSourceFactory)

JDBC Driver(DataSourceFactory)

DSF Component(XADataSource)

JDBC Driver(DataSourceFactory)

DSF Component(XADataSource)

DBCP Component(DataSource)

JDBC Driver(DataSourceFactory)

DSF Component(XADataSource)

DBCP Component(DataSource)

???

JDBC Driver(DataSourceFactory)

DSF Component(XADataSource)

DBCP Component(DataSource)

Liquibase DataSource Component(DataSource)

<databaseChangeLog objectQuotingStrategy="QUOTE_ALL_OBJECTS" logicalFilePath="org.everit.osgi.resource.ri.schema" xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.0.xsd">

<changeSet id="1.0.0" author="everit">

<createTable tableName="res_resource"> <column name="resource_id" type="bigint" autoIncrement="true"> <constraints primaryKeyName="pk_res_resource" primaryKey="true" nullable="false" /> </column> </createTable> </changeSet>

</databaseChangeLog>

Provide-Capability: liquibase.schema; name="org.everit.osgi.resource.ri"; resource="/META-INF/liquibase/resource.ri.liquibase.xml";

Provide-Capability: liquibase.schema;name="org.everit.osgi.resource.ri"; resource="/META-INF/liquibase/resource.ri.liquibase.xml"

JDBC Driver(DataSourceFactory)

DSF Component(XADataSource)

DBCP Component(DataSource)

Liquibase DataSource Component(DataSource)

Business Component Business Component Business ComponentQuerydsl Querydsl Querydsl

Examples from http://querydsl.com

Write Liquibase changelog

Write LQMG file

Add LQMG to the Capability

Add inclusions

Generate Querydsl Metadata

<lqmg xmlns="http://everit.org/lqmg" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" defaultPackage="org.everit.osgi.resource.ri.schema.qdsl"> <namingRules> <classNameRule> <entity>res_resource</entity> <class>Resource</class> <propertyMappings> <primaryKey> <name>pk_res_resource</name> <property>resourcePk</property> </primaryKey> </propertyMappings> </classNameRule> </namingRules></lqmg>

Write Liquibase changelog

Write LQMG file

Add LQMG to the Capability

Add inclusions

Generate Querydsl Metadata

Provide-Capability: liquibase.schema; name="org.everit.osgi.resource.ri"; resource="/META-INF/liquibase/resource.ri.liquibase.xml";

Provide-Capability: liquibase.schema;name="org.everit.osgi.resource.ri"; resource="/META-INF/liquibase/resource.ri.liquibase.xml"

Provide-Capability: liquibase.schema; name="org.everit.osgi.resource.ri"; resource="/META-INF/liquibase/resource.ri.liquibase.xml"; lqmg.config.resource="/META-INF/liquibase/resource.ri.lqmg.xml"

Provide-Capability: liquibase.schema;name="org.everit.osgi.resource.ri"; resource="/META-INF/liquibase/resource.ri.liquibase.xml";lqmg.config.re source="/META-INF/liquibase/resource.ri.lqmg.xml"

Write Liquibase changelog

Write LQMG file

Add LQMG to the Capability

Add inclusions

Generate Querydsl Metadata

<databaseChangeLog objectQuotingStrategy="QUOTE_ALL_OBJECTS" logicalFilePath="org.everit.osgi.authorization.ri.schema" xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.0.xsd http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd">

<include file="eosgi:org.everit.osgi.resource.ri" /> <include file="eosgi:org.everit.osgi.props.ri" />

<changeSet id="1.0.0" author="everit">

<createTable tableName="authr_permission"> <column name="authorized_resource_id" type="bigint"> <constraints nullable="false" foreignKeyName="fk_res_r_authr_p_a" referencedTableName="res_resource" referencedColumnNames="resource_id" /> </column> <column name="target_resource_id" type="bigint"> <constraints nullable="false" foreignKeyName="fk_res_r_authr_p_t" referencedTableName="res_resource" referencedColumnNames="resource_id" /> </column> <column name="action_" type="varchar(255)"> <constraints nullable="false" /> </column> </createTable> <addPrimaryKey constraintName="pk_authr_perm" tableName="authr_permission" columnNames="authorized_resource_id,target_resource_id,action_" />

<createTable tableName="authr_permission_inheritance"> <column name="parent_resource_id" type="bigint"> <constraints nullable="false" foreignKeyName="fk_res_r_authr_pi_p" referencedTableName="res_resource" referencedColumnNames="resource_id" /> </column> <column name="child_resource_id" type="bigint"> <constraints nullable="false" foreignKeyName="fk_res_r_authr_pi_c" referencedTableName="res_resource" referencedColumnNames="resource_id" /> </column> </createTable> <addPrimaryKey constraintName="pk_authr_perm_inheritance" tableName="authr_permission_inheritance" columnNames="parent_resource_id,child_resource_id" /> </changeSet>

</databaseChangeLog>

<include file="eosgi:org.everit.osgi.resource.ri" /> <include file="eosgi:org.everit.osgi.props.ri" />

Resource

Authorization

Provide-Capability: liquibase.schema; name="org.everit.osgi.resource.ri"; ...

Require-Capability: liquibase.schema; filter:=(name=org.everit.osgi.resource.ri)

<databaseChangeLog ...>

<include file="eosgi:org.everit.osgi.resource.ri" />

...

</databaseChangeLog>

Write Liquibase changelog

Write LQMG file

Add LQMG to the Capability

Add inclusions

Generate Querydsl Metadata

Start embeddedOSGi container

Deploy modules

Start embeddedH2 database

Initialize databaseschema

Generate QuerydslMetadata

LQMG

Evil magic here!!!

If Capability is not found after deployments, unsatisfied bundles are enhanced in the way that unsatisfied requirements are marked to be optional.

Use-cases???

Monoholitic

Authorization

Authentication

User

Documents

Articles

SiteMap

Blog

Portal(v5.1.2)

Modularized

Portal

Authorization (v1.1.0)

Authentication (v2.0.0)

User (v1.3.2)

Documents (v1.0.1)

SiteMap (v1.0.3)

Articles (v3.0.14)

Blog (v2.11.0)

User

User Address

User Address Country

User Address Country

Company

REAL PROJECT!!!

Authorization

Permission

● authorized_resource_id● action● target_resource_id

Resource

● resource_id

Permission Inheritance

● parent_resource_id● child_resource_id

SELECT ... FROM document d JOIN document.attachment a WHERE … LIMIT 10 OFFSET 1000;

EXISTS(SELECT 1 FROM permission p WHERE p.authorized_resource_id IN (?, …, ?) AND p.target_resource_id = d.resource_id AND action = ?)

SELECT ... FROM document d JOIN document.attachment a WHERE EXISTS(SELECT 1 FROM permission p WHERE p.authorized_resource_id IN (?, …, ?) AND p.target_resource_id = d.resource_id AND action = ?) AND … LIMIT 10 OFFSET 1000;

@Override public BooleanExpression authorizationPredicate(final long authorizedResourceId, final Expression<Long> targetResourceId, final String... actions) { if (authorizedResourceId == systemResourceId) { return BooleanTemplate.TRUE; }

Objects.requireNonNull(targetResourceId, "Parameter targetResourceId must not be null"); validateActionsParameter(actions);

long[] authorizationScope = getAuthorizationScope(authorizedResourceId);

SQLSubQuery subQuery = new SQLSubQuery();

QPermission permission = QPermission.permission;

BooleanExpression authorizedResourceIdPredicate; if (authorizationScope.length == 1) { authorizedResourceIdPredicate = permission.authorizedResourceId.eq(authorizationScope[0]); } else { // More than one as the scope contains at least one value (other branch) Long[] authorizationScopeLongArray = new Long[authorizationScope.length]; for (int i = 0, n = authorizationScope.length; i < n; i++) { if (authorizationScope[i] == systemResourceId) { return BooleanTemplate.TRUE; } authorizationScopeLongArray[i] = authorizationScope[i]; }

authorizedResourceIdPredicate = permission.authorizedResourceId.in(authorizationScopeLongArray); }

BooleanExpression actionPredicate = null;

if (actions.length == 1) { actionPredicate = permission.action.eq(actions[0]); } else { actionPredicate = permission.action.in(actions); }

return subQuery.from(permission) .where(permission.targetResourceId.eq(targetResourceId).and( actionPredicate.and(authorizedResourceIdPredicate))) .exists(); }

SQLQuery query = new SQLQuery(connection, configuration);QDocument document = new QDocument("d");

BooleanExpression permissionCheckPredicate = authorizationQdslUtil .authorizationPredicate(userId, document.resourceId, actions);

List<Long> list = query.from(document).where(permissionCheckPredicate) .list(document.documentId);

JDBC Driver(DataSourceFactory)

DSF Component(XADataSource)

DBCP Component(DataSource)

Liquibase DataSource Component(DataSource)

Business Component Business Component Business Component

QuerydslSupport

QuerydslConfiguration

QuerydslSQLTemplates

Querydsl Querydsl Querydsl

public List<Long> listDocumentIds() {

return qdsl.execute((connection, configuration) -> {

SQLQuery query = new SQLQuery(connection, configuration); QDocument document = new QDocument("d");

BooleanExpression permissionCheckPredicate = authorizationQdslUtil .authorizationPredicate(a1, document.resourceId, actions);

List<Long> list = query.from(document).where(permissionCheckPredicate) .list(document.documentId); }}

@Reference(name = "querydslSupport", bind = "setQdsl")private QuerydslSupport qdsl;

public void setQdsl(QuerydslSupport qdsl) { this.qdsl = qdsl;}

JDBC Driver(DataSourceFactory)

DSF Component(XADataSource)

DBCP Component(DataSource)

Liquibase DataSource Component(DataSource)

Business Component Business Component Business Component

QuerydslSupport

QuerydslConfiguration

QuerydslSQLTemplates

Querydsl Querydsl Querydsl

Transaction Helper

public long saveUser(String firstName, String lastName) { Objects.requireNonNull(firstName); Objects.requireNonNull(lastName);

return transactionHelper.required(() -> { // Logic that should be implemented return 0l; }); }

Caching

● Available Cache modules: cache-api, cache-infinispan, cache-noop

● Caching should be done in the persistent module by the programmer, who knows the business logic

● Caching and table updates should be done within the same component

● In case bulk update is done in another module, cache must be cleared

OpenSource modules

● Resource● Authorization● Authentication● Property Manager● Blobstore (Before review)● Audit (Before review)

Roadmap

HTML Output

LQMG / Maven

H2

Liquibase schema(Bundle-Capability)

HTML

LQMG / Maven

Database

Liquibase schema(Bundle-Capability)

Generate Schema

Manual Schema update

Webconsole plugin

DataSource(OSGi service)

Liquibase schema(Bundle-Capability)

Generate SchemaCustom User / Password

Manual Schema generation

Dump SQL file

Webconsole plugin

DataSource(OSGi service)

Liquibase schema(Bundle-Capability)

SQL

LQMG extension in changelog.xml

<databaseChangeLog objectQuotingStrategy="QUOTE_ALL_OBJECTS" logicalFilePath="org.everit.osgi.resource.ri.schema" xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:lqmg="http://everit.org/xml/ns/lqmg" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.0.xsd">

<changeSet id="1.0.0" author="everit">

<createTable tableName="res_resource" lqmg:class="QResource"> <column name="resource_id" type="bigint" autoIncrement="true" lqmg:property="resourceId"> <constraints primaryKeyName="pk_res_resource" primaryKey="true" nullable="false" /> </column> </createTable> </changeSet>

</databaseChangeLog>

Full Text Search

● Analize the different solutions– H2 → Lucene

– MySQL → Sphinx

– PostgreSQL → TSearch2

– SQL Server →???– Oracle → Oracle Text

● Create a common API for Querydsl based query extension

● Create a module for each database engine

Querydsl: http://www.querydsl.com/

Liquibase: http://www.liquibase.org/

Liquibase OSGi Bundle: https://github.com/everit-org/osgi-liquibase-bundle

LQMG: https://github.com/everit-org/osgi-lqmg

LQMG Maven Plugin: http://www.everit.org/lqmg-maven-plugin/

Liquibase DataSource: https://github.com/everit-org/osgi-liquibase-datasource

Querydsl SQLTemplates Component: https://github.com/everit-org/osgi-querydsl-templates

Querydsl Configuration Component: https://github.com/everit-org/osgi-querydsl-configuration

Querydsl Support Component: https://github.com/everit-org/osgi-querydsl-support

Cookbook chapter: http://cookbook.everit.org/persistence/index.html

Download from Maven central

Resource: https://github.com/everit-org/resource-ri

Authorization: https://github.com/everit-org/authorization-ri

Authentication: https://github.com/everit-org/authentication-simple

Property Manager: https://github.com/everit-org/property-manager-ri

Twitter: @EveritOrg