LDAP107: Federation w/RDBM Systems

If your find these documents useful and feel the need to express that opinion in a tangible way, consider selecting an item from my Amazon Wish List.

awilliam@whitemice.org

Copyright © 2004 Adam Tauno Williams (awilliam@whitemice.org)

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation with no Invariant Sections, no Front­Cover Texts, and no Back­Cover Texts. You may obtain a copy of the GNU Free Documentation License from the Free Software Foundation by visiting their Web site or by writing to: Free Software Foundation, Inc., 59 Temple Place ­ Suite 330, Boston, MA 02111­1307, USA.

If you find this document useful or further it's distribution we would appreciate you letting us know.

What is federation●Federation is the construction of one directory from multiple datasources.

● Other directories● Relation databases● etc...

● OpenLDAP provides various features to facilitate the construction of a federated directory:● back-sql

● Construct hierarchical views of ODBC connected relational databases

● back-meta● Proxy external directories into an existing tree

with caching and content rewriting rules.● Sort of the LDAP equivalent of NAT

● back-shell / back-perl● Anything you want.

● back-perl is a bit moldy.

Partitioning

Partitions

SubordinatePartitions

SuperiorPartitions

●Remember “partitioning” from LDAP101?● The power to create federated

directories stems from the ability to partition the Dit.● The concept of subordinate

partitions can be used to 'glue' data sources together.● The data needs to be munged into

LDAP at the partition level.

back-sql

●The back-sql OpenLDAP backend connects to an ODBC data-source and uses a set of meta-data tables to map the structure of the database into a Dit.

● Warning: This is not a straight forward task.● An SQL database is a group of related sets (tables),

not a hierarchical structure.● And many (most) are poorly designed with only loose

unenforced relationships.● An SQL database has no concept similar to

objectclasses or object inheritance.● Actually some advanced systems like DB2, PostgreSQL and

Informix do; but almost no applications make use of such features.

● An SQL database doesn't have functionality equivalent to subordinate or superior information or referrals.

Our ExampleTraditional Dit

●We have a existing LDAP infrastructure (servers and a Dit)

● This Dit already contains data useful to a variety of users.

● We want additional data sources to be subordinate partitions in our existing Dit.

dc=rss,dc=nw

ou=Federation

ou=OpenGroupware

ou=LegcacyCRM

ou=Customers

ou=SAM

The 'federated' part of the Dit.

Our ExampleOpenGroupware

●The OpenGroupware.org groupware server uses a PostgreSQL backend as a data-store.

● ODBC drives are available for PostgreSQL

● We want access to this data from LDAP clients

Examine your data●Determine what data needs to be presented via the DSA

● And how to structure this data in a hierarchical fashion.● Views can be terribly useful here.

●ou=Federation● ou=Enterprises

● opengroupwareid=● ou=Contacts

● opengroupwareid=● ou=Documents

● documentid=● ou=Accounts

● userid=● ou=Teams

● cn=● ou=AcessControl

● cn=

attributetype ( 1.3.6.1.4.1.6921.3.1.1 NAME 'opengroupwareid' DESC 'Unique OpenGroupware Id' EQUALITY integerMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.27 )

attributetype ( 1.3.6.1.4.1.6921.3.1.3 NAME 'opengroupwareversion' DESC 'OpenGroupware Revision Number' EQUALITY integerMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.27 )

objectclass ( 1.3.6.1.4.1.6921.3.2.1 NAME 'opengroupwareentity' DESC 'OpenGroupware Entity' SUP top AUXILIARY MUST ( opengroupwareid ) MAY ( opengroupwareversion ) )

attributetype ( 1.3.6.1.4.1.6921.3.1.1 NAME 'opengroupwareid' DESC 'Unique OpenGroupware Id' EQUALITY integerMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.27 )

attributetype ( 1.3.6.1.4.1.6921.3.1.3 NAME 'opengroupwareversion' DESC 'OpenGroupware Revision Number' EQUALITY integerMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.27 )

objectclass ( 1.3.6.1.4.1.6921.3.2.1 NAME 'opengroupwareentity' DESC 'OpenGroupware Entity' SUP top AUXILIARY MUST ( opengroupwareid ) MAY ( opengroupwareversion ) )

A Little Extra Schema

●Federating your data may require creation of additionally schema.

● You must acquire a valid OID

● Attribute and class names should be distinctive.● If you are federating a

common application, then SHARE your schema.

ODBC●You MUST have a working and tested ODBC connection to your database before trying to use back-sql

● Test, test, and test your ODBC connection, then test again.

[PostgreSQL]Description=ODBC for PostgreSQLDriver=/usr/lib/unixODBC/libodbcpsql.so.2Setup=/usr/lib/unixODBC/libodbcpsqlS.so.1FileUsage=1

[ogo]Description=OpenGroupwareDriver=PostgreSQLTrace=YesTraceFile=/tmp/ODBC.logDatabase=ogoServername=throwPort=5432Protocol=7.4ReadOnly=NoRowVersioning=YesShowSystemTables=NoShowOidColumn=NoFakeOidIndex=NoConnSettings=

moa:/etc/openldap # isql ogo ***** *****SQL> select COUNT(*) from person;+------------------+| 27305 |+------------------+SQLRowCount returns -11 rows fetchedSQL> quitmoa:/etc/openldap #

●DSN configuration will vary depending upon RDBMS.

back-sql Mapping●back-sql uses a set of tables in the relational database itself to store information on what table and field values correspond to a given LDAP attribute, and what database keys correspond to a given LDAP object.

ldap_oc_mappings

The rest of the database.

keytbl,keycol

ldap_attr_mappings

sel_expr, fr om_tbls, joi n_w

here

idoc_map_id

ldap_entries

keyval

id

oc_map_id

ldap_entry_objectclasses

id entry_id

ldap_referralsentry_id

id

●The keys into the database must be integers (which is standard practice).

● The mapping concept relies heavily upon table joins, so indexing the key fields is critical for performance.

ldap_oc_mappings

CREATE SEQUENCE ldap_oc_mappings_id_seq;CREATE TABLE ldap_oc_mappings (id int4 NOT NULL PRIMARY KEY DEFAULT nextval('ldap_oc_mappings_id_seq'),name varchar(64) NOT NULL,keytbl varchar(64) NOT NULL,keycol varchar(64) NOT NULL,create_proc varchar(255),delete_proc varchar(255),expect_return int NOT NULL);

objectclass

table name

integer key

Stored procedure to remove theobject from the RDBMS tablesbased upon the integer key.

Always 0?

Stored procedure to create objectsof the given objectclass in the RDBMS.

ldap_oc_mappingsExample

insert into ldap_oc_mappingsVALUES (1, 'person', 'ldap_person', 'company_id', NULL, NULL, 0);insert into ldap_oc_mappingsVALUES (2, 'organization', 'ldap_enterprise', 'company_id', NULL, NULL, 0);insert into ldap_oc_mappingsVALUES (3, 'groupOfNames', 'ldap_group', 'company_id', NULL, NULL, 0);insert into ldap_oc_mappingsVALUES (4, 'organizationalUnit', 'ldap_static_entries', 'id', NULL, NULL, 0);insert into ldap_oc_mappingsVALUES (6, 'account', 'ldap_account', 'company_id', NULL, NULL, 0);insert into ldap_oc_mappingsVALUES (7, 'document', 'ldap_document', 'document_id', NULL, NULL, 0);

objectclass table primary key

ldap_attr_mappingsCREATE SEQUENCE ldap_attr_mappings_id_seq;CREATE TABLE ldap_attr_mappings(id int4 NOT NULL PRIMARY KEY

default nextval('ldap_attr_mappings_id_seq'),oc_map_id int4 NOT NULL,name varchar(255) NOT NULL,sel_expr varchar(255) NOT NULL,sel_expr_u varchar(255),from_tbls varchar(255) NOT NULL,join_where varchar(255),

Corresponding objectclass id from ldap_oc_mappings

attribute

Expression used to selectthe field (table.fieldname)

Comma delimited list oftables involved in the query

?Expression used to jointables if multiple tables are involved in the query.(table1.fieldname1 = table2.fieldname2)*May be NULL.

ldap_attr_mappings

add_proc varchar(255),delete_proc varchar(255),param_order int NOT NULL,expect_return int NOT NULL,FOREIGN KEY (oc_map_id) REFERENCES ldap_oc_mappings(id));

Stored procedure to add a value tothis attribute given an object id and a value

Stored procedure to delete the value of this attribute given an object id and a value

Always 0?

Always 3?

ldap_attr_mappingsExample

insert into ldap_attr_mappings (id,oc_map_id,name,sel_expr,from_tbls,join_where,add_proc,delete_proc, param_order,expect_return)values (100,1,'cn','cn','ldap_person', NULL, NULL, NULL, 3, 0);insert into ldap_attr_mappings (id,oc_map_id,name,sel_expr,from_tbls,join_where,add_proc,delete_proc, param_order,expect_return)values (101,1,'givenname','firstname','ldap_person', NULL, NULL, NULL, 3, 0);insert into ldap_attr_mappings (id,oc_map_id,name,sel_expr,from_tbls,join_where,add_proc,delete_proc, param_order,expect_return)values (102,1,'sn','lastname','ldap_person', NULL, NULL, NULL, 3, 0);insert into ldap_attr_mappings (id,oc_map_id,name,sel_expr,from_tbls,join_where,add_proc,delete_proc, param_order,expect_return)values (103,1,'initials','initials','ldap_person', NULL, NULL, NULL, 3, 0);insert into ldap_attr_mappings (id,oc_map_id,name,sel_expr,from_tbls,join_where,add_proc,delete_proc, param_order,expect_return)values (104,1,'birthdate','birthday','ldap_person', NULL, NULL, NULL, 3, 0);

attribute field table

ldap_entries

CREATE SEQUENCE ldap_entries_id_seq;CREATE TABLE ldap_entries(id int4 NOT NULL PRIMARY KEY

DEFAULT nextval('ldap_entries_id_seq'),dn varchar(255) NOT NULL UNIQUE,-- dn_ru varchar(255),oc_map_id int4 NOT NULL,parent int NOT NULL,keyval int NOT NULL,UNIQUE (oc_map_id,keyval),FOREIGN KEY (oc_map_id) REFERENCES ldap_oc_mappings (id));

The virtual dn

The objectclass id from

ldap_oc_mappings

The object id of the parent object, used to create the heirarchical structure required by the LDAP data-model. The root object within the database has a parent of 0.

The integer key used to map this virtual dnto the actual content of the relational database.

●The purpose of ldap_entries is to map a dn to a database key, the last step in transforming the LDAP data-model to the SQL relational model.

Making The DN●We must create unique DNs for every object we manifest from the database.

● DNs must reflect the hierarchy of the structure we have created.

● DNs must be valid; that is the RDN must be present within the object, etc..

●There are two method to accomplish this end:● Trigger updates to ldap_entries

● ldap_entries is a table and is update by triggers on the tables referenced in ldap_oc_mappings.

● Union View● ldap_entries is a view that consists of a union of

multiple selects on the referenced tables.

ldap_entriesExample

CREATE VIEW ldap_entries (id, dn, oc_map_id, parent, keyval) ASSELECT id, dn, 4 as oc_map_id, pid as parent, id FROM ldap_static_entriesUNION SELECT company_id, 'opengroupwareid=' || company_id || ',ou=Contacts,ou=Federation', 1 as oc_map_id, 10 as parent, company_idfrom ldap_personUNION SELECT company_id, 'opengroupwareid=' || company_id || ',ou=Enterprises,ou=Federation', 2 as oc_map_id, 20 as parent,company_idfrom ldap_enterpriseUNION SELECT company_id, 'userid=' || ldap_account.userid || ',ou=Accounts,ou=Federation', 6 as oc_map_id, 70 as parent, company_idfrom ldap_account....

ldap_static_entries

CREATE TABLE ldap_static_entries ( id int not null, dn varchar(255), oc int not null, pid int not null, ou varchar(255), dc varchar(255));

insert into ldap_static_entries (id, dn, oc, pid, ou, dc) values (1, 'ou=Federation', 4, 0, 'OpenGroupware', 'mormail.com');insert into ldap_static_entries (id, dn, oc, pid, ou, dc) values (10, 'ou=Contacts,ou=Federation', 4, 1, 'Contacts','mormail.com');insert into ldap_static_entries (id, dn, oc, pid, ou, dc) values (20, 'ou=Enterprises,ou=Federation', 4, 1, 'Enterprises','mormail.com');insert into ldap_static_entries (id, dn, oc, pid, ou, dc) values (30, 'ou=AccessControl,ou=Federation', 4, 1, 'AccessControl','mormail.com');insert into ldap_static_entries (id, dn, oc, pid, ou, dc) values (70, 'ou=Accounts,ou=Federation', 4, 1, 'Accounts', 'mormail.com');insert into ldap_static_entries (id, dn, oc, pid, ou, dc) values (80, 'ou=Teams,ou=Federation', 4, 1, 'Teams', 'mormail.com');insert into ldap_static_entries (id, dn, oc, pid, ou, dc) values (90, 'ou=Documents,ou=Federation', 4, 1, 'Documents', 'mormail.com');

●ldap_static_entries supplements the data from the application to provide a hierarchy.

ldap_entry_objclasses

CREATE TABLE ldap_entry_objclasses(entry_id int4 NOT NULL,oc_name varchar(64),FOREIGN KEY (entry_id) REFERENCES ldap_entries(id));

●ldap_entry_objclasses is used to assign objectclass attributes to a virtual object.

The id of the virtual object as defined in ldap_entries

The objectclass name

●The oc_map_id of ldap_entries only permits an object to have a single objectclass.

● The ldap_entry_objectclasses allow an object to have multiple objectclass values.

ldap_entry_objclassesExample

CREATE TABLE ldap_objectclass_list ( oc_map_id int, objectclass varchar(255));--Contactinsert into ldap_objectclass_list values(1, 'orgPerson');insert into ldap_objectclass_list values(1, 'inetOrgPerson');insert into ldap_objectclass_list values(1, 'officePerson');insert into ldap_objectclass_list values(1, 'evolutionPerson');insert into ldap_objectclass_list values(1, 'top');insert into ldap_objectclass_list values(1, 'opengroupwareentity');...CREATE VIEW ldap_entry_objclasses ( entry_id, oc_name ) AS SELECT id, objectclass FROM ldap_entries, ldap_objectclass_list WHERE ldap_entries.oc_map_id = ldap_objectclass_list.oc_map_id;

ldap_referrals

CREATE TABLE ldap_referrals(entry_id int4 NOT NULL,url text NOT NULL,FOREIGN KEY (entry_id) REFERENCES ldap_entries(id));

The id of the object, as defined in ldap_entries(id).

Where to refer the client to, the URL.

●ldap_referrals allows you to declare objects mapped from the relational database as referral objects to other LDAP servers or sections of the Dit.

back-sql slapd.confdatabase sqlsuffix "ou=Federation"rootdn "cn=root,ou=Federation"rootpw secretdbname miecrdbuser ************dbpasswd ************upper_func "upper"strcast_func "text"concat_pattern"?||?"schemacheck onlastmod offhas_ldapinfo_dn_ru no

●For debugging back-sql issues run slapd with a debug level of 5 (“-d 5”)

Take a look$ ldapsearch -x -b"ou=Contacts,ou=Federation" -h localhost givenname=Brucedn: opengroupwareid=98990,ou=Contacts,ou=FederationobjectClass: personobjectClass: evolutionPersonobjectClass: inetOrgPersonobjectClass: officePersonobjectClass: opengroupwareentityobjectClass: orgPersonobjectClass: topo: Armstrong Internationalo: Kalamazoo Linux Users Groupcn: Bruce Smithsn: Smithmail: bruce@armintl.cominitials: BSgivenName: Bruceopengroupwareid: 98990

Proxy Cache●Federated directories present three problems.

● Solutions like back-sql are labor intensive, performing lots of data mangling.

● Geographically distant DSAs present high latency and potentially intermittent availability

● These trees of external information need to be glued into the existing directory tree in order to hide their 'externalness' from users.

●These are addresses with OpenLDAP's meta backend and the proxy cache overlay.

http://www.openldap.org/pub/kapurva/proxycaching.pdfhttp://www.bind9.net/manual/openldap/2.2/proxycache.html

Proxy CacheExample

database ldap# subordinatesuffix "ou=OpenGroupware,ou=MetaDirectory,o=Morrison Industries,c=US"uri ldap://localhost/ou=Federationsuffixmassage "ou=OpenGroupware,ou=MetaDirectory,o=Morrison...overlay proxycache proxycache ldbm 100 3 100 3600 proxyAttrset 0 mail postaladdress telephonenumber cn givenname ... proxyAttrset 1 o telephonenumber opengroupwareid proxyTemplate (sn=) 0 3600 proxyTemplate (opengroupwareid=) 0 3600 proxyTemplate (&(sn=)(givenName=)) 0 3600 proxyTemplate (cn=)) 0 3600 proxyTemplate (o=) 1 3600 proxyTemplate (opengroupwareid=) 1 3600

cachesize 20 directory /opt/dsa/cache index objectClass,opengroupwareid eq index cn,sn,uid,mail pres,eq,sub

●The details of proxy cache and back-meta will be covered in LDAP109

●You can add ACLs here.

Take A Look$ ldapsearch -x -b"ou=OpenGroupware,ou=MetaDirectory,o=Morrison Industries,c=US" -h localhost sn=Smithdn: opengroupwareid=98990, ou=Contacts, ou=OpenGroupware, ou=MetaDirectory, o=Morrison Industries, c=USo: Armstrong Internationalo: Kalamazoo Linux Users Groupcn: Bruce Smithsn: Smithmail: bruce@armintl.cominitials: BSgivenName: Bruceopengroupwareid: 98990

●Data is now available as a transparent portion of the directory information tree.

Global Federation

(2) DNS Query

For SRV Record.

(3) DNS

Response

(1) LDAP Query

(4) LDAPReferral

(5) LDAP Query

(6) LDAP Response

●OpenLDAP provides a facility that uses SRV records in DNS to provide a globally federated directory.

● For sites using RFC2247 style naming.

SRV Configuration●slapd.conf

● Place -● database dnssrv

suffix “”● - at the end of the file, server will automatically

generate referrals for non-local queries.● If no SRV record exists for the domain of the search

based then a noSuchObject error is returned.● A search with a base of dc=example,dc=com

● results in a SRV search for “_ldap._tcp.example.com”● if a server is found it is sent as a referral to the client

●Setting “referral ldap://root.openldap.org:389” on the servers at the top of your Dit will cause them to use the SRV backend on OpenLDAP's servers to create out-of-tree referrals.

Big Picture

SyncRepl

Prox

yProxy

DNS SRV

Master DSA

Business System(Informix RDBMS)

Groupware(PostgreSQL)

Collaberation DSA

Client