Custom Indexing ConnectorsHow to integrate external systems into SharePoint Search

Heinrich UlbrichtGermany, MEZApril 16th /17th, 2014

SP24 Search SessionsSession

Time (GMT)

Time (EST) Title Level Speaker Twitter Handle

#SP24S090 5:00 1:00 Using the Content Enrichment Web Service with SharePoint Server 2013 Search

Technical - Advanced

Sezai Komur - NEC Australia @sezai

#SP24S021 6:00 2:00 2013 Search, Display Templates, Query Rules, Result Types

Technical - Beginner

Omer Zubair - CSC @Omer_Zubair

#SP24S047 8:00 4:00 Real-life experience building search-driven applications for product-centric sites

Technical - Advanced

Marius Constantinescu - blue-infinity SA @c_marius

#SP24S088 10:00 6:00Custom Indexing Connectors - How to integrate external systems into your SharePoint Enterprise Search

Technical - Advanced

Heinrich Ulbricht - Communardo Software GmbH

@h_ulbricht

#SP24S083 11:00 7:00 Search First Migration with SharePoint 2013 – Benefits / Disadvantages

Technical - Intermediate

Max Melcher - Alegri International Service GmbH

@maxmelcher

#SP24S034 14:00 10:00 The Search Immaturity Cycle, and How to Create a Search Strategy

Business - Intermediate

Jeff Fried - BA Insight @jefffried

#SP24S009 16:00 12:00 10 Things to Know about Search in SP2013 and Office 365

Business - Beginner

Agnes Molnar - Search Explained @molnaragnes

#SP24S019 18:00 14:00 Search Topology and Optimization Technical - Advanced

Mike Maadarani - MCM Consulting @mikemaadarani

#SP24S050 22:00 18:00 Search Driven Application Development in SharePoint 2013

Technical - Intermediate

Matt Youngstrom - Magenic @spguru

#SharePoint#Apps#Search#WindowsPhone

Heinrich UlbrichtSoftware Developer for SharePoint

Communardo Software GmbHDresden, Germany

Contact@h_ulbrichtheinrich.ulbricht@communardo.dehttp://sharepointbrainpump.blogspot.com/

Session ObjectivesPresent extensibility of SharePoint SearchIndexing Connectors: good understanding of architecture

Key TakeawaysSharePoint can be the Enterprise Search for all your contentIf you’ve never developed an indexing connector – get help

Session Objectives and Takeaways

Vision “One Enterprise Search” with SharePointExtensibility of SharePoint SearchSearch Connector FrameworkConnector for SharePoint Search for ConfluenceLessons Learned

Agenda

Challenge: No Integration of Systems

Challenge: Non-integrated Systems

SAP

SQLWCF

Siebel

Dynamics

Confluence?

No Integration – Consequences User is confused

Where to find information? Different search centers with different usability and power (filters, …) Different search result formatting Different search result quality (ranking, …)

Efficiency drops!

Solution: Search Integration

1. No integration (no solution…)2. Links

3. Federated Search

4. Integrated Search

Level of Search Integration

Links Yammer-”Integration” came with SP2013 SP1 -> Link to Yammer Search Center

Federated Search Search results next to each other

Result blog for Bing search results

SharePoint results

Integrated Search One search result view; E.g.: SharePoint + Confluence:

Common relevancy, Sorting, Refiners, Managed Properties

“One Enterprise Search”@jeffteper: “Best Enterprise Search in the World” #SPC14

Transparent System LandscapeSAP

SQL WCF

SiebelDynamics

ConfluenceWork like

a network

Vision• One Enterprise

Search• Go to one

place for information

• Search spans all systems

Communardo

The Road to an Indexing ConnectorTechnical Building Blocks

Building Blocks

1) BCS3) Search Connector Framework

2) SharePoint

Search4)

Confluence

Indexing Connector

Business Connectivity ServicesConnect with external systems

Connect to External Systems with SharePoint

Access external data with Business Connectivity Services (BCS) Use stereotypical operations to access external system (“Read

Entity Instance”, “Create Entity Instance”) Structure / Projection described in BDC Model (XML File) Access operations executed in OOB or Custom Connectors

SQL, WCF, Oracle, SAP, Siebel, Dynamics, Confluence

BCS

BCS – Integrate External Systems with Connectors

Source: Professional SharePoint® 2013 Development (ISBN: 978-1-118-49582-7)

Extend Search with BCS Declarative

BCS can access external content using External Content Types (ECT) Good for reaching out to SQL data sources SharePoint Designer Uses OOB connectors Indexed by Search

Custom Code (>= SP 2010 .NET) If there it no OOB connector available If there is no Connector Product available

Extending SharePoint SearchTop Down into Extensibility with Indexing Connectors

SharePoint Farm

Source: TechNet, http://zoom.it/t7G3#full

Distributed Search Roles

Source: TechNet, http://zoom.it/ktlC#full

Search and Extensibility*

Source: TechNet, http://msdn.microsoft.com/en-us/library/office/jj163300.aspx

Connectors

Source: Professional SharePoint® 2013 Development (ISBN: 978-1-118-49582-7)

* Indexing Connectors

Crawl Component – Entering the Feeding Chain

Source: http://searchunleashed.wordpress.com/2013/12/23/crawling-content-with-sharepoint-2013-search/

Indexing Connector

Source: TechNet, http://msdn.microsoft.com/en-us/library/office/jj163300.aspx

Search Connector Framework“Connector development is harder than it looks.”(Professional SharePoint 2013 Development)

Building Blocks

Start addresses

Protocol Handler

Content Source ConfluenceIndexing

Connector

Register DevelopConfigure

Confusing Terms Search Connector Indexing Connector Custom Indexing Connector Custom Connector BCS Connector BCS Custom Connector BCS Entity Connector BCS Indexing Connector Custom Connector (BCS) .NET Assembly Connector Protocol Handler

What is an Indexing Connector? Indexing Connector = BCS Connector + X

+ X =“Translation” BCS <-> Search

Source: Professional SharePoint® 2013 Development (ISBN: 978-1-118-49582-7)

Source: TechNet, http://msdn.microsoft.com/en-us/library/office/jj163300.aspx

Development: Tools Visual Studio

SharePoint Project -> SP solution Class Library -> Connector (GAC Deployment)

PowerShell Deploy Solution Register Connector

BDC Model (XML) Describes external system (see External Content Type) May use SharePoint Designer as editor or to get ideas

Existing Connector or sample highly recommended!

System Requirements SharePoint 2010 & 2013 SharePoint Foundation not supported Full Trust Solution GAC Deployment on Crawl Servers

Sample: Content Source for Confluence Connector

Custom Connector

Register Connector

MSSearch.exe

Protocol HandlerMSSDmn.exe

Custom Connector

HKLM:\SOFTWARE\Microsoft\Office Server\<spversion>\Search\Setup\ProtocolHandlers\

OSearch15.ConnectorProtocolHandler.1-> COM-Component ConnectorPH.dll

New-SPEnterpriseSearchCrawlCustomConnector

.NET

External System: Confluence

BDC Model*

BCS Connector Parts*

LobUri*INamingContainer*

CustomConnector

Crawl DBMSSearch

starts

BCS Connector

8

MSSDmn.exe

3 5

6 7

9

12

14

4

1 2 10 11

CPP

15

16

13

Content Source

*Dev

1) confluence://server/?entity=page2) { Entity = Page, Identity = null }5) PageIDs: [1,2,3,…]6) {Entity = Page, Identity = 1 }7) AccessUri =

confluence://server/?id=1&ispage=110) confluence://server/?id=1&ispage=111) { Entity = Page, Identity = 1 }14) Page: { DATA }

Crawl

BDC Model*

BCS Connector Parts*

LobUri*INamingContainer*

CustomConnector

Crawl DBMSSearch

BCS Connector

MSSDmn.exe

5

14

4

CPP

13

Content Source

*Dev

5) PageIDs: [1,2,3,…]Incremental Crawl needs additional information14) Page: { DATA }Security trimming needs ACL or Claims directly attached to the element

Crawl

Incremental Crawl

Entity IDs + Change Date BDCLastModifiedTimestamp

Change Log: Changed and deleted IDs since the last crawl ChangedIDEnumerator, DeletedIDEnumerator

Page IDs: 1, 2, …• 1: Change date: 02.03.2014• 2: Change date: 03.03.2014

MSSDmn

Page IDs: 1, 2, …• Changed: 1, 2• Deleted: 3

MSSDmn

Goal: only crawl items changed since the last crawl

Dev Wrap-up – Indexing Connector Debugging Tips

Attach to MSSDmn.exe Increase timeouts for crawling (CA) otherwise MSSDmn.exe will be terminated net stop osearch15

Keywords LobUri INamingContainer

Sample Project MyFileConnector: http://

code.msdn.microsoft.com/office/SharePoint-2013-MyFileConne-79d2ea26 Resources

Default Connectors in SharePoint 2013: http://technet.microsoft.com/en-us/library/jj219746.aspx

Search Connector Framework in SharePoint 2013: http://msdn.microsoft.com/en-us/library/office/ee556429.aspx

Search Connector for ConfluenceCommunardo: Product “Connector for SharePoint Search” for Confluence

Click to insert photo.

What is Confluence?

Image source: https://www.atlassian.com/software/confluence

• Enterprise Wiki from Atlassian

• Spaces• Wiki Pages in

Spaces, Hierarchical

Confluence Content StructureSpaces

Pages

Comments

Blog Posts

Attachments

Planning the Search Connector for Confluence

Choose content to index Determine content structure of confluence (Spaces, Pages, …)

Plan Permissions Permissions are attached directly to the indexed element Hierarchical Permissions from Confluence cannot be represented 1:1

Evaluate Confluence Webservice Determine API limits

Choose Content for Crawling Confluence has hierarchical content:

Spaces Pages Attachments …

Flatten hierarchy for crawl:

Space Page/Blog Post

Attachment

Comment

Space Page

Attachment Comment

Blog Post

Map Confluence Users to AD Accounts Challenge: Confluence might not be attached to AD; can contain local users and groups

Map Confluence users and groups to AD principals during crawl

Space Page / Blog Post

Attachment

Comment AD

Map hierarchical permissions to single ACLs Confluence uses hierarchical permissions

Confluence permissions cannot be mapped 1:1 to single item ACLs

Translation „Confluenc Permission -> ACL“ necessary

Space Page / Blog Post

ACL

+

SharePointConfluence Connector

Lessons Learned“Experience is a hard teacher because she gives the test first, the lesson afterwards.” - Vern Law

Lessons Learned Planning is everything

Identity-Mapping, API Limits, Network limits, Permissions Plan permissions well

No hierarchical permissions in Search Index ACL? Claims? …

Not all BCS methods are supported BCS method “AssociationNavigator” (for parent-child relationships ) cause

problems with incremental crawl Stress Test

SharePoint Search is an excellent tool for DoS Use Crawler Impact Rules

Outlook

Office 365 Cloud Search cannot be extended

No GAC Deployment Hybrid setup to get external search results into the cloud Search Connector on-premise Inbound Hybrid Search brings results to Office 365

Federated Search Results No Refinement Roadmap contains solution, but no date given (talks 306 & 320

#SPC14)

Questions?

Thank you!