sp24s088 - custom indexing connectors - how to integrate external systems into sharepoint search
Post on 18-Nov-2014
681 Views
Preview:
DESCRIPTION
TRANSCRIPT
Custom Indexing ConnectorsHow to integrate external systems into SharePoint Search
Heinrich UlbrichtGermany, MEZApril 16th /17th, 2014
SP24 Search SessionsSession
Time (GMT)
Time (EST) Title Level Speaker Twitter Handle
#SP24S090 5:00 1:00 Using the Content Enrichment Web Service with SharePoint Server 2013 Search
Technical - Advanced
Sezai Komur - NEC Australia @sezai
#SP24S021 6:00 2:00 2013 Search, Display Templates, Query Rules, Result Types
Technical - Beginner
Omer Zubair - CSC @Omer_Zubair
#SP24S047 8:00 4:00 Real-life experience building search-driven applications for product-centric sites
Technical - Advanced
Marius Constantinescu - blue-infinity SA @c_marius
#SP24S088 10:00 6:00Custom Indexing Connectors - How to integrate external systems into your SharePoint Enterprise Search
Technical - Advanced
Heinrich Ulbricht - Communardo Software GmbH
@h_ulbricht
#SP24S083 11:00 7:00 Search First Migration with SharePoint 2013 – Benefits / Disadvantages
Technical - Intermediate
Max Melcher - Alegri International Service GmbH
@maxmelcher
#SP24S034 14:00 10:00 The Search Immaturity Cycle, and How to Create a Search Strategy
Business - Intermediate
Jeff Fried - BA Insight @jefffried
#SP24S009 16:00 12:00 10 Things to Know about Search in SP2013 and Office 365
Business - Beginner
Agnes Molnar - Search Explained @molnaragnes
#SP24S019 18:00 14:00 Search Topology and Optimization Technical - Advanced
Mike Maadarani - MCM Consulting @mikemaadarani
#SP24S050 22:00 18:00 Search Driven Application Development in SharePoint 2013
Technical - Intermediate
Matt Youngstrom - Magenic @spguru
#SharePoint#Apps#Search#WindowsPhone
Heinrich UlbrichtSoftware Developer for SharePoint
Communardo Software GmbHDresden, Germany
Contact@h_ulbrichtheinrich.ulbricht@communardo.dehttp://sharepointbrainpump.blogspot.com/
Session ObjectivesPresent extensibility of SharePoint SearchIndexing Connectors: good understanding of architecture
Key TakeawaysSharePoint can be the Enterprise Search for all your contentIf you’ve never developed an indexing connector – get help
Session Objectives and Takeaways
Vision “One Enterprise Search” with SharePointExtensibility of SharePoint SearchSearch Connector FrameworkConnector for SharePoint Search for ConfluenceLessons Learned
Agenda
Challenge: Non-integrated Systems
SAP
SQLWCF
Siebel
Dynamics
Confluence?
No Integration – Consequences User is confused
Where to find information? Different search centers with different usability and power (filters, …) Different search result formatting Different search result quality (ranking, …)
Efficiency drops!
Solution: Search Integration
1. No integration (no solution…)2. Links
3. Federated Search
4. Integrated Search
Level of Search Integration
Links Yammer-”Integration” came with SP2013 SP1 -> Link to Yammer Search Center
Federated Search Search results next to each other
Result blog for Bing search results
SharePoint results
Integrated Search One search result view; E.g.: SharePoint + Confluence:
Common relevancy, Sorting, Refiners, Managed Properties
“One Enterprise Search”@jeffteper: “Best Enterprise Search in the World” #SPC14
Transparent System LandscapeSAP
SQL WCF
SiebelDynamics
ConfluenceWork like
a network
Vision• One Enterprise
Search• Go to one
place for information
• Search spans all systems
Communardo
Building Blocks
1) BCS3) Search Connector Framework
2) SharePoint
Search4)
Confluence
Indexing Connector
Connect to External Systems with SharePoint
Access external data with Business Connectivity Services (BCS) Use stereotypical operations to access external system (“Read
Entity Instance”, “Create Entity Instance”) Structure / Projection described in BDC Model (XML File) Access operations executed in OOB or Custom Connectors
SQL, WCF, Oracle, SAP, Siebel, Dynamics, Confluence
BCS
BCS – Integrate External Systems with Connectors
Source: Professional SharePoint® 2013 Development (ISBN: 978-1-118-49582-7)
Extend Search with BCS Declarative
BCS can access external content using External Content Types (ECT) Good for reaching out to SQL data sources SharePoint Designer Uses OOB connectors Indexed by Search
Custom Code (>= SP 2010 .NET) If there it no OOB connector available If there is no Connector Product available
Extending SharePoint SearchTop Down into Extensibility with Indexing Connectors
SharePoint Farm
Source: TechNet, http://zoom.it/t7G3#full
Distributed Search Roles
Source: TechNet, http://zoom.it/ktlC#full
Search and Extensibility*
Source: TechNet, http://msdn.microsoft.com/en-us/library/office/jj163300.aspx
Connectors
Source: Professional SharePoint® 2013 Development (ISBN: 978-1-118-49582-7)
* Indexing Connectors
Crawl Component – Entering the Feeding Chain
Source: http://searchunleashed.wordpress.com/2013/12/23/crawling-content-with-sharepoint-2013-search/
Indexing Connector
Source: TechNet, http://msdn.microsoft.com/en-us/library/office/jj163300.aspx
Search Connector Framework“Connector development is harder than it looks.”(Professional SharePoint 2013 Development)
Building Blocks
Start addresses
Protocol Handler
Content Source ConfluenceIndexing
Connector
Register DevelopConfigure
Confusing Terms Search Connector Indexing Connector Custom Indexing Connector Custom Connector BCS Connector BCS Custom Connector BCS Entity Connector BCS Indexing Connector Custom Connector (BCS) .NET Assembly Connector Protocol Handler
What is an Indexing Connector? Indexing Connector = BCS Connector + X
+ X =“Translation” BCS <-> Search
Source: Professional SharePoint® 2013 Development (ISBN: 978-1-118-49582-7)
Source: TechNet, http://msdn.microsoft.com/en-us/library/office/jj163300.aspx
Development: Tools Visual Studio
SharePoint Project -> SP solution Class Library -> Connector (GAC Deployment)
PowerShell Deploy Solution Register Connector
BDC Model (XML) Describes external system (see External Content Type) May use SharePoint Designer as editor or to get ideas
Existing Connector or sample highly recommended!
System Requirements SharePoint 2010 & 2013 SharePoint Foundation not supported Full Trust Solution GAC Deployment on Crawl Servers
Sample: Content Source for Confluence Connector
Custom Connector
Register Connector
MSSearch.exe
Protocol HandlerMSSDmn.exe
Custom Connector
HKLM:\SOFTWARE\Microsoft\Office Server\<spversion>\Search\Setup\ProtocolHandlers\
OSearch15.ConnectorProtocolHandler.1-> COM-Component ConnectorPH.dll
New-SPEnterpriseSearchCrawlCustomConnector
.NET
External System: Confluence
BDC Model*
BCS Connector Parts*
LobUri*INamingContainer*
CustomConnector
Crawl DBMSSearch
starts
BCS Connector
8
MSSDmn.exe
3 5
6 7
9
12
14
4
1 2 10 11
CPP
15
16
13
Content Source
*Dev
1) confluence://server/?entity=page2) { Entity = Page, Identity = null }5) PageIDs: [1,2,3,…]6) {Entity = Page, Identity = 1 }7) AccessUri =
confluence://server/?id=1&ispage=110) confluence://server/?id=1&ispage=111) { Entity = Page, Identity = 1 }14) Page: { DATA }
Crawl
BDC Model*
BCS Connector Parts*
LobUri*INamingContainer*
CustomConnector
Crawl DBMSSearch
BCS Connector
MSSDmn.exe
5
14
4
CPP
13
Content Source
*Dev
5) PageIDs: [1,2,3,…]Incremental Crawl needs additional information14) Page: { DATA }Security trimming needs ACL or Claims directly attached to the element
Crawl
Incremental Crawl
Entity IDs + Change Date BDCLastModifiedTimestamp
Change Log: Changed and deleted IDs since the last crawl ChangedIDEnumerator, DeletedIDEnumerator
Page IDs: 1, 2, …• 1: Change date: 02.03.2014• 2: Change date: 03.03.2014
MSSDmn
Page IDs: 1, 2, …• Changed: 1, 2• Deleted: 3
MSSDmn
Goal: only crawl items changed since the last crawl
Dev Wrap-up – Indexing Connector Debugging Tips
Attach to MSSDmn.exe Increase timeouts for crawling (CA) otherwise MSSDmn.exe will be terminated net stop osearch15
Keywords LobUri INamingContainer
Sample Project MyFileConnector: http://
code.msdn.microsoft.com/office/SharePoint-2013-MyFileConne-79d2ea26 Resources
Default Connectors in SharePoint 2013: http://technet.microsoft.com/en-us/library/jj219746.aspx
Search Connector Framework in SharePoint 2013: http://msdn.microsoft.com/en-us/library/office/ee556429.aspx
Search Connector for ConfluenceCommunardo: Product “Connector for SharePoint Search” for Confluence
Click to insert photo.
What is Confluence?
Image source: https://www.atlassian.com/software/confluence
• Enterprise Wiki from Atlassian
• Spaces• Wiki Pages in
Spaces, Hierarchical
Confluence Content StructureSpaces
Pages
Comments
Blog Posts
Attachments
Planning the Search Connector for Confluence
Choose content to index Determine content structure of confluence (Spaces, Pages, …)
Plan Permissions Permissions are attached directly to the indexed element Hierarchical Permissions from Confluence cannot be represented 1:1
Evaluate Confluence Webservice Determine API limits
Choose Content for Crawling Confluence has hierarchical content:
Spaces Pages Attachments …
Flatten hierarchy for crawl:
Space Page/Blog Post
Attachment
Comment
Space Page
Attachment Comment
Blog Post
Map Confluence Users to AD Accounts Challenge: Confluence might not be attached to AD; can contain local users and groups
Map Confluence users and groups to AD principals during crawl
Space Page / Blog Post
Attachment
Comment AD
Map hierarchical permissions to single ACLs Confluence uses hierarchical permissions
Confluence permissions cannot be mapped 1:1 to single item ACLs
Translation „Confluenc Permission -> ACL“ necessary
Space Page / Blog Post
ACL
+
SharePointConfluence Connector
Lessons Learned“Experience is a hard teacher because she gives the test first, the lesson afterwards.” - Vern Law
Lessons Learned Planning is everything
Identity-Mapping, API Limits, Network limits, Permissions Plan permissions well
No hierarchical permissions in Search Index ACL? Claims? …
Not all BCS methods are supported BCS method “AssociationNavigator” (for parent-child relationships ) cause
problems with incremental crawl Stress Test
SharePoint Search is an excellent tool for DoS Use Crawler Impact Rules
Office 365 Cloud Search cannot be extended
No GAC Deployment Hybrid setup to get external search results into the cloud Search Connector on-premise Inbound Hybrid Search brings results to Office 365
Federated Search Results No Refinement Roadmap contains solution, but no date given (talks 306 & 320
#SPC14)
top related