Toronto SharePoint User Group

Bill BrockbankNavantis Inc.billb@navantis.com

Architecture and TopologyLogical Architecture - Components• Crawl• Content Processing• Analytics Processing• Index• Administration• Query Processing

• Search Process• Topologies - Examples

Logical Architecture

Crawling the Content• The crawl role is responsible for crawling content

sources. It delivers crawled items – both the actual content as well as their associated metadata – to the content processing component• Invokes connectors or protocol handlers to content sources to

retrieve data• Does not do any document parsing (Content Processing

Component does that)• Information about content sources, schedules, etc. are

synchronized to the registry on crawl role servers from the search admin database

• The Crawl Database is used by the crawl component to store information about crawled items and to track crawl history• Holds information such as the last crawl time, the last crawl ID

and the type of update during the last craw.

Content Processing Content Processing Component

Processes crawled items and feeds these items to the index component

Document Parsing happens through Format Handlers iFilter is supported through a Generic iFilter format handler iFilters are still the extensibility platform for SharePoint 15

Heart of the indexing process: transforms crawled items into artifacts that can be included in the search index by carrying out operations such as document parsing and property mapping

Performs linguistic processing at index time (e.g. language detection and entity extraction)

Writes information about links and URLs to the Link Database directly

Generates phonetic name variations for people search

Analytics Processing Component Search Analytics analyzes crawled items and how users interact

with search results. Usage analytics analyzes usage events e.g. like views from the

event store When an user does an action (e.g. view a page) the event is

collected in usage files on the WFE’s and regularly pushed to event store where they are stored until processed

Results are then returned to the Content Processing Component to be included in the search index

You can add write code to add custom events Analytics Processing Component supports scaling out:

Add more APC roles to have analysis complete faster Adding more Link databases to increase capacity for links

and search clicks, and potentially speed up the DB extraction

Adding more reporting databases to store more reports as well as improving SQL throughtput in retrieveing reports

List of Sub Analyses Search Analytics

Link and Anchor text analysis Click Distance Search Clicks Deep Links Social Tags Social Distance Search Reports

Usage Analytics Recommendations Usage Counts Acivity Ranking

Index Component Used in both feeding and query processes:

Feeding: receives processed items from the content processing component and writes those items to index files

Query: receives queries from the query processing component and provides results sets in return

It also physically moves around indexed content when the index architecture is changed by the Search Administration Component

Query Processing Component

Performs linguistic processing at query time: Word breaking, stemming, query spellchecking, thesaurus

Analyzes and processes search queries and results. When the component receives a query from the search front-end, it analyzes and processes the query to attempt

to optimize precision, recall and relevancy. The processed query is then submitted to the index component(s). As part of this it also decides which query rules are applicable, which index to send the query to, and whether to do any pre- or post-

processing of the query

The index component returns a result set based on the processed query back to the query processing component, which in turn processes that result set before sending it back to the search front-end.

Search Administration Search Admin Component

Runs number of system processes required for search Is responsible for search provisioning and topology changes Coordinates search components – Content Processing, Query

Processing, Analytics, and Indexing.

Search Admin DB Stores search configuration data:

Topology Crawl rules Query rules Managed property mappings Content sources Crawl schedules

Stores Analytics settings Does not store ACLs anymore

Search Processes Host Controller

A Windows Service that supervises NodeRunner process(es) on a given box It restarts failed NodeRunner processes: if a NodeRunner fails, the HostController will

detect this and restart it NodeRunner.exe

Is the process that hosts the search components There might be several instances of this process on a single box

MSSearch.exe Is the Windows Service who hosts the Crawl Component

Search Host Process Multiple NodeRunner instances can run on

the same server Each NodeRunner instance hosts one

search component E.g. If you have two index replicas on one server

you will have two NodeRunner instances – one for each replica

On a default single server install there will be 5 instances of the NodeRunner.exe process

NodeRunner.exe

Crawl Component

NodeRunner.exe

Query Processing Component

NodeRunner.exe

Analytic Processing Component

NodeRunner.exe

Content Processing Component

NodeRunner.exe

Index Component

Enterprise Search – Query Configuration

Query ConfigurationRankingQuery Spell CorrectionQuery Rules

Ranking Ranking model describes which criteria are included in sorting, how

much they contribute to the rank score and how they relate to one another

Custom ranking models are managed through PowerShell and the public OM using XML files as it was in SharePoint 2010

Ranking model for a specific query can be selected at query time by setting the RankingModelId of the query; otherwise default is used NOTE: This property is currently not available in the OOB search results web part

Working with Ranking Models A search service administrator can do the following basic operations on

the rank models: List ranking models Specify a default ranking model Change an existing custom ranking model Delete an existing custom ranking model Create a new ranking model (either based on an existing one or from scratch) Import and export a ranking model to XML Specify how much weight the various properties should have in the model

These operations are all administered through PowerShell There may be a CodePlex GUI-based solution after RTM

Query Spelling Correction Customizations to Query Spelling Correction is now managed in the term

store – both inclusions and exclusions We also still have a dynamic dictionary that is created based on content in

the index, or you can switch to the static OOB dictionary Here’s an example:

Provides customizable •“did you mean” functionality:

Query Rules Query Rules are a new feature in SharePoint 15 that help act upon the

“intent” of a query Query Rules are composed of three top level elements:

Query Conditions (i.e. matching rules) Query Actions (i.e. what do you do when you find a match) Publishing Options (i.e. when should this rule be active)

Rules are created at the site collection level, but you will probably also be able to create them at the SSA level through PowerShell (TBD by RTM)

Several Query Rules are provided out of the box and can be found in the Site Settings for each site collection

Query Rules Implementation Query Rules allow you to have search requests from a user trigger

multiple queries and multiple result sets. This differs from SharePoint 2010 where there were only simple queries – one

query has one set of results. All matching Query Rules can generate results; the query orchestration

engine retrieves all the different sets of search results and organizes them for rendering to the user

The results themselves are then rendered using a new feature called Query Results display templates, which is discussed later

Query Rules - Conditions Conditions allow you to define the circumstances under which your query rule will

apply You can choose to match on these use cases:

Query contains a specific word or words Query contains a word in a specific dictionary Query contains an action word that matches a specific phrase or term set Query is common in a different source (like Videos Result Source) Results include a common result type (like file type) Advanced rule – match across a set of terms, dictionary, regular expression, etc.

You can also match if a query is against a specific Result Source or category (like a Topic Page category)

Each Query Rule can have multiple conditions

Query Rule Actions There are three types of actions:

Assign a “promoted result” to the query Promoted result is similar to Best Bets with SharePoint 2010 or Visual Best Bets with FAST

Search 2010, but much more flexible Create and display a Result Block

Result blocks specify an additional query to run and how to display results The feature includes a full query designer so you can build and test queries before finalizing

them You can include the results above those returned by core results, or interleaved by ranking You can also choose custom display templates instead of the default for the result

Change the query that returns the core results Change the original query by changing the query terms, adding additional terms, apply an

XRANK formula to the query, etc.

You can add multiple best bets or results blocks for each query rule

Query Rules Publishing Options Publishing options for a Query Rule determine when it will be used It works just like rules for publishing a page in a publishing site template

You decide whether the rule is active You decide when to start and when to finish using the rule, or don’t set dates at

all and it’s always used You can configure a review date, which is basically an email reminder for you to

look at the rule again Publishing a rule is particularly useful in commerce scenarios. For

example, queries that should return items you want to percolate to the top and sell during a particular period of time

Enterprise Search – New Search Results Display

Options

Search UI ConfigurationResult Types

Display TemplatesSearch NavigationSearch RefinementQuery SuggestionsThumbnail PreviewsSite Level Search Admin Summary

Result Types SharePoint 15 includes a new framework for presenting search results

to end users called result types Result types have several elements that enable each type to be

rendered differently:

Result Type Rules You can create a variety of different rules on which to match:

Equality(= or !=), comparison (< or >), or logical (AND or OR or NOT) You can have multiple rules as a condition You can use managed properties in a rule

In B1R there will be a new UI for creating these rules, so what you see in the B1 bits will change quite a bit

Name Rule Matches…Word FileExtension=docx OR FileExtension =doc All Word documentsSpec ContentType=Spec AND

(FileExtension=docx OR FileExtension =doc)Specs which are also Word documents

Example

Result Type Management Result types are managed as follows:

Result Types can apply to an entire search service application or to a specific site or web

OOB Result Types are immutable but can be overridden with a copy There is an inheritance hierarchy to result types in a site: the

current web, plus the site collection root, plus the Search Service Application.

So if you add a new result type to some random web, the sub webs under it won’t inherit that type unless you are working in the site collection root web

Display Templates Display templates contain the following characteristics:

They define the visual layout of a Result Type A template is really just HTML You can edit it with ANY HTML designer you want – DreamWeaver,

SharePoint Designer, Visual Studio, Notepad – doesn’t matter You add placeholders to your HTML file where managed properties

should be emitted It gives you a real WYSIWIG experience when designing templates

A set of display templates is included OOB; you can find them in the Master Pages/Display Templates/Search folder Site collection admins can upload new display templates.

Result Type with Display Template Example Here’s an example from one of the Hands On Labs that demonstrates

building a custom result type and display template for .TXT file search results:

Result Types

demo Steve PeschkaSr. Principal ConsultantMicrosoft Corporation

Search Refinement There are two different modes for the refiner web part: search results

and faceted navigation With search results the refinement data works essentially the same as

SharePoint 2010 With faceted navigation it uses a term from the term store to filter what kind of

data should be displayed (explained on next slide) Refinement is different with SharePoint 15 in that you can define

display templates to use for rendering different kinds of refinements In SharePoint 2010 you had to write your own custom refiner

Faceted Navigation with Search Refiners With Faceted Navigation, it is used in conjunction with term sets that

are used for navigation With each term you select which managed properties should be used

as refiners with that term Within the managed property you need to configure it as “Refinable” Example:

You have term store terms Camera and Laptop You have managed properties Megapixel Count, Color and Manufacturer For Camera term, you add refiners for Megapixel Count and Manufacturer For Laptop term you add refiners for Color and Manufacturer

Another Example: Department is the term store term, Customers and Projects could be refiners

Query Suggestions Query suggestions take a big leap forward in usability. It improves on

the experience in SharePoint 2010 as follows: Your personal SharePoint activity factors into the query suggestions, i.e. you

have a personal query log It includes weighting based on sites that you have previously visited It uses the most frequent queries across all users that “match” the search terms

The behavior of the query suggestions turns into more of a “browse and find” kind of experience

You can also add inclusion and exclusion lists for suggestions via the SSA admin pages

Query Suggestion Types There are two types of query suggestions – what you see

when you are entering a query, and what you see when you get results When entering a query you will see two types of suggestions:

A list of items you have clicked on before from your personal query log A list of items that others are typing for their queries

When you get query results back, you will get another set of suggestions

They are a list of links that you have clicked through at least twice before and match your search criteria

Query Suggestion Examples• Here are some examples of how query suggestions might look:

Before the Query Suggestion Above are items based on what other users have

queried and found Bottom 3 items are ones you clicked before

After the Query Suggestion Top 3 items are ones you clicked before

Thumbnail Previews The new Office Web Apps is the engine for thumbnail previews in SharePoint 15 The BIG WIN HERE – you can now browse through the entire document in the

preview See all pages, see animations, zoom in, scroll through the entire document The point of this is to allow users to find the exact item they’re looking for right in search

results – no more clicking a result, hitting the back button, and on and on until they find the one they’re looking for

It also addresses the two major shortcomings of thumbnails in SharePoint 2010: It could only be used with FAST Search It did NOT work with claims authentication Since there’s only one search engine in SharePoint 15 you get document previews out of the box

In a different twist, previews only work with claims authentication – it will not work with classic Windows authentication

Summary: New Site Search Admin Tasks Just to summarize, here are things site admins can do to manage

search themselves:Task Site

Collection Admin

Site Admin

Create and/or override Query Rules, including Promoted Results

Create and/or override Result types and display templates

Create and/or override Result Sources, either for remote locations or as a custom search “vertical”

Create managed properties

Create refiners

Start a local crawl – can even be done down to the list level

Online Content links:• IT Pro training for SharePoint 2013

http://technet.microsoft.com/en-us/sharepoint/fp123606

• SharePoint 2013 enterprise search Module:http://technet.microsoft.com/en-us/sharepoint/fp123606