How to manage and troubleshoot Search: A practical guide Brian Pendergrass and Jon WaitePremier Field EngineersMicrosoft

SPC375

Who is this session for?SharePoint IT professionals who support/manage Search environments300 level contentDiscussion of Search internals

Ask the ExpertsCome find us to talk more about Search

The decoder ring…SSA QPC SQ&SSSearch Service Application Query Processing Component Search Query & Site Settings

Juno/MarsInternal naming conventions for Search that appear in ULS for various activities

WCF (Windows Communication Foundation)A framework for building service-oriented applications where clients communicate with services through defined endpoints

WSDL (Web Services Description Language)Describes the complete contract for application communication

AAMs( Alternate Access Mappings)Enable a Web application that receives a request for an internal URL to return pages that contain links to the public URL for the zone

SharePoint 2013 Search Architecture

Unit of Scale / Role boundary

Admin

mssearch.exe noderunner.exe xnoderunner.exe

Query Processing(IMS) WFE

noderunner.exe

Search

Admin

Analysis Engine (APC)

CRAWL

STORECrawl

Store

Searc

h Q

uery

an

d S

ite S

ett

ing

s

(

wit

h s

earc

hse

rvic

e.s

vc

En

d

Poin

ts)

Analytics Reporting

SS

A P

roxy

Query

OM

Serv

er-

side

UI

Query

CSO

M

Links StoreUsage DB

SharePoint Crawler(Crawl)

Content Processing Search core

FAST search core

Custom Enrichment Web Service

mssdmn.exe

mssdmn.exe

mssdmn.exe

mssdmn.exe

CSS (Load

Balancer)

…

Processing Flows

Index Router

Journal

\\srv\gthrsvc network

share

Content Plug-In API

Content

Protocol Handlers• sts4[s]

(SharePoint)• sps3[s] (People)• http[s] (Web)• file (File Share)• bdc (Business)

Rele

vanc

y E

ngin

e

Que

ry F

low

s

IMS Query Pool

…

Result Mixer

Mars Lookup Operator

(MLO)

even

ts

w3wp.exe

Content Router

mssearch.exe w3wp.exe

w3wp.exe

noderunner.exe

UX

Crawling and Feeding Chain

Index Core

Query Chain

Analytics Service

noderunner.exe

Five key takeaways from this session…Application Server Administration Service Timer JobSynchronizes search instances across the farm

Crawl state lives in the databaseMore precise state can be found in the MSSCrawlHistory/MSSCrawlComponentsState tables

Follow a crawled document via ULSUse VerboseEx ULS to track from Crawler to CPC to Index

Follow a query via ULSUse Merge-SPLogfile, Query starts at the Web Front End

Crawl the default zoneDon’t use server name mappings for SharePoint content

Troubleshooting Crawling and Feeding

Problem: My Crawl is stuck!Where to look:Check Central Admin Search Topology page for errorsCheck Timer Service/Timer JobsCheck ULS logs on Crawler for errorsCheck Search databases

MSSCrawlHistoryWhat is the crawl status? Requested status?

MSSCrawlComponentStateWhat are the component states for your crawl?

MSSCrawlQueueAre items being added/removed?

MSSCrawlURLAre items marked with this crawl ID being changed?

Are there blocking/blocked queries on the databases?

Central Admin Search TopologyCheck Central Admin Search Topology page for errors Ensure the components are online (green checkmark)

Check Timer Service/Timer JobsApplication Server Administration Service Timer Job Ensure this job is running and Synchronize is succeeding by examining ULS

Message: Name=Timer Job job-application-server-admin-serviceMessage: synchronizing search service instance

Message: Done synchronizing SearchServiceInstance.Message: Leaving Monitored Scope (Timer Job job-application-server-admin-service). Execution Time=XXXX

ULS LogsExample:The application cannot be initialized.Context: Application 'Search_Service_Application'

Details:The gatherer is shutting down (0x80040d23)

ERRORApplication Server Administration job failed for service instance Microsoft.Office.Server.Search.Administration.SearchServiceInstance (ee54f9c5-2b43-4466-9e21-483389331abb).

Reason: The gatherer is shutting down.

Technical Support Details:System.Runtime.InteropServices.COMException (0x80040D23): The gatherer is shutting down. at Microsoft.Office.Server.Search.Administration.MSSITLB.IGatherManagerAdmin3.get_ConfigurationVersion() at Microsoft.Office.Server.Search.Administration.Gatherer.ProvisionGlobalProperties() at Microsoft.Office.Server.Search.Administration.SearchServiceInstance.Synchronize() at Microsoft.Office.Server.Administration.ApplicationServerJob.ProvisionLocalSharedServiceInstances(Boolean isAdministrationServiceJob)

ULS LogsCheck ULS logs on Crawler for errorsIf Application Server Administration Service Timer Job isn’t running, you may see no entries at all.

SSP Job Control Service instance Status = onlineHidden “Service on Server”PowerShell:

$sspjob = (Get-SPFarm).Servers[“<servername>"].ServiceInstances | where {$_.typename -eq "SSP Job Control Service"}

The SharePoint Crawl is stateful with transitional “ing” states (e.g. starting, completing, stopping, pausing) and corresponding rest states (e.g. started, completed,

stopped, paused)

Crawl state lives in the database

Status SubStatus

This combination means… Crawl Component Crawl DB

0 0 New crawl, requesting start x1 1 Starting, Add Start Address(es) x1 2 Starting, waiting on Crawl Component(s) x4 1 Crawling4 2 Crawling, Unvisited to Queue4 3 Crawling, Delete Unvisited x4 4 Crawling, Wait for All Databases x5 0 Failed to Start (e.g. Another Crawl Already Running)7 Resuming x8 1 Pausing, Waiting on Crawl Component(s) to Pause x8 2 Pausing, Complete Pause x9 Paused

11 0 Completed12 Stopped13 1 Stopping, Waiting on Crawl Component(s) to Stop x13 2 Stopping, Complete Stop x14 1 Completing, Waiting on Crawl Component(s) to Complete x14 2 Completing x14 4 Completing, Get Deletes Pending x

Translating MSSCrawlHistory Status

Source:[SSA].[dbo].[proc_MSS_CrawlAdmin]

For Crawls Waiting on Crawl Components…SELECT   hist.CrawlId, hist.CrawlType, hist.ContentSourceId, hist.StartTime, hist.Status, hist.SubStatus, hist.Request,  ccState.ComponentID, ccState.Status as ccStatus, ccState.SuspendedCount FROM [SSA].[dbo].[MSSCrawlHistory] AS hist WITH (nolock) INNER JOIN   [SSA].[dbo].[MSSCrawlComponentsState] AS ccState   ON hist.CrawlId = ccState.CrawlId WHERE hist.Status Not In (5,11,12)and hist.crawlid > 2

Are there blocking queries on the databases?Check with the DBACheck the SQL Server Blocked Queries ReportDoes SQL have adequate drive space for database/logs?

If you do find a blocked query, you need to understand why/what it’s doingSearch will often simply re-issue the same blocking query as soon as it failsThis may mean updating statistics so SQL can choose a better execution plan

Yes, what do we do?

Crawling/Gathering

Crawl Component

Content Processin

g Componen

t

Follow a crawled document via ULS

Content

\\srv\gthrsvc network

share

blob

Crawl Store

AF7YN

Connector

DS86

AI3EF

Index Component

AJPND

E5G1

Feeding/Processing

Troubleshooting Queries

Searc

h

Ind

ex

Core

Following the path of a Query…

User quer

y

Web01

JunoSendRequest MarsLookupOperator

[HTTP] EndPoint: SearchService.svc

SQ&SSSearch Query

& Site Settings

WCFSendRequest

For a query, a single Correlation ID spans across components and servers

QPCQuery Processing

Component

[net.tcp] EndPoint ImsQueryInternal

http://web01/results.aspx?k=tps

Query: Sorry, something went wrong.

SharePoint 2013 SharePoint 2010

Demo1. Run a query …and watch if fail

with “Sorry, something went wrong”

2. Use the Correlation ID to filter ULS events across all applicable servers Merge-SPLogFile -Path $target -Correlation $id

A case of something going wrong…

SharePoint Foundation Topology e5mc MediumWcfSendRequest: RemoteAddress: 'http://qpcServer:32843/-GUID-of-the-SSA-/SearchService.svc' Channel: 'Microsoft.Office.Server.Search.Administration.ISearchServiceApplication' Action: 'http://tempuri.org/ISearchQueryServiceApplication/Execute'

SharePoint Server Search Query dka5 HighSearchServiceApplicationProxy::Execute--Error occured: System.ServiceModel.ServerTooBusyException: The HTTP service located at http://qpcServer:32843/-GUID-of-the-SSA-/SearchService.svc is unavailable. This could be because the service is too busy or because no endpoint was found listening at the specified address. Please ensure that the address is correct and try accessing the service again later. ---> System.Net.WebException: The remote server returned an error: (503) Server Unavailable. at System.Net.HttpWebRequest.GetResponse()

User quer

y

Web01

SQ&SSSearch Query

& Site Settings

WCFSendRequest SQ&SSSearch Query

& Site Settings

Common networking issues

Ports and ProtocolsPorts used by the search index component — TCP 16500-16519 (intra-farm only)Ports required for Windows Communication Foundation communication — TCP 808Ports required for communication between Web servers and service applications (the default is HTTP):• HTTP binding: TCP 32843• HTTPS binding: TCP 32844

Disable TCP Chimney/Task Offloadnetsh int tcp show global | findstr Chimney

netsh int ip show global | findstr "Task Offload"

Important event in ULS for Query PerfSearch Query Processing aizgm MediumMicrosoft.Office.Server.Search.Query.Pipeline.Executors.QueryPipelineFlowExecutor : FlowExecutor done: SearchApplication=cb4954e3-e4ad-40bd-b0e2-1b1db290c668. ParentFlow=Microsoft.ProductivitySearchFlow SubFlowTimings: QueryRuleConditionMatching=468 QueryTransformer=31 QueryRouter=1093 ResultsMixer=0 LayoutsSelector=0 QueryLogging=15

Search Search Component ajkph HighMicrosoft.Ceres.SearchCore.Query.MarsLookupComponent.LookupService.QueryClient.QueryExecutor: ExecuteQuery timings for correlation: dad1789c-7d1d-8016-3b36-b562c4fdae55, task dispatch 0 ms, blocked waiting 254 ms, total hits: 84, with dupes: 92, bytes received: 33238, 1 tasks: (cell: I.0.0 at IndexComponent1, total task time: 79 ms, query push queue: 174 ms, send to index call duration: 20 ms, total in transit: 63 ms, index node execute queue: 0 ms, query lookup: 7 ms, docsum lookup: 9 ms)

Where’s this SQ&SS you speak of?

Search Query and Site Settings Service

Note: The SQ&SS will be automatically started on all servers with a QPC, but should not be enabled on any other server

Search Query and Site Settings ServiceGet-SPEnterpriseSearchQueryAndSiteSettingsServiceInstanceTypeName Status Id-------- ------ --Search Query and Site Settings Service Disabled 838d344e-cbe2-4045-9350-1f4d19c2413eSearch Query and Site Settings Service Disabled 0110417b-163b-41cb-9676-63cb80d65b52Search Query and Site Settings Service Disabled 6ec6ea28-c732-48dd-ab83-5458a577a300Search Query and Site Settings Service Disabled d301abbf-4a74-4897-933c-3a4041c40a0fSearch Query and Site Settings Service Online 65708c36-66fb-41b5-b622-6a97c4e0aa26

foreach ($pt in $SSA.EndPoints) { $pt.ListenUris.AbsoluteUri }http://jwspc14spc5:32843/cb4954e3e4ad40bdb0e21b1db290c668/SearchService.svchttps://jwspc14spc5:32844/cb4954e3e4ad40bdb0e21b1db290c668/SearchService.svc

Search Load Balancing IMS/Query Flows

User quer

y

Web01

WCFSen

dReque

st Searc

h

Ind

ex

Core

QPCQuery Processing

Component

JunoSendRequest MarsLookupOperatorSQ&SSSearch Query

& Site Settings

QPCQuery Processing

Component

SQ&SSSearch Query

& Site Settings

Farm Load Balancing WCF Service EndPoints

Farm: Load Balancing the SSA EndPoint

Search Load Balancing IMS/Query Flows

User quer

y

Web01WCFSendRequest

Searc

h

Ind

ex

Core

QPCQuery Processing

Component

JunoSendRequest

MarsLookupOperatorSQ&SSSearch Query

& Site Settings

QPCQuery Processing

Component

SQ&SSSearch Query

& Site Settings

Farm Load Balancing WCF Service EndPoints

Farm: Load Balancing the SSA EndPoint

Farm Topology: Error Handling

SharePoint Foundation Topology 8313 CriticalA failure was reported when trying to invoke a service application: EndpointFailure Process Name: OWSTIMER Process ID: 12345 AppDomain Name: DefaultDomain AppDomain ID: 1 Service Application Uri: urn:schemas-microsoft-com:sharepoint:service:144f1459cc294525bf691… #authority=urn:uuid…&authority=https://someserver:32844/Topology/topology.svcActive Endpoints: 2 #the total number of known EndPoints for this Service Application Failed Endpoints:1 Affected Endpoint: http://qpcServer:32843/-GUID-of-the-SSA-/SearchService.svc

User quer

y

Web01

WCFSendRequest

SQ&SSSearch Query

& Site Settings

SQ&SSSearch Query

& Site Settings

SQ&SSSearch Query

& Site Settings

The Farm takes this failed EndPoint out of rotation for 10

min

The impact of Alternate Access Mappings

Auto-magically mapped to current zone

User quer

y

Web01 Query Results:

• http://usersUrl/foo/DispForm.aspx?ID=3• http://usersUrl/TPSReports/

report1234.pdf• http://usersUrl/Memos/reminder_03-

2014.txt

http://usersUrl/results.aspx?k=tps

Auto-magically mapped to current zone

User quer

y

Web01 Query Results:

• http://non-default-url/foo/DispForm.aspx?ID=3• http://non-default-url/TPSReports/

report1234.pdf• http://non-default-url/Memos/reminder_03-

2014.txt

http://non-default-url/results.aspx?k=tps

Web App B

Web App A

Query Results:• http://morefoo/foo/DispForm.aspx?ID=3• http://morefoo/TPSReports/report1234.pdf • http://morefoo/Memos/reminder_03-2014.txt

http://spdemo/results.aspx?k=tps

Intranethttp://spdemo

Defaulthttp://sptest

AlternateAccess

MappingsIntranethttp://morefoo

Defaulthttp://foo

Queries across Web Apps use same zone

Demo 1. Test query from Search Center2. Test contextual query (e.g.

“This Site”)

Search Index

Items in Index

Why this does not work…

OriginalPath(string): http://non-default-url/Docs/foo.pptx

Path(string): http://non-default-url/Docs/foo.pptx

SitePath(string): http://non-default-url/Docs/foo.pptx

ParentLink(string): http://non-default-url/Docs/AllItems.aspx

ServerRedirectedUrl(string):http://non-default-url/_layouts/PowerPoint.aspx

SPSiteURL(string):http://non-default-url

QPCQuery Processing

Component

Query comes in via http://non-default-url

Look for content with root

http://usersurl

Demo Break it by manually swapping AAMs

Search Index

Items in Index

Server Name Mappings are not intended for SharePoint Content

OriginalPath(string): http://usersUrl/Docs/foo.pptx

Path(string): http://usersUrl/Docs/foo.pptx

SitePath(string): http://usersUrl/Docs/foo.pptx

ParentLink(string): http://nonDefaultURL/Docs/AllItems.aspx

ServerRedirectedUrl(string):http://nonDefaultURL/_layouts/PowerPoint.aspx

SPSiteURL(string):http://nonDefaultUrl

OriginalPath(string): http://usersUrl/Docs/foo.pptx

Path(string): http://usersUrl/Docs/foo.pptx

SitePath(string): http://usersUrl/Docs/foo.pptx

ParentLink(string): http://non-Default-URL/Docs/AllItems.aspx

ServerRedirectedUrl(string):http://non-Default-URL/_layouts/PowerPoint.aspx

SPSiteURL(string):http://nonDefaultUrl

MPs have inconsistent root URLs when using

Server Name Mappings

Five key takeaways from this session…Application Server Administration Service Timer JobSynchronizes search instances across the farm

Crawl state lives in the databaseMore precise state can be found in the MSSCrawlHistory/MSSCrawlComponentsState tables

Follow a crawled document via ULSUse VerboseEx ULS to track from Crawler to CPC to Index

Follow a query via ULSUse Merge-SPLogfile, Query starts at the Web Front End

Crawl the default zoneDon’t use server name mappings for SharePoint content

See you at the Search booths & Search tables at Ask the Experts WED @6:15pm!

Session Session Room Time

Develop Advanced Search-Driven SharePoint 2013 Apps SPC402 Palazzo I, J Tue 1:45pm

Best practices for Hybrid Search deployments SPC306 Veronese 2401 Tue 5:00pm

SharePoint 2013 Search Analytics SPC340 Palazzo M, N Wed 9:00am

How to manage and troubleshoot Search: A practical guide SPC375 Veronese 2401Wed 10:45am

6 Proven Steps to Get the Best Out of Search in SharePoint 2013 SPC265 Delphino 4001 Wed 1:45pm

Best practices for Information Architecture and Enterprise Search SPC207 Veronese 2401 Wed 1:45pm

Search content enrichment and extensibility in SharePoint 2013 SCP414 Palazzo K, L Wed 1:45pm

Customizing Search experiences with Azure Hosted Data and Bing Maps SPC321 Veronese 2401 Wed 3:15pm

Futuristic Search applications using Kinect and Yammer! SPC405 Palazzo M, N Wed 3:15pm

Search architecture and sizing in SharePoint 2013 SPC336 Titian 2201 Wed 5:00pm

Effective Search deployment and operations in SharePoint 2013 SPC360 Veronese 2401 Thu 9:00am

SharePoint 2013 Search display templates and query rules SPC322 Palazzo M, N Thu 9:00am

Managing Search Relevance in SharePoint 2013 and O365 SPC382 Veronese 2401 Thu 12:00pm

Searc

h R

ela

ted S

ess

ion

s

MySPCSponsored by

connect. reimagine. transform.

Evaluate sessionson MySPC using yourlaptop or mobile device:myspc.sharepointconference.com

© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.