indexed content migrate content create links content for future indexing
TRANSCRIPT
Making enterprise search successful at EYEd DaleSearch Services ManagerEY
SPC278
Pat VeldermanConsultantMicrosoft
About this sessionObjectives Share a basic search analysis approachLearn the elements of search quality monitoring and improvementUnderstand how search fits in EY’s hybrid environment
TakeawaysSearch quality can be measured and improved.Search installation is the start – not the end.Simple steps to improve search quality in your company
EY: a global organization175,000 people in 140 countries
Core services of audit, tax, transaction and advisory
Diverse business units
Hundreds of communities
Constant change and growth
Mobile and remote
Our enterprise search visionTo provide a single, cross-organization search capability that enables practitioners to find the right information at the right time
This is critical to improving the way we deploy, share and reuse knowledge and collaborate with each other when serving clients.
We will achieve our vision through six key driversThe right content is available in the index.Content is optimized for search.The search algorithm is measured and tuned.The search user interface is effective.Search understands user’s needs and behavior.Search is consistent across the organization’s tools.
Starting search analysis — when do you have a problem?Search use trending flat or downward over timeAbandoned queries trending flat or upward over timeZero-hit queries at 10% or higherNo trending in top query reportAnecdotal reports from users of search failures
Starting search analysis — how can you fix the problem?Start simpleLook at the search logsExamine the top queriesFocus on the queries where the results are poor
Make changesExpand indexAdjust contentCreate new content
Keep repeating
Applying the drivers at EY
The right content is available in the index.
Content is optimized
for search.
Search log analysis — core questionsWhat are people looking for?Are they finding what they need?If they aren’t, what can we do to help?
What are people looking for?
What are people looking for?Approximately 3,000 terms showed up consistently every month over a 12-month period.Of these, 175 consistently ranked in the top 500.
34%
8%
3%
19%
21%
15% Service delivery — related to services, service lines or industries
Engagement management
Business development
Client administration — including travel, expenses and technology
Tools
Personal administration — includes benefits and development
Why aren’t they finding what they need?Not all content sources are included in the index.Content is not optimized for search.There are content gaps.Search queries are poorly constructed.
Improving results — expand the indexWe used subject matter resources to identify the “best” links for our search results.Migrated some content to an indexed sourceCreated links in indexed sources with carefully selected keywordsDeveloped a prioritized list of sources to be considered for indexing
Indexed content
Migrate content
Create links
Content for future indexing
Improving results — best bets
“EY professionals recommend,” or best bets, is an alternative way to get to content when the source isn’t indexed.Every month, we evaluate the top 100 queries and ask our subject matter resources to recommend the best sources.We promote the process to encourage content owners to proactively submit keywords with their content.
Improving results — educating contributorsIdentify content submittersConducted 1-hour search engine optimization training Developed quick reference guides
Publish monthly lists of popular keywords Drives use of common terminology
Promote search team Assists with troublesome queries
Improving resultsExpand your indexIndex the right content sourcesCreate best bets to point to content you cannot index
Optimize your contentTeach the creators how to make their content appear higherUse logs to find content that needs to be created
Teach the usersLeverage communications, newsletters, team communications
The search algorithm is measured and tuned.
Measuring relevanceMeasuring the relevance of search results often requires a subjective, user-based evaluation of a query and its results.This takes significant effort.We can automate this process as much as possible, though the approach still requires regular updating to keep current.
One way to measure relevanceIdentify “ideal” documents or results for a given keyword searchUse your users; these documents represent the best possible result in our experts’ opinion.
Measure the search against this idealConduct the keyword search against your content and count how many ideal documents are found and where
This test can be used to gather three metrics of relevancy:Precision, which measures how exact the result set isRecall, which measures how complete the result set isMean reciprocal rank, which measures the position of the best document within the result set
Example — 0% recall, 0% precisionRelevant documents Search results
Example — 50% recall, 50% precisionRelevant documents Search results
Example — 100% recall, 50% precisionRelevant documents Search results
Example — 50% recall, 100% precisionRelevant documents Search results
Results
Across top 10 results
Month Precision RecallMean
reciprocal rank (MRR)
December 2012 0.086 0.130 0.488
March 2013 0.071 0.057 0.437 April 2013 0.204 0.140 0.490
Precision Recall MRR0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.0860.130
0.488
0.071 0.057
0.437
0.204
0.140
0.490
Relevancy metrics across top 10 results
Dec'12 Mar'13 Apr'13
EY’s hybrid environment
Relevant IT factorsMajor, concurrent transformationsUpgrading Windows XP to Windows 7Consolidating data centers Moving from Lotus platforms (Notes, Connections, Quickr and SameTime) to Microsoft (Outlook, Lync and SharePoint)
Moving from customized applications to standard products
Focus on information security and risk management
Journey to the cloud for intranet content2011: selected SharePoint 2010 with FAST search
2012: changed target to SharePoint 2013
2012: selected hybrid cloud with O365-D“Cloud first” hosting strategyOn-premises hosting for applications not appropriate for the cloud and for content which must reside within the country
Hybrid intranet – release 1
Hybrid intranet – release 2
Hybrid intranet enterprise search
Search
Crawl
Full intranet search
Remote
Sources
Search
Full extranet search
Remote
Source
Search
Balancing control and flexibility
Search teamMonitor qualityTest with full index
Information securityCannot move production dataCannot simulate users
OperationsCannot test in production environmentCannot touch production servers
Change
managementMust test all changesMitigate risks
BusinessQuality resultsQuick changes
Search components to manageThe user interface
Result pages:RefinersResult presentationsPreview
The search engineCrawling and document
processingIndexingReturn results to:
Search centersWeb partsApplicationsSharePoint
Enterprise search site has seven pages
Result pages (verticals)
Other pages
Everything
People
Conversations
Sites
Preferences
Advanced search input
Search input
Enterprise search site collection
Managing changes to production
Enterprise search center
Enterprise search validation site
Enterprise search development sites
Service request
Site admin: search teamSite admin: search team Site admin: operations
Changerequest
LiveQuality assurance
Development
(scripts) (scripts)
Hybrid search experience summarySearch advantagesDisplays single relevance-ranked result setCrawls large on-premises content sourcesAbility to extend result page interactivity with BA Insight and custom XML previewEnriches documents as they are indexedDirectly manages the crawler and content sourcesProvides search service to non-SharePoint applications
DisadvantagesComplex; requires greater coordinationForced update scheduleLimitations on social featuresStill requires remote result sources to integrate in-country farm content
Making your enterprise search successful
Step 1: analyze your current queriesAnecdotes have limited value.Review standard reports:
Users tell you what they want with each search … … are you listening?
Successful?
Not successful?
Step 2: fix the top queries
Fix the “head”Analyze up to 100 queriesExecute and fix failing queriesLarge impact from few corrections
Fix top unsuccessful queriesNo result queriesAbandoned queries
Head Tail
Frequency of unique search phrases
Nu
mb
er
of
searc
hes
Step 3: report the impact
Use a simple graphTotal queriesNo-result and abandoned queries
Identify your actions
US holidays Added people search
No results
Weekly query volume
Start small; demonstrate value; expandInitial effortAllocate 4 hours per weekSkills: technical person with knowledge of your contentCreate baseline reports and fix 10 to 20 queriesShare the impact report with content owners and business units
Expand to one dedicated search specialistSkills: analysis, content knowledge, organization skills and technical conversancyImprove additional queriesReport regularlyEngage content owners
Expand into a search management program
In summaryObjectives Share a basic search analysis approachLearn the elements of search quality monitoring and improvementUnderstand how search fits in EY’s hybrid environment
TakeawaysSearch quality can be measured and improved.Search installation is the start – not the end.Simple steps to improve search quality in your company
Questions?
Contact informationEd [email protected]@EdDalewww.linkedin.com/in/eddale/
EY | Assurance | Tax | Transactions | Advisory
About EYEY is a global leader in assurance, tax, transaction andadvisory services. The insights and quality services wedeliver help build trust and confidence in the capitalmarkets and in economies the world over. We developoutstanding leaders who team to deliver on ourpromises to all of our stakeholders. In so doing, we playa critical role in building a better working world for ourpeople, for our clients and for our communities.
EY refers to the global organization, and may refer toone or more, of the member firms of Ernst & YoungGlobal Limited, each of which is a separate legal entity.Ernst & Young Global Limited, a UK company limited byguarantee, does not provide services to clients. For moreinformation about our organization, please visit ey.com.
Ernst & Young LLP is a client-serving member firm ofErnst & Young Global Limited operating in the US.
© 2014 Ernst & Young LLPAll Rights Reserved.
ED None
This material has been prepared for general informational purposes only and is not intended to be relied upon as accounting, tax, or other professional advice. Please refer to your advisors for specific advice.
The views expressed herein are those of the presenter and do not necessarily reflect those of the EY organization or any member firm thereof.
MySPCSponsored by
connect. reimagine. transform.
Evaluate sessionson MySPC using yourlaptop or mobile device:myspc.sharepointconference.com
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.