osp311. © copyright ba insight page 3 about the speaker: jeff fried
TRANSCRIPT
Building Search-Driven Applications with Microsoft FAST Search Server 2010 for SharePointJeff FriedCTO & VP EngineeringBA Insight
OSP311
© Copyright BA Insight Page 3
About The Speaker: Jeff Fried
CTO, BA InsightPreviously VP Advanced Solutions for FAST, then Microsoft technical product manager for FAST and SharePoint 2010 Search
AuthorProfessional Microsoft SearchProfessional SharePoint 2010 Development
Blog:http://www.ba-insight.net/Blogs/Sharepoint-Search-Expert/default.aspx
Twitter:http://twitter.com/jefffried
Based in Boston, MA, USA
Outline
Search-Based Applications are found everywhere
FAST Search for SharePoint and SharePoint 2010 provide a great platform and ecosystem
Custom and Packaged Search Applications are possible in ways they never were before
Case Study: Pharma R&D
R&D researched where it was spending its time, and discovered
56% of R&D time (human capital and budget) spent:
Duplicating existing research
It blew them away, until they reviewed why…
Top 3 reasons for 56% effort duplication:
1) Research done in separate groups• Seemingly unrelated research projects• Later in lifecycle (mfg, reg/test)
2) Data not accessible• Isolated content source• Restricted / limited access• Source not searchable• Special knowledge required
3) Data not linked• Various names/changes leave data disconnected• People not connected to data (experts)• Data managed in many unconnected systems
Case Study: Pharma R&D
1. Documentum Image2. SharePoint Doc3. Regulatory Record4. MEDLINE article
Multiple Sources One Search
Search: amgen 655
Relationships Discovered:Antibodies: mAbReceptors: DR5, IGF-1RLabs: Oncology 1People: David Chang
Recognizing Search Driven ApplicationsThey are everywhere
“How do I support theunique search needs of teams
and work that impact our business?”
To do so, you need a search platform that has• A deep understanding of your
information• Flexible relevance to meet
diverse needs• A customizable UX to increase
user efficiency
Sales: 360o Customer Insight
Services: Knowledge Browser
Marketing: Competitive Intelligence
Research & Development:Innovation Portal
Support:Call Center Advisor
Operations:Systems/Logistics Portal
Legal, HR, IT, Finance, ……
Search-Based Applications
Research Portal Unified View Customer Service Compliance
Analyst’s workbenchManagement AdviserInnovation Center
Voice of the CustomerLogistics CenterConsolidated Dashboard
Call CenterOnline ServiceSales Dashboard
Fraud CenterE-DiscoveryInfo Governance
Search Based Applications are found in every industry and every functionTraditionally, search vendors describe these as possibilities using their platforms;
but implementation costs have been >$1M
Packaged apps are now possible, leveraging the SharePoint ecosystem
demo
Search Driven Application
Contoso Consulting Demo
How would you create this?Content Crawling: bring in data from lots of places
OOB connectors to SharePoint (reports, account documents), shared files; CMS systems across multiple divisions of contoso
Crawl web intelligently for background content
Content processing: creating metadata
Names of projects, offerings, key concepts, clients, external experts
Industry terms & taxonomy (from external sources)
Synonyms for key concepts
OOB web parts configured for style
Federation, People Search, Search actions, Scopes
Custom Relevance Profile
Custom web parts for visual navigation Tab-like selectors, sliders, maps, and taxonomy-aware refiners
Build PowerPoint on the Fly Custom development using the OpenXML SDK
SharePoint workflows for act-on-selected-items
Federa
tion O
penSearc
hContent
Processor CrawlerIndexerQuery Processor
Search Center
Content
User Profiles
…
Fo
rma
tC
on
ve
rsio
n
La
ng
ua
ge
Det
ec
tio
n
En
tity
Ext
rac
tio
n
Le
mm
ati
zati
on
Ma
pp
er
…
Content Processing Pipeline
People
Searc
h
Index Partition
MetadataCreation
RelevanceControl
UserContext
IndexingConnectivity
User Experience
Federation
FAST Search for SharePointDesigned for Customization
FAST Search for SharePoint UI OOB
Built on SharePoint Search CenterOpen Web Parts, Federation, query suggestions, related queries, Did you mean?
Visual results Thumbnails for Word and PowerPointVisual Best BetsPreview in browser
Deep Federation
Thumbnails
Previews
Sorting
What can I do with a Managed Property?Metadata quality is critical to a good search experience
Precise hit counts in deep refiners are computed across
the whole result set.
And many more…
Concepts
Products
Companies
File Formats ,
Metadata is also used for relevancy tuning, multi-level sorting and advanced search
Enables deep refinementMakes search conversational, guiding users to navigate and refine, while summarizing the results that are found
Enables precision relevancyManaged properties are also used for relevancy tuning & ranking, multi-level sorting, advanced (or fielded) search
PowerShell
$crawledProps = Get-FASTSearchMetadataCrawledProperty -Name “crawled1”
$mp = New-FASTSearchMetadataManagedProperty -Name “managed1” -type $dataType
New-FASTSearchMetadataCrawledPropertyMapping -ManagedProperty $mp -CrawledProperty $crawled
TechNet Script Repository
Introducing the Processing Pipeline
Sequential stages perform specific tasks while ingesting contentBreaks down content to the smallest addressable chunks to build meaningUnderstands file encoding, data formats, and written languages Supports 400+ file formats, 80+ languages
Process your content to make it searchableNormalizes content so that a consistent relevancy model can be appliedIdentifies structured and unstructured metadata in your contentMaps document metadata to SharePoint Crawled Properties
A systematic approach to interpreting your content
Map Crawled Properties
Maps all of the metadata that was discovered by the various pipeline stages Web Link Analysis Analyzes documents for hyperlinks extracting anchor text which reinforces the authority ranking of a document.
Document Vector Creates a unique representation of a document that reflects important terms and frequency of occurrence. Used to find similar documents.
Date and Time Normalization
Converts dates and times to a standard representation, to handle locale specific representations. For example, knows that 14-Mar-10 is equivalent March 14, 2010.
Entity Extraction Finds terms in the content and maps them to predefined categories. Out of the box support for People, Companies and Locations, but can be extended to any category.
Lemmatization Finds the root of a word for a given language. For English it maps run, runs, running and ran back to a single lemma. Understands language specific grammar and context.
Tokenization Apply the language specific rules for identifying words, concepts, idioms and phrases. Also applies custom word breakers found in part numbers or telephone numbers.
Language Encoding and Detection
Identifies the native written language and locale specific encoding so that the proper dictionaries can be used by the tokenization and lemmatization stages
Format Conversion Extracts plain text from multiple file formats, encodings, and applications
Extending Pipeline capabilities
Configure Optional Processing StepsXML Properties mapperOffensive Content FilterVerbatim (wholeword) extractor
Use a dictionary for custom extraction
Pipeline ExtensibilityCalls external applications for custom item processing
Field CollapsingEntity Extraction
Straightforward way to add custom text analysis functionality
Add Custom ProcessingPipeline Extensibility is a specially defined stage that takes a set of crawled properties, as flat text as input and maps output to another crawled property
Sandboxed executionExecutable arguments and temporary files are automatically handled with timeouts.Runs just before the Crawled Property Mapper, providing accessibility within SharePoint
DEMO: Pipeline Extensibility
ConfigurationLogging and generating test dataDebuggingUseful included FAST tools
…
ExtensibilityMapperStandard
processing
Let’s make easier
Pipeline Extensibility example
PurposeCreate a new property based on value of an existing propertyEnable searching for a value being empty
ImplementationGet value of ows_ContentType using Linq for XMLSet new property mycontentcheck to either
hascontenttypenocontenttype
Tune RelevancyImprove accuracy and control with Rank Profiles
Quality Also known as static rank, consists of multiple managed properties including site, URL depth (preference for shorter URLs), and relative importance of links to this document.
Authority Rank given when the query word falls in the link or anchor text.
Query Authority Rank given when the query word falls within items that have been selected in previously executed queries.
Freshness Rank given for new content, based on last modified parameter.
Proximity Rank given depending on where query terms fall and how close they are to each other within a document
Context Rank given depending on where a query term falls. If the query term falls within a managed property, it is given an additional boost.
Field Boosts Any managed property can be used for an additional boost.
Rank Profiles are exposed by modifying the sorting web part.
Rank Profiles are made by combining multiple ranking elementsCreate custom ranking algorithms to combine multiple ranking properties
Rank Profiles created in PowerShell
demo
FAST Search for SharePoint
Platform Capabilities
Microsoft SharePoint 2010The Business Collaboration Platform for the Enterprise and the Internet
Deliver the Best Productivity Experience
Cut Costs with a Unified Infrastructure
Rapidly Respond to Business Needs
Communities
Search
Sites
Composites
ContentInsights
SharePoint as an Application Platform
Differentiated applications
Rich tools and platform
services
Manageable applications
Thriving ecosystem
Application LifecycleDeveloper Machine
Development
TestingF
5 D
eplo
y
Team Foundation Server
Check In
Staging
Automated Testing
Warm-Blooded User Testing
TFS Build Server
SharePoint Projects
SP2010 DLL’s
Build
Run Tests?
Fix Bugs (repeat as necessary)
NightlyBuildorContinuous Integration
Deploy Using PowerShell
Open/Close Bugs
WSP
Customizing Search
Configure Extend Create
Intranet SearchPeople Search
Site Search
Research PortalCase Management
Save Results to Excel file…..
IP Portfolio mgmtIntel/SurveillanceDrug Discovery
….
“Out of the Box” Search ExperienceA powerful baseline for customization
Refinement summarizes and narrows resultsQuery federation brings together results(FAST) Previews provide rich interactionQuery Suggestions guide user keywords(FAST) User Context provides a personal experience
Top Customization Scenarios
Modify the OOB End User Experience Add new Refinement category Show results from federated location Modify the look and feel of OOB end user
experience Enable sorting by custom metadata Add visual Best Bet for upcoming sales
event Configure different ranking for HR vs.
Engineering department
Create a new Search Verticals
Create new visual elements
Query & Result Pipeline Plug-ins
Query & Indexing Shims
Create new Search Driven Applications
Create new customer search experience Indexing content Define content for search Design search experience
Create new Audio/Video/Image search experience
Show Location refinement on Chart/Maps Show tags in tag cloud Enable export results to Spread Sheet Summarize Financial Information from
customers in Graphs Expand query terms based on synonyms
defined in Term Store Augment customer results with project
information Show popular customers/people inline with
search results Show people results from other sources
Show email results from personal mailbox on Exchange Server through the EWS
Index content from custom repositories like Documentum
Create content processing plug-ins to create new metadata
create a new customer page that shows: Customer Contact Details Customer Project Details Customer Contacts Internal Experts Customer related documents
Search 2010 ArchitectureThe platform for Search Customization
SearchWeb Parts
SharePoint Search Index
OpenSearch /Custom Source
SharePoint Indexer
Federation OM Web Service, RSS
FASTSearch Index
FAST Indexer
What’s New in 2010?Primary Search Web Parts now UnsealedFederation now a key Public OM layer
All Web Parts built on federationQuery alteration, custom Runtimes, blending results from multiple sources
Web Service / RSS Enhancements
FAST Search / SharePoint Search:Shared Web Parts, RSS, Web ServiceShared Federation OMIndex and Crawling SeparateFAST unique Content Processing Pipeline
SearchWeb Parts
Federation / Query OM Web Service, RSS
Content Processing
Example: Legal Application
Attorneys spend too much time looking for information and assessing the relevancy of search results
• Must download entire document and conduct manual review to determine relevancy
• No way to internally search documents & items
• Excessive burden to network resources for geo-distributed offices
• No way to quickly review, annotate , compile, and reuse relevant information
Longitude Search
Packaged Application
demo
Search: Find and Explore
Penicillin
LSD
Uranus
Viagra
Safety glass
Infrared radiation
Microwave ovens
Inkjet printers
Corn Flakes
Chocolate chip cookies
Research Portal example
Similar Patterns across IndustriesComponent-based
Example Mockup
Dictionaries/Taxonomies3rd party contentWorkflowUI (Complete)File types (preview/assembly)App IntegrationConnectorsFederationContent CorrelationMobileMultiMedia ClassificationApp IntegrationUI (Web Parts & Templates)WorkflowAnalytics/BIAgent/AlertingExpertise Build/FindAssociative SearchAssemblyPreviewCore Connectors
Vertical /
MicroVertical
Components
(also available seperately)
Search Platform
SharePoint (Content, UI, Workflow, BI, Social, …)
Multiple Search Centers
The Search Platform Concept Content Integration to enable applications
Legacy Data RDBMS, Sharepoint, Salesforce
Portals RSS Feeds,Data Streams
DMS, CMS, FilesApplications Email Rich Media XML Internet
FAST Search PlatformSharePoint 2010
Information Integration
Site Search; Transactions;
Media Content
Intranet Search; Knowledge Management; CRM; Fraud Detection; Risk & Compliance
Longitude Connectors
Customer Facing Apps Enterprise-Facing AppsSOA-enabled
Services
Search-Based Applications
Research Portal Unified View Customer Service Compliance
Analyst’s workbenchManagement AdviserInnovation Center
Voice of the CustomerLogistics CenterConsolidated Dashboard
Call CenterOnline ServiceSales Dashboard
Fraud CenterE-DiscoveryInfo Governance
Search Based Applications are found in every industry and every functionTraditionally, search vendors describe these as possibilities using their platforms;
but implementation costs have been >$1M
Packaged apps are now possible, leveraging the SharePoint ecosystem
Tips for successful FAST projects
Start early with OOB experienceStand up FAST, show it to users
Keep an active staging systemFull scale, with production contentSearch exposes/’audits’ security issues!
Grow incrementally & continuallyAdditional content sourcesDesign and feature changes and additionsContent grooming / gardeningUse regular rhythm to debug/tune
Don’t be afraid to customizeBuy what you can, build what you can’t
Establish success early, build onMany search apps will emerge
Outline
Search-Based Applications are found everywhere
FAST Search for SharePoint and SharePoint 2010 provide a great platform and ecosystem
Custom and Packaged Search Applications are possible in ways they never were before
DEV Track Resources
http://www.microsoft.com/visualstudio http://www.microsoft.com/visualstudio/en-us/lightswitch http://www.microsoft.com/expression/http://blogs.msdn.com/b/somasegar/http://blogs.msdn.com/b/bharry/http://www.microsoft.com/sqlserver/en/us/default.aspxhttp://www.facebook.com/visualstudio
Resources
www.microsoft.com/teched
Sessions On-Demand & Community Microsoft Certification & Training Resources
Resources for IT Professionals Resources for Developers
www.microsoft.com/learning
http://microsoft.com/technet http://microsoft.com/msdn
Learning
http://northamerica.msteched.com
Connect. Share. Discuss.
Complete an evaluation on CommNet and enter to win!
Scan the Tag to evaluate this session now on myTech•Ed Mobile
© 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to
be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS
PRESENTATION.