virtual techdaysINDIA │ 18-20 august 2010
FAST Search for SharePoint 2010Allirajan Ramachandran │ Technology Specialist, Microsoft Corp
Cut Costs with a Unified Infrastructure
Rapidly Respond to Business Needs
Deliver the Best Productivity Experience
Microsoft SharePoint 2010The Business Collaboration Platform for the Enterprise and the Internet
Communities
Search
Sites
Composites
ContentInsights
One platform, Multiple Service Opportunities
User Interface is visual and actionableVisual and conversational interaction with precise control
Built on SharePoint Search CenterLeverages all of innovations in SharePointOpen Web Parts, Federation, query suggestions, related queries, Did you mean?
Visual results connects users with contentThumbnails for Word and PowerPointVisual Best Bets highlight premium content Preview in browser without leaving the results
Deep Refinement
Thumbnails
Previews
Sort on any field
Similar Results
Visual Best BetsVisual cues to highlight essential content
Built on SharePoint KeywordsMatches keywords and synonyms that are contextually relevant to users. Include banners, videos, external websites.
Easy and quick to setupPoint and click setup for site admins. Set and forget with content expiration dates . Web Parts allow for easy page customization
Visual Notification
Web Part Flexibility
Deep RefinementEnables precise control of results
Contains exact countsLeads to discovering non-obvious relationships, key data trends, and deep analysis of your content
Enables conversational experience You will never miss any content; enabling better findability and exploration across the entire result set
Exact Counts
Sorted by frequency
Provides a sorted viewEach refiner is sorted by frequency, from highest to lowest, indicating the importance of each term
Customize search to meet your business needs
Key ingredients to a great customized search experience
Search in the language of your business
Deliver results that are contextually relevant
Tune relevancy to meet diverse needs
Process content with advanced linguistics
Customize the user experience to build engaging applications
Type Doc ID Title Author Date Size Keywords Companies Expertise Industry ... Body Text
123 Sales Forec… John Doe 2010-04-15 386 KB sales; pipe… Contoso; Tail.. Strategic Con.. Financial Ser… … IT Market Gr…
456 … … … … … … … … … …
963 … … … … … … … … … …
Map metadata to Managed Properties Automatic association of metadata to content
Expertise
Strategic Consulting
Market Analysis
IT Implementation
…
Industry
Financial Services
Manufacturing
Technology
...
Crawled Properties
Managed PropertiesIndex Schema:
Companies
Contoso
Tailspin Toys
Woodgrove Bank
…
Mapped in Central Admin Or PowerShell
Crawled Properties Standard document metadata discovered by the crawler or extracted from the full text by the FAST Content Processing Pipeline
Managed Properties Map one or more Crawled Properties to a single field. Enables sorting, refinement, relevance tuning and fielded searching
Title
Sales Forecast
Any data can befound and mapped
What can I do with a Managed Property?Metadata quality is critical to a good search experience
Precise hit counts in deep refiners are
computed across the whole result set.
And many more…
Concepts
Products
Companies
File Formats ,
Metadata is also used for relevancy tuning, multi-
level sorting and advanced search
Enables deep refinementMakes search conversational, guiding users to navigate and refine, while summarizing the results that are found
Enables precision relevancyManaged properties are also used for relevancy tuning & ranking, multi-level sorting, advanced (or fielded) search
How does it work?
Built on a SharePoint List or custom extractorEdit the Search Center Results PageModify the shared web part by adding tags to the refinement panel XMLCreate your own labelsSave and Publish
Custom Collections
Add refiners to user interface
Context mattersUsers need to access multiple types of content
HRLegal
Finance
Depends on role, location, responsibility and task. This can change day to day, or hour to hour.
Marketing Sales R&DCustomerSupport
ProfessionalServices
ManufacturingOperations
. . .
Ente
rpri
seC
onte
nt
Deliver results that are contextually relevantwith search that can understands your business and role
”What should I know about selling ERP?”
- Alan Brewer, Sales Lead
”What should I know about implementing ERP?”
- Renee Lo, Consultant
Role-specific relevance
Business drivenrefinement
Targeted Best Bets / Visual
Best Bets
Tune relevancy to meet diverse needsA flexible solution for your organizations, groups and individuals
Optimize Relevancy for broad intranet use
Query results with the default relevancy
”I want to know about my customer Woodgrove Bank and customers in Financial Services"
- Alan Brewer, Sales Lead
New Default Sorting
Promotes relevantResults
Quickly tailor relevancy models Deliver the right results to the right people by creating new Rank Profiles
”I want to get right to the technical documents"
- Renee Lo, Consultant
Documentation, RFPs and SOWs are now promoted Same
Results different order Users can select rank profiles in the sort by box or create their own default views by modifying the web part
Rank ProfilesTune relevancy without impacting the default algorithm
Quality Also known as static rank, consists of multiple managed properties including site, URL depth (preference for shorter URLs), and relative importance of links to this document.
Authority Applies when the query word falls in the link or anchor text.
Query Authority
Maps the popularity of a document, or the click-through rate when documents are clicked as a result of a query
Freshness Increases the relevancy if a document was recently created or modified, based on the last modified property.
Proximity Applies to where query terms fall and how close they are to each other within a document
Context Increases the rank of a document if the query term is a managed property associated with that document
Managed Property
Effects relevancy when a managed property contains a specific value, such as Woodgrove Bank or Financial Services
Out of the box relevancyTuned for great general productivity experience, relevancy improves with click-throughs and link text analysis.
Extend the default algorithmsCreate new default relevancy models. Blend static and dynamic ranking parameters to instantly improve search results.
How to create a Rank ProfileIT Pros are empowered to create new profiles quickly
Rank Profiles created in PowerShell by extending the default relevancy algorithm…
… and are exposed in the user interface by modifying the
sorting web part.
Breaks down content to the smallest addressable chunks to build meaningUnderstands file encoding, data formats, and written languages Supports 400+ file formats, 80+ languages
Process content with advanced linguisticsAutomatic and detailed analysis creates a great search experience
Map Crawled Properties
Maps all of the metadata that was discovered by the various pipeline stages
Web Link Analysis Analyzes documents for hyperlinks extracting anchor text which reinforces the authority ranking of a document.
Document Vector Creates a unique representation of a document that reflects important terms and frequency of occurrence. Used to find similar documents.
Date and Time Normalization
Converts dates and times to a standard representation, to handle locale specific representations. For example, knows that 14-Mar-10 is equivalent March 14, 2010.
Entity Extraction Finds terms in the content and maps them to predefined categories. Out of the box support for People, Companies and Locations, but can be extended to any category.
Lemmatization Finds the root of a word for a given language. For English it maps run, runs, running and ran back to a single lemma. Understands language specific grammar and context.
Tokenization Apply the language specific rules for identifying words, concepts, idioms and phrases. Also applies custom word breakers found in part numbers or telephone numbers.
Language Encoding and
Detection
Identifies the native written language and locale specific encoding so that the proper dictionaries can be used by the tokenization and lemmatization stages
Format Conversion
Extracts plain text from multiple file formats, encodings, and applications
FAST Content Processing Pipeline
FQL provides a robust and expressive query language
Wildcard support - *, ?Numeric Data types (Integer, Float, Decimal, Datetime)
OperatorsDirect field access (e.g., title:othello, author:shakespeare)Numeric (COUNT, RANGE, <, <=, >, >=)Boolean (AND, OR, ANY, NOT)Rank (RANK, XRANK)Proximity (NEAR, ONEAR)
String (operator support for strings) Boundary (starts-with, ends-with, equals)Filter
Robust query languageUse FAST Query Language (FQL) for precise query development
FAST Search Server ArchitectureCommon platform scales with your business need
One search platformwith a single UI framework, common APIs and developer experiences across products
Common connectorswith a consistent framework for capturing and indexing your information
Unified administrationwhich allows you to manage FAST Search and SharePoint in a centralized admin console
User Experience
SharePointSearch
FAST Search for SharePoint
Information
FAST Search Server 2010Summary of architectural components
Custom Front-End
OpenSearch or Other Sources
SharePoint Front-end
People Search
Qu
ery
Obje
ct M
od
el
Query and
Result Processin
g
Security AccessModule
SearchCore
Indexing
Federation Object Model
Query Web Service
AdvancedContent Processin
g
Linguistics
WebLink
Analysis
Connectors
• Web Crawler
• JDBC
Connectors
• SharePoint• File
Traverser• Web • BDC• Exchange• Notes• Documentu
m
Microsoft System Center Operations Manager
Monitoring Services
Administration and Schema Object Model
Site Collection Level Admin UI
• Keyword Management• User Context
Management• Site
Promotion/Demotion
PowerShell
• Schema configuration• Admin configuration• Deployment
configuration
Central Administration UI • Property mapping• Property extraction• Spell-checking
FAST Server(s)
SharePoint Server(s)
Other Server(s)
Content
Secure, unified access to informationIndex or federate with content, applications, and services
Access content securelywith a full-featured content crawler and both index- and query-time trimming of results
Federate queriesusing OpenSearch to quickly access existing search indexes and online information services
Extend your reachusing Business Connectivity Services to access your content sources and business applications
OpenSearch Federation
Indexing Connectors
EnterpriseContent
Business Application
s
Information Services
User Experience
Search Index
Websites and Intranet
File Shares
IBM Lotus Notes
EMC Documentum
SharePoint Server
Exchange Public Folders
AD & LDAP Profiles
Etc …
Connect to all of your enterprise contentExtend your reach with Business Connectivity Services
Build custom connectorsUse SharePoint Designer to configure data model and connect to SharePoint. Connect to WCF services, or create your own .NET assembly connector with Visual Studio.
Quickly connect to contentUse a consistent framework to quickly connect both inside and outside of SharePoint including content management systems, web services, databases, and line of business systems
Simplified, powerful administrationA high-end enterprise search solution that’s easy to deploy and manage
Deploy easilyusing wizard-driven installation, a topology designer, and native support for 64-bit virtualization
Manage efficiently with full support for Microsoft System Center and PowerShell scripting to automate tasks
Streamline administrationwith a simplified admin console that helps you manage search services across your enterprise
FAST Search for SharePoint Scaleout
Content Volume
Query Volume
Scale-out multiple “dimensions”
Query VolumeContent VolumeIndexing freshness
Redundancy optionsSearchIndexing
Performance targets*30M Docs/node50 QPS/node35 docs/sec
*Depends on content and hardware specifics
Search and Indexing
Crawling and Content
Processing
Query and Result
Processing
Back-end with extreme and flexible scale out options
No theoretical upper bounds!
virtual techdaysINDIA │ 18-20 august 2010
RESOURCES
www.sharepoint.microsoft.com
www.microsoft.com/enterprisesearch
Channel9.msdn.com/posts/matthijs/fast-search-for-sharepoint-2010-capabilities-deep-dive
virtual techdaysTHANKS│18-20 august 2010
Thank You