examiness hints and tips from the trenches

Ismail MayatSenior Web Developer

@ The Cogworks

Examiness

Hints and tips from the trenches

What this talk is not

• How to install• How to configure

What we will cover• Tools to help you• Hints and tips regarding indexing• GatheringNodeData event is your friend!• Indexing media (pdf,word etc)• Deep in the bowels with DocumentWriting event• Search highlighting• Deployment to staging / production environments• Faceting (Not exactly examine but still useful)• Food for thought• Questions and answers

Tools to help you

Tools to help you“Use the source Luke!”

http://code.google.com/p/luke/

Tools to help you

• http://luke.codeplex.com/ (.net port)• Subset of common features present• Scripting with Rhino missing etc

Using Luke

• Writing out generated queries to test in lukevar criteria = searcher.CreateSearchCriteria(IndexTypes.Content);

IBooleanOperation query = criteria.NodeTypeAlias("NewsItem");

query = query.Not().Field("umbracoNaviHide", 1.ToString());

var results = searcher.Search(query.Compile());criteria.ToString();

Generates the following querySearchIndexType: content, LuceneQuery: +(+__NodeTypeAlias:newsitem -umbracoNaviHide:1) +__IndexType:content

Tools to help youhttp://our.umbraco.org/projects/developer-tools/examine-dashboard

GatheringNode Data

• Examine has rich event system• In all my implementations I have used

GatheringNode– Merge into one contents field– Searching on path– Adding nodeTypeAlias field into pdf index

GatheringNode DataMerge into contents field

• Example query

var query = searchCriteria.Field("nodeName","hello").Or().Field("metaTitle","hello").Field("metaDescription","hello").Compile();

GatheringNode DataMerge to contents field

public class ExamineEvents:ApplicationBase { public ExamineEvents() {

ExamineManager.Instance.IndexProviderCollection[Constants.ATGMainIndexerName].GatheringNodeData += ATGMainExamineEvents_GatheringNodeData;

}

void ATGMainExamineEvents_GatheringNodeData(object sender, IndexingNodeDataEventArgs e) { AddToContentsField(e);

}

private void AddToContentsField(IndexingNodeDataEventArgs e) {

var fields = e.Fields; var combinedFields = new StringBuilder();

foreach (var keyValuePair in fields) { combinedFields.AppendLine(keyValuePair.Value);

}e.Fields.Add("contents", combinedFields.ToString());

} }

GatheringNode DataMerge to contents field

• Query now looks likequery.Field(“contents”,”hello”)

• Adding new fields is just case of rebuild index

GatheringNode DataCreating a searchable path

• Path is in index as 1,1056,1078 not tokenised• Add new field with , replaced with space

GatheringNode Data

• How to query when no value e.g sql query like select where value=‘’

• Select all• Cannot do query like this in Examine / Lucene• However can use GatheringNode data event

to inject in some arbitrary value then query on that.

• E.g. field noData_Title value 1

GatheringNode Data

• Re Indexing errors• MNTP field referencing a node that no longer

exists• Use try catch and log the offending node

Document writing event• You need lower level Lucene access• E.g. boosting a field• What is boosting? Not all documents are equal you need to artificially give

higher ranking to certain documents . When sort by is just not enough e.g.

– Person doc type. If they have important title they need to appear at top of person search list

– Boost documents by age. Penalize older documents useful for news and business documents.

– Boost based on unique views (would need to know up front also base on time slots e.g last month, last week)

– Documents with more likes (custom like functionality)– Tagging using XFS Term selector with weighting http

://our.umbraco.org/projects/website-utilities/xfs-term-selector

Document writing eventvar indexer = (UmbracoContentIndexer)ExamineManager.Instance.IndexProviderCollection[Constants.ATGMDirectoryIndexerName]; indexer.DocumentWriting += indexer_DocumentWriting;

void indexer_DocumentWriting(object sender, Examine.LuceneEngine.DocumentWritingEventArgs e) {

var title= e.Document.GetField("title");

if(title==“Partner”){ e.Document.SetBoost(1.5f);

} }

Indexing media• Pdf indexer. Only indexes pdf content.• CogUmbracoExamineMediaIndexer (Available as package on our)

– Uses apache tika. Indexes content and any associated meta data– XML and derived formats– Microsoft Office document formats– OpenDocument Format– Portable Document Format– Electronic Publication Format– Rich Text Format– Compression and packaging formats– Text formats– Audio formats (MP3 etc)– Image formats– Video formats– Java class files and archives– The mbox format

Search highlighting

• Lucene contrib package Highlighter.net• Highlights occurrences of your search term in

search results summary fragment.• Wiki on our

http://our.umbraco.org/wiki/how-tos/how-to-highlight-text-in-examine-search-results

Deployment to staging / production environments

• Cannot copy index• Can check in but could corrupt• Selenium with ashx to rebuild index

Deployment to staging / production environments

public class RebuildIndexes : IHttpHandler { readonly List<string> indexes = new List<string> { "ATGIndexer", "InternalIndexer", "directoryIndexer" }; public void ProcessRequest(HttpContext context) { context.Response.ContentType = "text/plain"; try { if(string.IsNullOrEmpty(context.Request.QueryString["index"])) { foreach (var index in indexes) { ExamineManager.Instance.IndexProviderCollection[index].RebuildIndex(); } } else { ExamineManager.Instance.IndexProviderCollection[context.Request.QueryString["index"]].RebuildIndex(); } context.Response.Write("done"); } catch(Exception ex) { context.Response.Write(ex.ToString()); } }

public bool IsReusable { get { return false; } } }

Deployment to staging / production environments

[SetUp] public void SetupTest() { selenium = new DefaultSelenium("localhost", 4444, "*chrome", "http://mydevsite"); selenium.Start(); _verificationErrors = new StringBuilder(); }

[Test] public void RebuildIndex() { //not proper test but a hack to get indexes rebuilt after a deployment try { selenium.Open("/umbraco/webservices/RebuildIndexes.ashx");

} catch (SeleniumException se) { if (!se.Message.StartsWith("Timed out")) { throw; } } catch (AssertionException e) { _verificationErrors.Append(e.Message); } }

Faceting• Faceted search, also called faceted navigation or faceted

browsing, is a technique for accessing information organized according to a faceted classification system, allowing users to explore a collection of information by applying multiple filters

• Amazon, LinkedIn http://www.linkedin.com/search/fpsearch?type=people&keywords=umbraco&pplSearchOrigin=GLHD&pageKey=member-home&search=Search

• LinkedIn uses Bobo browser. Written in java it has been ported to .net http://bobo.codeplex.com/

• Demo is SimpleFacetHandler others are available e.g RangeFacet,PathFacet, GetFacet

Food for thought• Using the index as object db ala RavenDb• Scenario: You have nodes with large number of multi tree node pickers used as look ups

Index as object db

Index as object db

Index as object db

Food for thought

• In index node ids are stored as CSV list if MNTP set to csv.

• Use GatheringNodeData event to do lookups create a POCO with lookup data, serialise POCO to JSON and store that in index.

• Advantage: Instant lookup all data ready to use• Disadvantage: Need to keep up with lookup changes.

E.g. If Country code changes then you would need to lookup code already in use and update.

• Nice approach if lookup data is fairly static

Food for thought

• POCO hydration using activelucenenet ala USiteBuilder

• Create pocos and decorate with attributes public class Product { [LuceneField(“sku")] public string Sku { get; set; }

[LuceneField(“productName")] public string ProductName { get; set; } }

Food for thought

var luceneProductDoc = GetItFromLucene(1234);var product = LuceneMediator<Product>.ToRecord(luceneProductDoc );

Would need to use Lucene directly as there is a no way of getting the lucene document from examine search result wrapper?

Take home today

• Use the index!!!

Questions

• ????• http://twitter.com/ismailmayat