11 boolean search

Click here to load reader

Post on 02-Sep-2015

218 views

Category:

Documents

0 download

Embed Size (px)

DESCRIPTION

db

TRANSCRIPT

  • Elementary IR: Scalable Boolean Text Search

    (Compare with R & G 27.1-3)

  • Information Retrieval: HistoryA research field traditionally separate from DatabasesHans P. Luhn, IBM, 1959: Keyword in Context (KWIC)G. Salton at Cornell in the 60s/70s: SMARTAround the same time as relational DB revolutionTons of research since thenEspecially in the web eraProducts traditionally separateOriginally, document management systems for libraries, government, law, etc.Gained prominence in recent years due to web searchStill used for non-web document management. (Enterprise search).

  • Today: Simple (nave!) IRBoolean Search on keywordsGoal:Show that you already have the tools to do this from your study of relational DBsWell skip:Text-oriented storage formatsIntelligent result ranking (hopefully later!)ParallelismCritical for modern relational DBs tooVarious bells and whistles (lots of little ones!)Engineering the specifics of (written) human languageE.g. dealing with tense and pluralsE.g. identifying synonyms and related wordsE.g. disambiguating multiple meanings of a wordE.g. clustering output

  • IR vs. DBMSSeem like very different beasts

    Under the hood, not as different as they might seemBut in practice, you have to choose between the 2 today

    IRDBMSImprecise SemanticsPrecise SemanticsKeyword searchSQLUnstructured data formatStructured dataRead-Mostly. Add docs occasionallyExpect reasonable number of updatesPage through top k resultsGenerate full answer

  • IRs Bag of Words ModelTypical IR data model:Each document is just a bag of words (terms)Detail 1: Stop WordsCertain words are not helpful, so not placed in the bage.g. real words like the e.g. HTML tags like Detail 2: StemmingUsing language-specific rules, convert words to basic form e.g. surfing, surfed --> surfUnfortunately have to do this for each languageYuck!

  • Boolean Text SearchFind all documents that match a Boolean containment expression:Windows AND (Glass OR Door) AND NOT MicrosoftNote: query terms are also filtered via stemming and stop wordsWhen web search engines say 10,000 documents found, thats the Boolean search result sizeMore or less ;-)

  • Text IndexesWhen IR folks say text indexusually mean more than what DB people meanIn our terms, both tables and indexesReally a logical schema (i.e. tables)With a physical schema (i.e. indexes)Usually not stored in a DBMSTables implemented as files in a file systemWell talk more about this decision soon

  • A Simple Relational Text IndexGiven: a corpus of text filesFiles(docID string, content string)Create and populate a tableInvertedFile(term string, docID string)Build a B+-tree or Hash index on InvertedFile.termSomething like Alternative 3 critical here!! Keep lists of dup keys sorted by docIDWill provide interesting orders later on!Fancy list compression important, tooTypically called a postings list by IR peopleNote: URL instead of RID, the web is your heap file!Can also cache pages and use RIDsThis is often called an inverted file or inverted indexMaps from words -> docs, rather than docs -> words

    Given this, you can now do single-word text search queries!

  • An Inverted FileSnippets from:Old class web pageOld microsoft.com home pageSearch fordatabasesmicrosoft

    TermdocID

    Sheet1

    termdocURL

    datahttp://www-inst.eecs.berkeley.edu/~cs186

    databasehttp://www-inst.eecs.berkeley.edu/~cs186

    datehttp://www-inst.eecs.berkeley.edu/~cs186

    dayhttp://www-inst.eecs.berkeley.edu/~cs186

    dbmshttp://www-inst.eecs.berkeley.edu/~cs186

    decisionhttp://www-inst.eecs.berkeley.edu/~cs186

    demonstratehttp://www-inst.eecs.berkeley.edu/~cs186

    descriptionhttp://www-inst.eecs.berkeley.edu/~cs186

    designhttp://www-inst.eecs.berkeley.edu/~cs186

    desirehttp://www-inst.eecs.berkeley.edu/~cs186

    developerhttp://www.microsoft.com

    differhttp://www-inst.eecs.berkeley.edu/~cs186

    disabilityhttp://www.microsoft.com

    discussionhttp://www-inst.eecs.berkeley.edu/~cs186

    divisionhttp://www-inst.eecs.berkeley.edu/~cs186

    dohttp://www-inst.eecs.berkeley.edu/~cs186

    documenthttp://www-inst.eecs.berkeley.edu/~cs186

    documenthttp://www.microsoft.com

    microsofthttp://www.microsoft.com

    microsofthttp://www-inst.eecs.berkeley.edu/~cs186

    midnighthttp://www-inst.eecs.berkeley.edu/~cs186

    midtermhttp://www-inst.eecs.berkeley.edu/~cs186

    minibasehttp://www-inst.eecs.berkeley.edu/~cs186

    millionhttp://www.microsoft.com

    Mondayhttp://www.microsoft.com

    morehttp://www.microsoft.com

    mosthttp://www-inst.eecs.berkeley.edu/~cs186

    mshttp://www-inst.eecs.berkeley.edu/~cs186

    msnhttp://www.microsoft.com

    musthttp://www-inst.eecs.berkeley.edu/~cs186

    necessaryhttp://www-inst.eecs.berkeley.edu/~cs186

    needhttp://www-inst.eecs.berkeley.edu/~cs186

    networkhttp://www.microsoft.com

    newhttp://www-inst.eecs.berkeley.edu/~cs186

    newhttp://www.microsoft.com

    newshttp://www.microsoft.com

    newsgrouphttp://www-inst.eecs.berkeley.edu/~cs186

    newsletterhttp://www.microsoft.com

    nowhttp://www.microsoft.com

    ofhttp://www.microsoft.com

    officehttp://www.microsoft.com

    phttp://www.microsoft.com

    pacifichttp://www.microsoft.com

    pagehttp://www.microsoft.com

    partnerhttp://www.microsoft.com

    personalhttp://www.microsoft.com

    portalhttp://www.microsoft.com

    presshttp://www.microsoft.com

    privacyhttp://www.microsoft.com

    producthttp://www.microsoft.com

    professionalhttp://www.microsoft.com

    http://www.microsoft.com

    http://www.microsoft.com

    http://www-inst.eecs.berkeley.edu/~cs186

    http://www-inst.eecs.berkeley.edu/~cs186

    http://www.microsoft.com

    Sheet2

    Sheet3

    Sheet1

    termdocURL

    datahttp://www-inst.eecs.berkeley.edu/~cs186

    databasehttp://www-inst.eecs.berkeley.edu/~cs186

    datehttp://www-inst.eecs.berkeley.edu/~cs186

    dayhttp://www-inst.eecs.berkeley.edu/~cs186

    dbmshttp://www-inst.eecs.berkeley.edu/~cs186

    decisionhttp://www-inst.eecs.berkeley.edu/~cs186

    demonstratehttp://www-inst.eecs.berkeley.edu/~cs186

    descriptionhttp://www-inst.eecs.berkeley.edu/~cs186

    designhttp://www-inst.eecs.berkeley.edu/~cs186

    desirehttp://www-inst.eecs.berkeley.edu/~cs186

    developerhttp://www.microsoft.com

    differhttp://www-inst.eecs.berkeley.edu/~cs186

    disabilityhttp://www.microsoft.com

    discussionhttp://www-inst.eecs.berkeley.edu/~cs186

    divisionhttp://www-inst.eecs.berkeley.edu/~cs186

    dohttp://www-inst.eecs.berkeley.edu/~cs186

    documenthttp://www-inst.eecs.berkeley.edu/~cs186

    documenthttp://www.microsoft.com

    microsofthttp://www.microsoft.com

    microsofthttp://www-inst.eecs.berkeley.edu/~cs186

    midnighthttp://www-inst.eecs.berkeley.edu/~cs186

    midtermhttp://www-inst.eecs.berkeley.edu/~cs186

    minibasehttp://www-inst.eecs.berkeley.edu/~cs186

    millionhttp://www.microsoft.com

    mondayhttp://www.microsoft.com

    morehttp://www.microsoft.com

    mosthttp://www-inst.eecs.berkeley.edu/~cs186

    mshttp://www-inst.eecs.berkeley.edu/~cs186

    msnhttp://www.microsoft.com

    musthttp://www-inst.eecs.berkeley.edu/~cs186

    necessaryhttp://www-inst.eecs.berkeley.edu/~cs186

    needhttp://www-inst.eecs.berkeley.edu/~cs186

    networkhttp://www.microsoft.com

    newhttp://www-inst.eecs.berkeley.edu/~cs186

    newhttp://www.microsoft.com

    newshttp://www.microsoft.com

    newsgrouphttp://www-inst.eecs.berkeley.edu/~cs186

    newsletterhttp://www.microsoft.com

    nowhttp://www.microsoft.com

    ofhttp://www.microsoft.com

    officehttp://www.microsoft.com

    phttp://www.microsoft.com

    pacifichttp://www.microsoft.com

    pagehttp://www.microsoft.com

    partnerhttp://www.microsoft.com

    personalhttp://www.microsoft.com

    portalhttp://www.microsoft.com

    presshttp://www.microsoft.com

    privacyhttp://www.microsoft.com

    producthttp://www.microsoft.com

    professionalhttp://www.microsoft.com

    http://www.microsoft.com

    http://www.microsoft.com

    http://www-inst.eecs.berkeley.edu/~cs186

    http://www-inst.eecs.berkeley.edu/~cs186

    http://www.microsoft.com

    Sheet2

    Sheet3

  • Handling Boolean LogicHow to do term1 OR term2?Union of two postings lists (docID sets)!How to do term1 AND term2?Intersection of two postings lists!Can be done via merge-join over postings listsRemember: postings list per key sorted by docID in indexHow to do term1 AND NOT term2?Set subtractionAlso easy because sorted (basically merge join logic again)How to do term1 OR NOT term2Union of term1 and NOT term2.Not term2 = all docs not containing term2. Yuck!Usually not allowed!Query Optimization: what order to handle terms if you have many ANDs?

  • Boolean Search in SQL(SELECT docID FROM InvertedFile WHERE word = window INTERSECT SELECT docID FROM InvertedFile WHERE word = glass OR word = door) EXCEPT SELECT docID FROM InvertedFile WHERE word=Microsoft ORDER BY magic_rank() Theres only one SQL query template in Boolean SearchSingle-table selects, UNION, INTERSECT, EXCEPTmagic_rank() is the secret sauce in the search enginesHopefully well study this later in the semesterCombos of statistics, linguistics, and graph theory tricks!

    Windows AND (Glass OR Door) AND NOT Microsoft

  • One step fancier: Phrases and NearSuppose you want a phrase E.g. Happy DaysDifferent schema:InvertedFile (term string, position int, docID string)Alternative 3 index on termPostings lists sorted by (docID, position)Post-process the resultsFind Happy AND DaysKeep results where positions are 1 offCan be done during merge-join to AND the 2 lists!Can do