internal pattern matching queries in a text and applicationskociumaka/files/soda2015b_slides.pdf ·...

Internal Pattern Matching Queries in a Text andApplications

Tomasz Kociumaka Jakub RadoszewskiWojciech Rytter Tomasz Waleń

University of Warsaw, Poland

SODA 2015San Diego, California, USA

January 4, 2015

Tomasz Kociumaka, J. Radoszewski, W. Rytter, T. Waleń Internal Pattern Matching Queries in a Text 1/16

Classic Data Structures for Text Processing

T : a b a a b a b a a b a a b a b a a b

a b a a b a b a a b a a b a b a a bconstruction

data structurequeryquery

P : a b a b apos: 4, 12

T [4..8]?= T [12..16]

YES

Indexing SubwordsGiven a pattern P, find alloccurrences of P in T .

Indexing queries:

a string is part of the queryinput.

Subword EqualityGiven two subwords x , y of T ,decide whether x = y .

Internal queries:

concern subwords of T only(identified by positions).




a b a a b a b a a b a a b a b a a b

construction

data structure

queryquery

P : a b a b apos: 4, 12

T [4..8]?= T [12..16]

YES


Indexing queries:



Internal queries:






construction


P : a b a b apos: 4, 12

T [4..8]?= T [12..16]

YES


Indexing queries:



Internal queries:






construction


P : a b a b a

pos: 4, 12

T [4..8]?= T [12..16]

YES


Indexing queries:



Internal queries:




T :




P : a b a b apos: 4, 12

T [4..8]?= T [12..16]

YES


Indexing queries:



Internal queries:






construction


P : a b a b apos: 4, 12

T [4..8]?= T [12..16]

YES


Indexing queries:



Internal queries:




T :




P : a b a b apos: 4, 12

T [4..8]?= T [12..16]

YES


Indexing queries:



Internal queries:






construction


P : a b a b apos: 4, 12

T [4..8]?= T [12..16]

YES


Indexing queries:



Internal queries:






construction


P : a b a b apos: 4, 12

T [4..8]?= T [12..16]

YES


Indexing queries:



Internal queries:←− this talk



Internal Queries

Internal queries

Solve a problem for inputs being subwords of a given text T .

O(1) query input size allows for O(1) query time,Ω(|P|) required if the “pattern” P is in the input.

Motivation:

All input data must be known in advancePrimitives for algorithms and data structures

linear size and O(1) query time crucial for applicability,efficient construction important for applications in algorithms.


Internal Queries

Internal queries



Motivation:

All input data must be known in advance

Primitives for algorithms and data structureslinear size and O(1) query time crucial for applicability,efficient construction important for applications in algorithms.


Internal Queries

Internal queries



Motivation:

All input data must be known in advancePrimitives for algorithms and data structures

linear size and O(1) query time crucial for applicability,efficient construction important for applications in algorithms.


Internal Queries: Previous Techniques

13

6

8

1

10

3

12

5

7

9

2

11

4

$ab

a

b

a

b

b

$

b$ babababb$

b$ babababb$

b

$ a

b

a

babb$

bb$

bb

$abababb$

b

$abababb$

Suffix tree



Range successor queries



Suffix tree13

6

8

1

10

3

12

5

7

9

2

11

4

$ab

a

b

a

b

b

$

b$ babababb$

b$ babababb$

b

$ a

b

a

babb$

bb$

bb

$abababb$

b

$abababb$

Range successor queries

Suffix tree +problem-specific tools:

Longest common prefix

Min/Max suffix

+ efficient, often O(1) timequeries with O(n) space,

− limited use.

Suffix tree +orthogonal range queries:

Pattern matching-related

Period queries

+ wide applicability,

− lower bounds for query time,slow construction.


Internal Pattern Matching Queries

Fact

Let Occ(x , y) be the positions in y where occurrences of x start.Occ(x , y) forms an arithmetic progression if |y | ≤ 2|x |.

x : a b a b a b a b b a b a b a b a b a b a ay :

b b a b a b a b a b a b a ab b a b a b a b a b a b a ab b a b a b a b a b a b a a

Occ(x , y) =

3, 5, 7

T :

Problem (Internal Pattern Matching Queries)

Given subwords x and y of T with |y | ≤ 2|x |, report alloccurrences of x in y (as an arithmetic progression).

Θ(|y ||x |

)space is necessary to encode Occ(x , y),

Occ(x , y) can be computed with Θ(|y ||x |

)IPM Queries.



Fact


x : a b a b a b a

b b a b a b a b a b a b a a

y : b b a b a b a b a b a b a a

b b a b a b a b a b a b a ab b a b a b a b a b a b a a

Occ(x , y) = 3,

5, 7

T :



Θ(|y ||x |



)IPM Queries.



Fact


x : a b a b a b a


y :




Occ(x , y) = 3, 5,

7

T :



Θ(|y ||x |



)IPM Queries.



Fact


x : a b a b a b a


y :

b b a b a b a b a b a b a ab b a b a b a b a b a b a a


Occ(x , y) = 3, 5, 7

T :



Θ(|y ||x |



)IPM Queries.



Fact


x : a b a b a b a b b a b a b a b a b a b a ay :

b b a b a b a b a b a b a ab b a b a b a b a b a b a ab b a b a b a b a b a b a a

Occ(x , y) = 3, 5, 7

T :



Θ(|y ||x |



)IPM Queries.


IPM Queries: Our Results



Theorem

IPM Queries can be answered in O(1) time by a data structureof size O(n), which can be constructed in O(n) time.

Technical assumptions:

Word-RAM model with word-size w = Ω(log n).Construction:

randomized (Las Vegas, expected time),





Theorem



Word-RAM model with word-size w = Ω(log n).

Construction:randomized (Las Vegas, expected time),





Theorem



Word-RAM model with word-size w = Ω(log n).Construction:

randomized (Las Vegas, expected time),


Applications


Prefix-Suffix and Period Queries

T : b a b a b a b a b b a a

b a b a b a b a b b a ab a b a b a b a b b a ab a b a b a b a b b a a

x :

d

2d

b a b b a b a b a b a

b a b b a b a b a b ab a b b a b a b a b ab a b b a b a b a b a

y :

d

2d

lengths: 4

, 6, 8

Problem (Prefix-Suffix Queries)

Given subwords x and y of T and d ∈ N, report all prefixes of x oflength between d and 2d that are also suffixes of y .

O(1) time using IPM Queries.

x : a b b a b a b b a b a b b a b a b b

Problem (Period Queries)

Given a subword x of T , report all periods of x .

O(log |x |) time using IPM Queries.



T :

b a b a b a b a b b a a


b a b a b a b a b b a ab a b a b a b a b b a a

x :

d

2d



b a b b a b a b a b ab a b b a b a b a b a

y :

d

2d

lengths: 4

, 6, 8










T :

b a b a b a b a b b a ab a b a b a b a b b a a



x :

d

2d

b a b b a b a b a b ab a b b a b a b a b a



y :

d

2d

lengths: 4, 6

, 8










T :


b a b a b a b a b b a ax :

d

2d


b a b b a b a b a b ay :

d

2d

lengths: 4, 6, 8












x :

d

2d



y :

d

2d

lengths: 4, 6, 8












x :

d

2d



y :

d

2d

lengths: 4, 6, 8







O(log |x |) time using IPM Queries.Tomasz Kociumaka, J. Radoszewski, W. Rytter, T. Waleń Internal Pattern Matching Queries in a Text 8/16

Applications in Substring Compression

T :

y :i ji ′ j ′

x :

Substring Compressionserver

client

T [i , j ]? LZ (y)

y :

T [i ′, j ′]? LZ (x |y)

x :

Problem (Cormode & Muthukrishnan; SODA 2005)

Given subwords x and y , compute LZ (x |y), i.e., the section of theLempel-Ziv (LZ77) factorization LZ(y$x) that corresponds to x .

data structure of O(n) size,

query time improved from O(C log |x |C logε n) to

O(C log log |x |C logε n) where C is the output size,

combines orthogonal range queries with IPM Queries.



T :

y :i ji ′ j ′

x :


clientT [i , j ]?

LZ (y)

y :

T [i ′, j ′]? LZ (x |y)

x :









T : y :i j

i ′ j ′x :


clientT [i , j ]?

LZ (y)

y :

T [i ′, j ′]? LZ (x |y)

x :









T : y :i j

i ′ j ′x :


clientT [i , j ]? LZ (y)

y :

T [i ′, j ′]? LZ (x |y)

x :









T : y :i j

i ′ j ′x :

Generalized Substring Compressionserver


y :

T [i ′, j ′]?

LZ (x |y)

x :









T : y :i ji ′ j ′

x :



y :

T [i ′, j ′]?

LZ (x |y)

x :









T : y :i ji ′ j ′

x :



y :

T [i ′, j ′]? LZ (x |y)

x :








Other Applications

Problem (Cyclic Equivalence Queries)

Decide whether given subwords x and y are cyclic shifts of eachother (x = uv and y = vu for some strings u and v).

O(1) query time using IPM Queries.

Definition (δ-subrepetition)

A δ-subrepetition is a fragment x such that per(x) ≤ |x |1+δ and

x cannot be extended (to the left or right) preserving per(x).

Improved algorithm for finding δ-subrepetitions in a word:first (expected) linear-time algorithm for δ = Θ(1).

More applications through wavelet suffix trees:substring suffix rank & selection,substring compression using Burrows-Wheeler transform.


High-level Ideas


IPM Queries: Outline

x :

z :

2k

y :

Main idea:Design (locally consistent) representative assignment

representatives of length 2k with |x|4 < 2k ≤ |x|

2 ,store information about location of representative only.

Query algorithm:

1. Compute the representative z assigned to x .

2. Find occurrences of z within y .

3. Check which occurrences of z extend to occurrences of x .



x : z :

2k

y :




Query algorithm:


2. Find occurrences of z within y .




x : z :

2k

y :




Query algorithm:


2. Find occurrences of z (as representatives) within y .




x : z :

2k

y :




Query algorithm:


2. Find occurrences of z (as representatives) within y .

3. Check which occurrences of z extend to occurrences of x .Tomasz Kociumaka, J. Radoszewski, W. Rytter, T. Waleń Internal Pattern Matching Queries in a Text 12/16

Periodic Representatives

x :

y :

Representative: any periodic subword of appropriate length.

use combinatorial and algorithmic tools for repetitions.

Query algorithm:

1. Extend z to a run.

2. Find runs of the same period in y .

3. Generate candidates using run endpoints.

4. Check which candidates are indeed occurrences.



x :

y :



Query algorithm:

1. Extend z to a run. (Suppose it does not cover x .)


3. Generate candidates using run endpoints.

4. Check which candidates are indeed occurrences.



x :

y :



Query algorithm:

1. Extend z to a run. (Suppose it covers x .)


3. Synchronize runs using Lyndon roots.

4. Output occurrences in bulk.


Non-periodic Representatives

x : z :

x : z :x : z :x : z :x : z :

y :

Non-periodic representatives:

occurrences of z in y cannot overlap too much;O(1) occurrences within y .

Representative assignment:minimal subword of appropriate length wrt. a random order

guaranteed local consistence,neighbouring patterns often share the representative,in total O(n) representatives with O(n) occurrences as arepresentative.



x : z :

x : z :x : z :x : z :x : z :

y :

Non-periodic representatives:occurrences of z in y cannot overlap too much;

O(1) occurrences within y .





x : z :

x : z :x : z :x : z :x : z :

y :



Representative assignment:

minimal subword of appropriate length wrt. a random orderguaranteed local consistence,neighbouring patterns often share the representative,in total O(n) representatives with O(n) occurrences as arepresentative.



x : z :

x : z :x : z :x : z :x : z :

y :




guaranteed local consistence,

neighbouring patterns often share the representative,in total O(n) representatives with O(n) occurrences as arepresentative.



x : z :

x : z :x : z :x : z :x : z :

y :




guaranteed local consistence,neighbouring patterns often share the representative,

in total O(n) representatives with O(n) occurrences as arepresentative.



x : z :

x : z :

x : z :x : z :x : z :

y :








x : z :x : z :

x : z :

x : z :x : z :

y :








x : z :x : z :x : z :

x : z :

x : z :

y :








x : z :x : z :x : z :x : z :

x : z :

y :







Conclusions & Open Problems

Our results:

a O(n)-size data structure for IPM Queries with O(1)-timequeries and O(n)-expected time construction,several applications of IPM Queries:

further internal queries,faster algorithms.

Open problems:

More applications of IPM Queries.Design a deterministic construction algorithm. . .

or an algorithm running in O(n) time w.h.p.

Can shortest periods be computed in o(log |x |) time?Is there any lower bound for this problem?



Our results:



Open problems:

More applications of IPM Queries.

Design a deterministic construction algorithm. . .or an algorithm running in O(n) time w.h.p.




Our results:



Open problems:

More applications of IPM Queries.Design a deterministic construction algorithm. . .

or an algorithm running in O(n) time w.h.p.



Thank you

Thank you for your attention!


internal pattern matching queries in a text and applicationskociumaka/files/soda2015b_slides.pdf ·...

Documents