kuromoji fst
TRANSCRIPT
Kuromoji FST2015/06/25
Yoshinari Fujinuma
Overview• Motivation
• Building FST
• How freezing works
• How equivalent detection works
• Compiled FST and Virtual Machine
Motivation• Efficient Key value store for dictionary look up
during tokenization
• String -> integers
• int -> token info
Why FST and not Trie? • Finite State Transducer (FST) = Finite State Automaton +
Output
• Able to merge both prefixes and suffixes too
• e.g. “can”, “cats”, “dogs”
Overview of how the build works
List of sorted words,
list of integers
FST Builder
FST Compiler
Object-based FST
Compiled FST
How Building / Compiling works
• two variables are the key
• previous word (prev)
• current word (current)
1. Skip common prefix between prev and current
2. make arcs to the temp states
3. Freeze (Finalize) states which suffix differ betw. prev and current
Toy example
• cat -> 0
• cats -> 1
• catx -> 2
Initializing states• InitializationFrozen states
Temp states
Freezing states• prev word = “”, current word = catFrozen states
Temp states t/0a/0c/0
Add Arc to suffix
Frozen states
Temp states s/1t/0a/0c/0
• prev word = cat, current word = cats
Freeze differing suffix• pre word = cats, current word = catxFrozen states
Temp states s/1t/0a/0c/0 x/2
Freeze differing suffix• pre word = cats, current word = catxFrozen states
Temp states s/1t/0a/0c/0 x/2
HashCode 1
Freeze differing suffix• pre word = cats, current word = catxFrozen states
Temp states s/1t/0a/0c/0 x/2
HashCode 1
Merge Equivalent states• pre word = catx, current word =“”Frozen states
Temp states s/1t/0a/0c/0
x/2
Freezing states• pre word = catx, current word = “”Frozen states s/1t/0a/0c/0
x/2
Temp states
Equivalent state detection• We want to merge equivalent states!
• Key-value store using HashMap
• Key: State.hashCode()
• Value: State Object
• Collisions are resolved by chaining
Arc Equivalence
c/0
• Same transition character
• Same destination state
• Same output
c/0
State Equivalence• All the outgoing set of arcs are equivalent
• Both states are of the same type of state
c/0
c/0
How Compiled FST works• Generates a “Program”
• Running a Program = look up a word in a dictionary
• Program runs on a Virtual Machine which we implemented
Compiled FST = “Program”
Virtual Machine
Worde.g. “cat”
Integer if exists in
dictionary
-1, it not
OR
Program• List of Instructions, 11 bytes each
• Operation code (Op code)
• Math or Accept, Match, Fail
Op code 1byte
transition char 2 bytes
output 4 bytes
target address 4 bytes
Match• Transition to a given address
• Accumulator += output
0 Fail None None None1 M / A x 2 02 M / A s 1 03 Fail4 Match t 0 2
….
Fail• Stop running the Program and return -1
• e.g. “tss”
0 Fail None None None1 M / A x 2 02 M / A s 1 03 Fail4 Match t 0 2
….
Match or Accept
• If the current character is the final char,
• Ends running the program and returns the accumulator
• Else Match
Instructions vs. Arcs• What instructions represent
0 Fail None None None1 M / A x 2 02 M / A s 1 03 Fail4 Match t 0 2
….
s/1x/2
t/0
Virtual Machine running backwards
0 Fail None None None1 M / A x 2 02 M / A s 1 03 Fail4 Match t 0 25 Fail6 Match a 0 47 Fail8 Match c 0 6
• Because of freezing from suffixes
Use of Cache• The lookup for next state is done by linear search
• The num. of outgoing arcs from the start state is large
• Therefore, we cache those outgoing arcs
Summary• FST is theoretically more compact than tries
• Implemented FST Builder which builds
• Object-based FST
• Compiled FST, compact form
• Uses Virtual Machine to run the compiled program (= lookup a word)
References• Direct Construction of Minimal Acyclic Subsequential
Transducers, http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.24.3698
• Smaller representation of finite-state automata http://www.sciencedirect.com/science/article/pii/S0304397512003787
• Blog post by Ikawa-san http://qiita.com/ikawaha/items/be95304a803020e1b2d1
• This code is available at https://github.com/atilika/fst