groonga with postgresql
Post on 13-Sep-2014
7.710 views
DESCRIPTION
@groongaを囲む夕ベ 2011/11/29TRANSCRIPT
![Page 1: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/1.jpg)
groonga with PostgreSQL
2011/11/29 groongaを囲む夕べ #2
板垣 & 奥野(@choplin)
フォルシア株式会社
1
![Page 2: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/2.jpg)
Agenda
• Extensibility of PostgreSQL
• How to work with groonga
• textsearch_groonga
• groonga_fdw
2
![Page 3: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/3.jpg)
Extensibility ofPostgreSQL
3
![Page 4: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/4.jpg)
http://www.slideshare.net/HitoshiHarada/extensions-on-postgresql
©2011 FORCIA, Inc.
Extensibility of PostgreSQL
Index
Table
GiST/GIN,textsearch_senna, textsearch_groonga
pl/R, pl/v8js, pl/Ruby, pl/Lua, …
cube, ltree, citext, hstore, json, …
Executor
Optimizer
Data TypesFunctions PL
Parser
dblink, adminpack, pgcrypto, ...
auto_explain
pgadviser
SQL/MED (9.1!)
c.f. external tools: pgpool, slony, pg_bulkload…
4
![Page 5: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/5.jpg)
PGXN
• PostgreSQL Extension Network
• Central distribution system for PostgreSQL Extension
• Inspired by CPAN for Perl library
• Provides:
• Manager, API, Search, Client
• under development ...
5
![Page 6: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/6.jpg)
PGXN
6
![Page 7: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/7.jpg)
Only three steps to install an extension
$ pgxn install tinyint
$ psql
# CREATE EXTENSION tinyint;--That’s all, yay!
Coming soon
7
![Page 8: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/8.jpg)
How to workwith groonga
8
![Page 9: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/9.jpg)
http://www.slideshare.net/HitoshiHarada/extensions-on-postgresql
©2011 FORCIA, Inc.
Extensibility of PostgreSQL
Index
Table
GiST/GIN,textsearch_senna, textsearch_groonga
pl/R, pl/v8js, pl/Ruby, pl/Lua, …
cube, ltree, citext, hstore, json, …
Executor
Optimizer
Data TypesFunctions PL
Parser
dblink, adminpack, pgcrypto, ...
auto_explain
pgadviser
SQL/MED (9.1!)
c.f. external tools: pgpool, slony, pg_bulkload…
9
![Page 10: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/10.jpg)
http://www.slideshare.net/HitoshiHarada/extensions-on-postgresql
©2011 FORCIA, Inc.
Extensibility of PostgreSQL
Index
Table
GiST/GIN,textsearch_senna, textsearch_groonga
pl/R, pl/v8js, pl/Ruby, pl/Lua, …
cube, ltree, citext, hstore, json, …
Executor
Optimizer
Data TypesFunctions PL
Parser
dblink, adminpack, pgcrypto, ...
auto_explain
pgadviser
SQL/MED (9.1!)
c.f. external tools: pgpool, slony, pg_bulkload…
As Index
9
![Page 11: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/11.jpg)
http://www.slideshare.net/HitoshiHarada/extensions-on-postgresql
©2011 FORCIA, Inc.
Extensibility of PostgreSQL
Index
Table
GiST/GIN,textsearch_senna, textsearch_groonga
pl/R, pl/v8js, pl/Ruby, pl/Lua, …
cube, ltree, citext, hstore, json, …
Executor
Optimizer
Data TypesFunctions PL
Parser
dblink, adminpack, pgcrypto, ...
auto_explain
pgadviser
SQL/MED (9.1!)
c.f. external tools: pgpool, slony, pg_bulkload…
As IndexAs FDW
groonga_fdw
9
![Page 12: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/12.jpg)
textsearch_groonga
10
![Page 13: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/13.jpg)
Full text searchin PostgreSQL
https://docs.google.com/present/view?id=dhk2smpg_3h44cnrdv
Builtin Extension
morphological analysis
Tsearch (8.3-)+
textsearch_ja
N-gram contrib/pg_trgm(9.1-)
Ludiatextsearch_senna
textsearch_groonga
11
![Page 14: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/14.jpg)
Motivationtextsearch_groonga䛾㛤Ⓨືᶵ
• textsearch_ja (༢ㄒ)
ᅛ᭷ྡモ䜔䜹䝍䜹䝘㐃ㄒ䛾᳨⣴₃䜜䛜ᛧ䛔
• textsearch_senna (N-gram)
©2010 FORCIA, Inc.
PostgreSQL 8.3䛾HOT᭦᪂䛸┦ᛶ䛜ᝏ䛔䛣䛸䛜ุSenna䜒䝯䞁䝔䝘䞁䝇䞉䝣䜵䞊䝈䛻ධ䜛䜏䛯䛔䛰䛧…
• textsearch_groonga
テキスト
http://www.slideshare.net/ItagakiTakahiro/textsearch-groonga-v0112
![Page 15: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/15.jpg)
Indexing䛔᪉ :䜲䞁䝕䝑䜽䝇సᡂ
• CREATE TABLE tbl(id serial, doc text);• COPY tbl(doc) FROM '...'; --䝕䞊䝍ᢞධ• CREATE INDEX idx ON tbl USING groonga (doc);
CTID ᩥ᭩ᩥ᭩ᩥ᭩ᩥ᭩ 䛭䛾䛭䛾䛭䛾䛭䛾 _key ᩥ᭩ᩥ᭩ᩥ᭩ᩥ᭩ CTID ༢ㄒ༢ㄒ༢ㄒ༢ㄒ _key
©2010 FORCIA, Inc.
CTID ᩥ᭩ᩥ᭩ᩥ᭩ᩥ᭩ 䛭䛾䛭䛾䛭䛾䛭䛾
(1, 0) foo 1.5
(1, 1) bar 2.0
(2, 0) baz 4.5
PostgreSQL䝔䞊䝤䝹
CTID =≀⌮⨨
_key ᩥ᭩ᩥ᭩ᩥ᭩ᩥ᭩ CTID
1 bar (1, 1)
2 baz (2, 0)
3 foo (1,0)
groonga䝔䞊䝤䝹
༢ㄒ༢ㄒ༢ㄒ༢ㄒ _key
ar 1
az 2
ba 1, 2
fo 3
oo 3
groonga㌿⨨⾲
http://www.slideshare.net/ItagakiTakahiro/textsearch-groonga-v0113
![Page 16: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/16.jpg)
Querying
http://www.slideshare.net/ItagakiTakahiro/textsearch-groonga-v01
䛔᪉ : ᳨⣴
• SELECT * FROM tbl WHERE …–༢⣧᳨⣴
• doc = '䜻䞊䝽䞊䝗' –ẚ㍑₇⟬Ꮚ䛻䜘䜛᳨⣴– <, <=, =, >=, >, <>
• ᩥ᭩䛸ᩘ್䜢」ᩘิ䜲䞁䝕䝑䜽䝇䛻䛧䛶䛔䜛ሙྜ➼–༢ㄒ᳨⣴
©2010 FORCIA, Inc.
–༢ㄒ᳨⣴• doc %% '䜻䞊䝽䞊䝗' – N-gram䛻䜘䜛᳨⣴• 䛂䜻䞊䝽䞊䝗䜢ྵ䜐䛃᳨⣴
– grn_expr᳨⣴• doc @@ groonga.query('䜻䞊䝽䞊䝗', 'ิ䛸㔜䜏')• ⮬⏤䛺᳨⣴
14
![Page 17: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/17.jpg)
groonga_fdw
15
![Page 18: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/18.jpg)
•Management of External Data
• SQL Standard (SQL:2003)
• Consists:
• Foreign Table
• DATALINK
SQL/MED
16
![Page 19: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/19.jpg)
•Management of External Data
• SQL Standard (SQL:2003)
• Consists:
• Foreign Table
• DATALINK
SQL/MED
Implemented as FDWin Postgres
16
![Page 20: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/20.jpg)
FDW• Foreign Data Wrapper
• External data as Table of PostgreSQL
• Available from 9.1
• Only “SELECT” statement is available now
• Very simple API interface
• Only 6 callbacks are required :)
17
![Page 21: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/21.jpg)
FDW
Parser Planner Executer InternalStorage
18
![Page 22: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/22.jpg)
FDW
Parser Planner Executer InternalStorage
ExternalData
FDW18
![Page 23: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/23.jpg)
FDW• Example:
• Oracle
• MySQL
• Redis
• Amazon S3
• and more... (see http://pgxn.org/tag/fdw/ )19
![Page 24: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/24.jpg)
groonga_fdw
• Foreign Data Wrapper for groonga
• You can query groonga via postgres with usual SQL
• Under Development
20
![Page 25: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/25.jpg)
How to use# CREATE EXTENTION groonga_fdw;
# CREATE SERVER groonga_server FOREIGN DATA WRAPPER groonga_fdw OPTIONS ( dbpath ‘ );
# CREATE FOREIGN TABLE groonga_fdw_tutorial ( _key text title text ) SERVER groonga_server OPTIONS ( ... );
Install
Connect
Table
21
![Page 26: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/26.jpg)
How to use
# SELECT * FROM groonga_fdw_tutorial;Query
22
![Page 27: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/27.jpg)
Compare
• textsearch_groonga
• <-> Tritton
• groonga_fdw
• <-> mroonga
23
![Page 28: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/28.jpg)
Summary
• PostgreSQL is quite extensible
• Two ways to work with groonga
• textsearch_groonga
• groonga_fdw
• PostgreSQL and groonga are good friend :)
24
![Page 29: groonga with PostgreSQL](https://reader034.vdocuments.net/reader034/viewer/2022042623/5413bd608d7f7299698b46d5/html5/thumbnails/29.jpg)
Thanks!
25