![Page 1: 20080330 Postgresqlconference2008 Pg In Web2.0 Samokhvalov](https://reader034.vdocuments.net/reader034/viewer/2022042700/5558ea3ad8b42ab8168b508d/html5/thumbnails/1.jpg)
Using PostgreSQLIn Web 2.0 Applications
How PostgreSQL helps to build Web 2.0 Apps
Nikolay SamokhvalovPostgresmen, LLC (Moscow, Russia)
PostgreSQL Conference East 2008
![Page 2: 20080330 Postgresqlconference2008 Pg In Web2.0 Samokhvalov](https://reader034.vdocuments.net/reader034/viewer/2022042700/5558ea3ad8b42ab8168b508d/html5/thumbnails/2.jpg)
Using PostgreSQL In Web 2.0 Applications
What Is Web 2.0?
![Page 3: 20080330 Postgresqlconference2008 Pg In Web2.0 Samokhvalov](https://reader034.vdocuments.net/reader034/viewer/2022042700/5558ea3ad8b42ab8168b508d/html5/thumbnails/3.jpg)
Using PostgreSQL In Web 2.0 Applications
What Is Web 2.0?
For users:
Collaborative web (UGC*, comments, rating system, etc)
Web applications, not web sites (AJAX, more interaction)
Web as a platform (interoperability, RSS, microformats, etc)
Rounded corners, mirrored logos etc :)
*) UGC — usergenerated content
![Page 4: 20080330 Postgresqlconference2008 Pg In Web2.0 Samokhvalov](https://reader034.vdocuments.net/reader034/viewer/2022042700/5558ea3ad8b42ab8168b508d/html5/thumbnails/4.jpg)
Using PostgreSQL In Web 2.0 Applications
What Is Web 2.0?
For software developers it means:
more users more developers more brands
more competitorson the market
higherrates of
development, shorter iterations
rapidly changingbusiness
requirements
Better technologies help to win!
largerdata
volumes
number of users,pageviews, TPS,
etc:N = et
![Page 5: 20080330 Postgresqlconference2008 Pg In Web2.0 Samokhvalov](https://reader034.vdocuments.net/reader034/viewer/2022042700/5558ea3ad8b42ab8168b508d/html5/thumbnails/5.jpg)
Using PostgreSQL In Web 2.0 Applications
Why PostgreSQL?
1. Performance, scalability
2. Reliability
3. Powerful capabilities
4. Standards compliance, proper approaches
5. Freedom
![Page 6: 20080330 Postgresqlconference2008 Pg In Web2.0 Samokhvalov](https://reader034.vdocuments.net/reader034/viewer/2022042700/5558ea3ad8b42ab8168b508d/html5/thumbnails/6.jpg)
Using PostgreSQL In Web 2.0 Applications
Why PostgreSQL?
1. Performance, scalability
2. Reliability
3. Powerful capabilities
4. Standards compliance, proper approaches
5. Freedom
Quality
Development efficiency
}
}}
HR
![Page 7: 20080330 Postgresqlconference2008 Pg In Web2.0 Samokhvalov](https://reader034.vdocuments.net/reader034/viewer/2022042700/5558ea3ad8b42ab8168b508d/html5/thumbnails/7.jpg)
Using PostgreSQL In Web 2.0 Applications
How to Deal With UGC?
1. Taxonomy
● Catalogs
2. Folksonomy
● Tags
3. Hybrid, two ways:
● Tags + Catalogs
● Both users and editors control Catalogs
![Page 8: 20080330 Postgresqlconference2008 Pg In Web2.0 Samokhvalov](https://reader034.vdocuments.net/reader034/viewer/2022042700/5558ea3ad8b42ab8168b508d/html5/thumbnails/8.jpg)
Using PostgreSQL In Web 2.0 Applications
UGC: Taxonomy
1. Taxonomy (Catalogs)
● EAV, where ATTRIBUTE table is [almost] constant
● intarray / hstore
![Page 9: 20080330 Postgresqlconference2008 Pg In Web2.0 Samokhvalov](https://reader034.vdocuments.net/reader034/viewer/2022042700/5558ea3ad8b42ab8168b508d/html5/thumbnails/9.jpg)
Using PostgreSQL In Web 2.0 Applications
EAV: EntityAttibuteValueEntity
Value
Attribute
![Page 10: 20080330 Postgresqlconference2008 Pg In Web2.0 Samokhvalov](https://reader034.vdocuments.net/reader034/viewer/2022042700/5558ea3ad8b42ab8168b508d/html5/thumbnails/10.jpg)
Using PostgreSQL In Web 2.0 Applications
intarray / hstoreitem
obj_id INT8item_section_id INT8item_vendor_id INT8item_model_id INT8item_year INT2item_price NUMERIC(30,6)item_props intarray
What about performance?
● This approach allows to save much space
● Performance is good if you mix GiST/GIN search with FTS search
● Better to cache tag values in external cache (e.g. Memcache) if you use intarray, but in this case using FTS is a bit harder
![Page 11: 20080330 Postgresqlconference2008 Pg In Web2.0 Samokhvalov](https://reader034.vdocuments.net/reader034/viewer/2022042700/5558ea3ad8b42ab8168b508d/html5/thumbnails/11.jpg)
Using PostgreSQL In Web 2.0 Applications
UGC: Folksonomy
1. Folksonomy (Tags)
1. EAV (again), usercontrolled ATTRIBUTE table
2. intarray / hstore (again)
— it's just almost the same, you just give control to your users
Tags:
![Page 12: 20080330 Postgresqlconference2008 Pg In Web2.0 Samokhvalov](https://reader034.vdocuments.net/reader034/viewer/2022042700/5558ea3ad8b42ab8168b508d/html5/thumbnails/12.jpg)
Using PostgreSQL In Web 2.0 Applications
UGC: Hybrid
1. Hybrid, two ways:
1. Tags + Catalogs
— common practice
2. Both users and editors control Catalogs
— is the most interesting, but is the most difficult to implement and maintain
● UGConly catalog entries are not shown in common <SELECT> lists, they are waiting for editors approval.
● 'Merge' procedure is really complicated (merge UGC with editors' data; merge duplicates, synonyms, etc).
● FTS (stemming, morphology, thesaurus), pg_trgm, metaphone, soundex, etc may help. BUT: human work is still needed.
![Page 13: 20080330 Postgresqlconference2008 Pg In Web2.0 Samokhvalov](https://reader034.vdocuments.net/reader034/viewer/2022042700/5558ea3ad8b42ab8168b508d/html5/thumbnails/13.jpg)
Using PostgreSQL In Web 2.0 Applications
UGC: More About Tags
1. Use FTS (tsearch2) to integrate tag searching in your search subsystem:
● use FTS categories to differ tag words from mere words when needed;
● to process tags, use separate FTS configuration, if needed.
2. Use "prefix search" for tag searching, but it's not straightforward (wait for the next slides ;) )
![Page 14: 20080330 Postgresqlconference2008 Pg In Web2.0 Samokhvalov](https://reader034.vdocuments.net/reader034/viewer/2022042700/5558ea3ad8b42ab8168b508d/html5/thumbnails/14.jpg)
Using PostgreSQL In Web 2.0 Applications
UGC: Tags And Prefix Search
"Prefix search" helps to build smth like this:
If you use simple LIKE 'bla%' the result will be somewhat dissapointing:
test=# EXPLAIN ANALYZE SELECT * FROM tag WHERE tag_name LIKE 'bla%'; QUERY PLAN Seq Scan on tag (cost=0.00..6182.75 rows=1 width=105) (actual time=0.951..102.779 rows=162 loops=1) Filter: ((tag_name)::text ~~ 'bla%'::text) Total runtime: 102.871 ms(3 rows)
Notice: ~300k unique tags in the table
![Page 15: 20080330 Postgresqlconference2008 Pg In Web2.0 Samokhvalov](https://reader034.vdocuments.net/reader034/viewer/2022042700/5558ea3ad8b42ab8168b508d/html5/thumbnails/15.jpg)
Using PostgreSQL In Web 2.0 Applications
Tags And Prefix Search:The Proper Solution
1. Use text_pattern_ops to speed up LIKE 'bla%' queries:test=# CREATE INDEX i_tag_prefix ON tag USING btree(lower(tag_name) text_pattern_ops);CREATE INDEX
test=# EXPLAIN ANALYZE SELECT * FROM tag WHERE lower(tag_name) LIKE lower('bla%'); QUERY PLAN Bitmap Heap Scan on tag (cost=43.55..2356.16 rows=1096 width=105) (actual time=0.164..0.791 rows=235 loops=1) Filter: (lower((tag_name)::text) ~~ 'bla%'::text) > Bitmap Index Scan on i_tag_prefix (cost=0.00..43.28 rows=1096 width=0) (actual time=0.116..0.116 rows=235 loops=1) Index Cond: ((lower((tag_name)::text) ~>=~ 'bla'::text) AND(lower((tag_name)::text) ~<~ ' '::text))мис Total runtime: 0.885 ms(5 rows)
Notices: (1) ILIKE is not acceptable, so use lower(); (2) be careful using nonASCII charactes (i.e. it's OK for Russian UTF8 except minor 'ё' & 'Ё' chars)
![Page 16: 20080330 Postgresqlconference2008 Pg In Web2.0 Samokhvalov](https://reader034.vdocuments.net/reader034/viewer/2022042700/5558ea3ad8b42ab8168b508d/html5/thumbnails/16.jpg)
Using PostgreSQL In Web 2.0 Applications
Tags And Prefix Search:The Proper Solution
2. Create tag_words (unique tag words) table to work with words, not with phrases:
CREATE TABLE tag_words AS SELECT DISTINCT word FROM ts_stat('SELECT to_tsvector(tag_name) FROM tag'); heavyDROP INDEX i_tag_prefix;CREATE INDEX i_tag_fts ON tag USING gin(to_tsvector(tag_name));CREATE INDEX i_tag_words_prefix ON tag_words USING btree(lower(word) text_pattern_ops);
test=# EXPLAIN ANALYZE SELECT * FROM tag WHERE to_tsvector('utf8_russian'::regconfig, tag_name::text) @@ to_tsquery('utf8_russian', '(' || ( SELECT array_to_string(array_accum(lower(word)), '|') FROM tag_words WHERE lower(word) LIKE 'bla%') || ')'); add '...&word1&word2' if needed/* plan is omitted */ Total runtime: 13.243 ms(11 rows)
Notices: (1) better to limit number of tag words found by the inner query (e.g. ordering by word age — dirty but it works); (2) word order in original query is lost, unfortunately; (3) GIN indexes are better than GiST here
![Page 17: 20080330 Postgresqlconference2008 Pg In Web2.0 Samokhvalov](https://reader034.vdocuments.net/reader034/viewer/2022042700/5558ea3ad8b42ab8168b508d/html5/thumbnails/17.jpg)
Using PostgreSQL In Web 2.0 Applications
Rate And Comment Everything
PostgreSQL Inheritance helps to achieve development efficiency
obj
obj_id INT8obj_status_did INT8obj_creator_obj_id INT8obj_created TIMESTAMPobj_modified TIMESTAMPobj_commented TIMESTAMPobj_marks_count INT4obj_marks_rating FLOAT8obj_tsvector tsvector
user
group
— Not SERIAL, wait for the next slide to see details— Dictionary value— ID of user who created the record (if applicable)
} NOT NULL DEFAULT CURRENT_TIMESTAMP
— Almost all business objects need FTS
} rate everything!
comment
comment_author_obj_idcomment_text
comment
comment_text
user2obj
u2o_user_obj_idu2o_obj_obj_idu2o_marku2o_is_favorite
![Page 18: 20080330 Postgresqlconference2008 Pg In Web2.0 Samokhvalov](https://reader034.vdocuments.net/reader034/viewer/2022042700/5558ea3ad8b42ab8168b508d/html5/thumbnails/18.jpg)
Using PostgreSQL In Web 2.0 Applications
Rate And Comment Everythingcreate table comment ( obj_id INT8 not null default (((nextval('comment_obj_id_seq'::regclass) * 223072849) % (1000000000)::bigint) + 41000000000) constraint c_obj_comment_obj_id check (obj_id between 41000000000 and 41999999999), comment_author_obj_id INT8, comment_body VARCHAR (2000) NOT NULL, constraint PK_MESSAGE primary key (obj_id))inherits (obj);
ID generation scheme: nextID = (N mod Y) * X + S, where X & Y are coprimes, and S is an interval shift
Use separate sequence per each table!
do not forget:SET constraint_exclusion ON;
![Page 19: 20080330 Postgresqlconference2008 Pg In Web2.0 Samokhvalov](https://reader034.vdocuments.net/reader034/viewer/2022042700/5558ea3ad8b42ab8168b508d/html5/thumbnails/19.jpg)
Using PostgreSQL In Web 2.0 Applications
Build your Google Maps mashup:with PostgreSQL it's easy
Ways to store & index geo data in PostgreSQL: two integer columns and Btree point column and Rtree PostGIS pgSphere Q3C
GiST
MirTesen.ru
![Page 20: 20080330 Postgresqlconference2008 Pg In Web2.0 Samokhvalov](https://reader034.vdocuments.net/reader034/viewer/2022042700/5558ea3ad8b42ab8168b508d/html5/thumbnails/20.jpg)
Using PostgreSQL In Web 2.0 Applications
Conclusion
PostgreSQL provides a great set of capabilities to meet Web 2.0 developer needs
PostgreSQL allows to develop quickly, w/o losing quality
![Page 21: 20080330 Postgresqlconference2008 Pg In Web2.0 Samokhvalov](https://reader034.vdocuments.net/reader034/viewer/2022042700/5558ea3ad8b42ab8168b508d/html5/thumbnails/21.jpg)
Using PostgreSQL In Web 2.0 Applications
Contacts
● Blog: http://nikolay.samokhvalov.com
● XMPP/GTalk: [email protected]
● Skype: samokhvalov OR postgresmen
● +7 905 783 9804