backend to frontend: when database optimization affects the full stack

© 2015 by Markus WinandiStockPhoto/Mitshu

From Backend to FrontendWhen database optimization affects the full stack

@MarkusWinand@SQLPerfTips@ModernSQL

© 2015 by Markus Winand

Full Stack, Really?


What Users Want are Used to


What Users Want are Used to

Let’sclick here


User Clicks on Page 2

GimmePage

2!

Gimme Rows11 to

20

SQL for “Gimme Rows 11 to 20”

se!e"# * $%om news whe%e #op&" ' 1234 o%de% b( da#e des", &d des" o!!se" 10 #$m$" 10

Inside the Database: Worst Case

L&m&# (a"#ua! %ows'10)-> So%# (a%"ua# &ows'20) So%# Me#hod: "op-N heapso&" Memo&(: 19)B -> B&#map Heap S"an (a"#ua! %ows'10000) Re"he") *ond: (#op&" ' 1234) -> B&#map Index S"an (a"#ua! %ows'10000) Index *ond: (#op&" ' 1234)


L&m&# (a"#ua! %ows'10)-> So%# (a%"ua# &ows'10000) So%# Me#hod: ex"e&na# me&*e D$s): 1200)B -> B&#map Heap S"an (a"#ua! %ows'10000) Re"he") *ond: (#op&" ' 1234) -> B&#map Index S"an (a"#ua! %ows'10000) Index *ond: (#op&" ' 1234)


Speed depends on the number of matched rows and the page fetched.

Fetching the last pagecan take considerablylonger than fetchingthe first page.

Improvement 1: Indexed order by

se!e"# * $%om news whe%e #op&" ' 1234 o%de% b( da#e des", &d des" o$$se# 10 !&m&# 10;

%&ea"e $ndex on news ("op$%, da"e, $d)

A single index to support the whe%e and o%de% b( clauses.


L&m&# (a%"ua# &ows'10)-> Index S"an Ba")wa%d (a%"ua# &ows'10) Index *ond: (#op&" ' 0)


L&m&# (a"#ua! %ows'10)-> So%# (a%"ua# &ows'10000) So%# Me#hod: ex"e&na# me&*e D$s): 1200)B -> B&#map Heap S"an (a"#ua! %ows'10000) Re"he") *ond: (#op&" ' 1234) -> B&#map Index S"an (a"#ua! %ows'10000) Index *ond: (#op&" ' 1234)


Fetching the first page is not affected by theBase-Set size!

Fetching the next pageis also faster. However, the database might revert to the other execution plan when browsing to the end.


It’s not just about performance


Offsets Yield Unstable Results


Why don’t we ask for the next rows after...


Asking for “Next After” in SQL

SQL For “Next After” (Key-Set Pagination)

se!e"# * $%om news whe%e #op&" ' 1234 and (da"e, $d) < (?, ?) o%de% b( da#e des", &d des" !&m&# 10 Look Ma,

no offset!

Key-Set Pagination: Worst Case


L&m&# (a%"ua# &ows'10)-> So%# (a%"ua# &ows'10) So%# Me#hod: "op-N heapso&" Memo&(: 18)B -> B&#map Heap S"an (&ows'10000) Re"he") *ond: (#op&" ' 1234) -> B&#map Index S"an (&ows'10000) Index *ond: (#op&" ' 1234)


L&m&# (a"#ua! %ows'10) -> So%# (a%"ua# &ows'10) So%# Me#hod: "op-N heapso&" Memo&(: 18)B -> B&#map Heap S"an (a%"ua# &ows'9990) Rows Remo+ed b( F$#"e&: 10 (new &n 9.2) -> B&#map Index S"an (a%"ua# &ows'10000) Index *ond: (#op&" ' 1234)

Key-Set Pagination w/o Index for order by

Always needs to retrieve the full base set, but the top-n sort buffer needs to hold only 10 rows.

The response time remains the same even whenbrowsing to the last page.And the memory footprintis very low!

Key-Set Pagination: Best Case

se!e"# * $%om news whe%e #op&" ' 1234 and (da#e, &d) < (?, ?) o%de% b( da#e des", &d des" !&m&# 10;

%&ea"e $ndex on news ("op$%, da"e, $d)


L&m&# (a%"ua# &ows'10)-> Index S"an Ba")wa%d (a%"ua# &ows'10) Index *ond: ((#op&" ' 1234) AND (ROW(d#, &d) < ROW(‘...’, 23456)))


Successively browsing back doesn’t slow down.

Neither the size of thebase set(*) nor the fetched page number affects the response time.

(*) the index tree depth still affects the response time.

ComparisonO

ffset

Key

-Set

Worst Case Best Case


Too good to be true?

Key-Set Pagination has serious limitations:

‣You cannot directly navigate to arbitrary pages‣because you need the values from the previous page

‣Bi-directional navigation is possible but tedious‣you need to reverse the o%de% b( direction and RV comparison

‣Tooling...???


Required Tooling for Key-Set Pagination



Proper Row-Values Support.

Don’t use OFFSET.NoSQL is

also Also for NoSQL



Proper Row-Values Support.

Don’t use OFFSET.

Offer Infinite Scrolling.

Don’t think in pages.

Ask the right question.

OFFSET is EVIL

http://use-the-index-luke.com/no-offset




About Markus Winand

Tuning developers forhigh SQL performance

Training & co (one-man show): winand.at

Geeky blog: use-the-index-luke.com

Author of: SQL Performance Explained

http://use-the-index-luke.com

http://use-the-index-luke.com

backend to frontend: when database optimization affects the full stack

Technology