columnstore index _ how_it_work - sqlsaturday

7/28/2019 ColumnStore Index _ How_it_work - SQLSaturday

1/26

With

columnstore

indexes, data is grouped and stored

one column at a time.

By Percy Reyes


2/26

Percy Reyes

CEO & Internal Senior DBA at SQL4Junkies

MCITP/MCTS: DBA + Dev

Web: www.sql4junkies.com

Email: [email protected]

DBA

As a Service
http://www.sql4junkies.com/mailto:[email protected]:[email protected]://www.sql4junkies.com/


3/26

Agenda

The Heart of Columnstore Indexes

Memory management

Columnstore indexes: interoperability

Columnstore Indexes Under the Hood Columnstore Indexes Performance Tuning

The Limitations of the Columnstore Indexes

How to load data for a Columnstore indexed table

The Columnstore Indexes Best Practises Resources

Q & A


4/26

The

heart of the

Columnstore

Indexes

What is columnstore?


5/26

The heart of the Columnstore Indexes

What is columnstore?

ProductKey DateKey ResellerKey Quantity Price Amount

... ...... ... ... ...

Row-Store

--Serialize all of the

attribute values in a

row together, thenthe values in the next

row, and so on.

Columns-Store--Serializes all of the

values of an attribute

(column) together


6/26


Why columnstore?

In

Memory

--Its All about I/O;--Its all about compression efficiency;--In columnstore, compression algorithms are working b

and a higher compression ratio can be achieved;--Columnstore compression improves CPU performance;

--The maximum memory is limited by OS,1TB? 2TB?;

--Get only columns which are queried;


7/26


SQL Server 2012 Columnstore Indexes Data is highly compressed by using xVelocity

Engine (aka, Vertipaq)

A new Batch mode execution model is introduced

A columstore index breaks each column into 1 millionrow chunks called segment.

Delivers order-of-magnitude gains for DW queries

Columnstore indexes work with table partitioning.

The index key record size limitation of 900 bytes also

does not apply to columnstore indexes. There is no concept of key columns in a columnstore

index,


8/26


How to create a columnstore index?CREATE NONCLUSTERED

COLUMNSTORE INDEX

[ci_FactResellerSales]

ON [dbo].[FactResellerSales]

(

[ProductKey],

[OrderDateKey],

[DueDateKey],

[ShipDateKey],

[ResellerKey],

[UnitPrice],[ExtendedAmount],

[UnitPriceDiscountPct],

[DiscountAmount],

[ProductStandardCost],

[ShipDate]

);


9/26

Memory management

Memory management is automatic

Columnstore persisted on disk

Needed columns fetched into memory

Column segments flow between disk and memory

SELECT C2, SUM(C4)

FROM T

GROUP BY C2;

T.C2T.C4

T.C2 T.C4

T.C2

T.C2

T.C2T.C1

T.C1

T.C1

T.C1

T.C1T.C3

T.C3

T.C3

T.C3

T.C3

T.C4

T.C4

T.C4


10/26

New showplan elements forcolumnstores

and batch processing


11/26


12/26

Columnstore index catalog tables


13/26

Columnstore indexes: interoperability with

the rest of SQL Server

Most things just work with columnstore indexes Backup and restore

Mirroring

Log shipping

SSMS

Administration, tools

Columnstore indexes cannot be combined with:

Page and row compression, and vardecimal storage format

Replication Change tracking

Change data capture

Filestream


14/26

Using hints with columnstore indexes

use the columnstore index

use a different index

ignore columnstore

select distinct (SalesTerritoryKey)

from dbo.FactResellerSales with (index (ncci))


from dbo.FactResellerSales with (index (ci))


from dbo.FactResellerSales

option(ignore_nonclustered_columnstore_index)


15/26

Columnstore Indexes Under the Hood

Data is highly compressed. Dramatically reduced IO. MoreData can fit into memory.

Row Store Column Store Column Store,

Compressed


16/26


Data is highly compressed. Dramatically reduced IO. Moredata can fit into memory.

Row Store Column Store,

Compressed


17/26


How data is compressed?

RLE

Run-le

ngth

Encod

ing


18/26


How data is compressed?

Diction

aryEncodin

g


19/26


A vector-based query execution method called Batch Modeprocessing is implemented. Dramatically reduced CPU

consumption time.

A Batch is an object that contains about 1000 rows


20/26


Segment elimination can skip large chunks of data to speedup scans.

A segment a 1 million of rows

Each segment stores min and max value

If no rows qualify, then the entire segment is skipped forscan


21/26

Columnstore Indexes Performance Tuning

Maximizing the use of Batch Mode processing


22/26

The Limitations of the Columnstore Indexes

Base table is not able to insert, delete and update data For current release, no clustered columnstore index

Columnstore Indexes are not designed for certain kind of

queries. No Seek.

Only one nonclustered columnstore index is allowed

Binary and varbinary

Varchar(max) and nvarchar(max)

Decimal/Numeric precision > 18

Ntext, text, and image

Uniqueidentifier Rowversion

Sql-variant

Datetimeoffset(>2)

CLR data type

XML

No Sparse column

No unique columnstore index

No customer sort option for

columns

No replication No change tracking

No change data capture

No filestream column

Cant be a filtered index

Data Type not allowed: Other restrictions:


23/26

How to load data for a columnstore indexed table

There is really no simple way that loads data into acolumnstore indexed table

Columnstore is not born for data updating

3 possible ways:

Drop columnstore index, load the data, recreate. (or disable, load

the data, rebuild)

Partition the columnstore indexed table and using partition switching.

Seems the best way.

Using t 2 tables, one with historical data with columnstore index,

another is just a normal table. Complicated, lots of maintenance


24/26

The Columnstore Indexes Best Practises

Include all columns in the columnstore index wheneverpossible

Put columnstore indexes on large tables only

Consider to create a clustered index on columns which

are frequently used, like date column

Structure your queries as star joins with grouping andaggregation as much as possible

Avoid joins and string filters directly on columns of

columnstore indexed tables.

Whenever possible, avoid constructing queries with outerjoin, Union all, and not in directly on columnstore indexed

Using integer whenever possible

Using table partitions


25/26

Resources

CREATE COLUMNSTORE INDEX (Transact-SQL)http://msdn.microsoft.com/en-us/library/gg492153.aspx

Eric Hanson: SQL Server Columnstore Performance Tuning

http://social.technet.microsoft.com/wiki/contents/articles/4995.sql-server-

columnstore-performance-tuning.aspx

BOL: Columnstore Indexes

http://msdn.microsoft.com/en-us/library/gg492088.aspx Joe Sack: Exploring Columnstore Index Metadata, Segment Distribution

and Elimination Behaviors

http://www.sqlskills.com/blogs/joe/post/Exploring-Columnstore-Index-

Metadata-Segment-Distribution-and-Elimination-Behaviors.aspx

Benjamin Nevarez: Improve the Performance of Data Warehouse Queries

with Columnstore Indexes

http://www.sqlmag.com/article/sqlserverdenali/data-warehouse-queries-

columnstore-indexes-141712

Stavros Harizopoulos, Daniel Abadi: Column-Oriented Database system
http://msdn.microsoft.com/en-us/library/gg492153.aspxhttp://social.technet.microsoft.com/wiki/contents/articles/4995.sql-server-columnstore-performance-tuning.aspxhttp://social.technet.microsoft.com/wiki/contents/articles/4995.sql-server-columnstore-performance-tuning.aspxhttp://social.technet.microsoft.com/wiki/contents/articles/4995.sql-server-columnstore-performance-tuning.aspxhttp://msdn.microsoft.com/en-us/library/gg492088.aspxhttp://msdn.microsoft.com/en-us/library/gg492088.aspxhttp://www.sqlskills.com/blogs/joe/post/Exploring-Columnstore-Index-Metadata-Segment-Distribution-and-Elimination-Behaviors.aspxhttp://www.sqlskills.com/blogs/joe/post/Exploring-Columnstore-Index-Metadata-Segment-Distribution-and-Elimination-Behaviors.aspxhttp://www.sqlskills.com/blogs/joe/post/Exploring-Columnstore-Index-Metadata-Segment-Distribution-and-Elimination-Behaviors.aspxhttp://www.sqlmag.com/article/sqlserverdenali/data-warehouse-queries-columnstore-indexes-141712http://www.sqlmag.com/article/sqlserverdenali/data-warehouse-queries-columnstore-indexes-141712http://www.sqlmag.com/article/sqlserverdenali/data-warehouse-queries-columnstore-indexes-141712http://www.sqlmag.com/article/sqlserverdenali/data-warehouse-queries-columnstore-indexes-141712http://www.sqlskills.com/blogs/joe/post/Exploring-Columnstore-Index-Metadata-Segment-Distribution-and-Elimination-Behaviors.aspxhttp://msdn.microsoft.com/en-us/library/gg492088.aspxhttp://social.technet.microsoft.com/wiki/contents/articles/4995.sql-server-columnstore-performance-tuning.aspxhttp://msdn.microsoft.com/en-us/library/gg492153.aspx


26/26

Q & A

columnstore index _ how_it_work - sqlsaturday

Documents