optimizing your bi semantic model for performance and scale dave wickert (ae7td)...
TRANSCRIPT
![Page 1: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/1.jpg)
Optimizing Your BI Semantic Model for Performance and Scale
Dave Wickert (AE7TD) [email protected] Program ManagerSQL Server Business IntelligenceMicrosoft
![Page 2: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/2.jpg)
My radios
Main rig:ICOM 7000100w HF into a 33’ Buddipole
![Page 3: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/3.jpg)
My radios
QRP rig: (on order )Elecraft KX310w HF into a 17M 28’ PAR EndFedz wire thrown up into a tree
![Page 4: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/4.jpg)
My radios
SDR ‘toy’:Funcube Dongle
![Page 5: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/5.jpg)
My radios
DStar digital radio in my office:DV Dongle
![Page 6: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/6.jpg)
Session Objectives
You will understand: architecture of Analysis Services in tabular modeoptimizing processing performancequery processing architecture
![Page 7: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/7.jpg)
Takeaway
Factors to think through for capacity planning
![Page 8: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/8.jpg)
BI Semantic Model: Architecture
BI Semantic Model
Data model
Business logic and queries
Data access ROLAP MOLAPxVelocityVertiPaq
DirectQuery
MDX DAX
Multi-dimensional
Tabular
Third-partyapplications
ReportingServices Excel PowerPivot
Databases LOB Applications Files OData Feeds Cloud Services
SharePointInsights
![Page 9: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/9.jpg)
VertiPaq Design Principles
1. Performance, Performance, Performance
![Page 10: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/10.jpg)
VertiPaq Design Principles
2. Query Performance >> Processing Performance
![Page 11: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/11.jpg)
VertiPaq Design Principles
3. Accommodate changes without forcing reload, if possible
![Page 12: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/12.jpg)
Encoding
Value EncodingArithmetic for value <-> dataIDGreat for dense value distributionAllows computation on dataIDs
Hash EncodingHash table for value<->dataIDGreat for sparse value distributionRequires decompression for computation
Encoding(per column) Compression
(per segment)
values dataID(int)value hash
![Page 13: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/13.jpg)
Encoding Example
Sales
1
2,000,000,000
Sales
1
2
Hash Encoding Value Encoding
Defined automatically to conserve space, discoverable through DMVs
![Page 14: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/14.jpg)
T2(C1,C2)
P2
Part
itio
n 1
CC1
6
VertiPaq StorageT1
(C1,C2,C3)
Part
itio
n
1Pa
rtit
ion 2
Dictionary
Col. Segment Hierarchy
CC1
Relationship
1
1
2
2
3
3
4
4
5
5
6
Table data stored in segments & dictionaries per columnCalculated columns are stored like regular columnsHierarchies can provide quicker access for querying
Relationship structures are created to accelerate lookups across tables, remove hard coupling between tables
Partitions are a group of segments and intended for data managementAny table can have partitions, defined independently of other tables. Partitions are only for segment data
![Page 15: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/15.jpg)
demo
Inspecting VertiPaq Storage
![Page 16: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/16.jpg)
Processing Architecture
![Page 17: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/17.jpg)
What’s going on?
Network
Memory
CPU
![Page 18: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/18.jpg)
Processing Architecture
Hands On
![Page 19: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/19.jpg)
Processing Phases
Read & Encode DataSegment N
Segment N
CompressSegment N
Read & Encode DataSegment N + 1
CompressSegment N+1
Build calc cols, hier,
relationships
Segment N +1
t
![Page 20: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/20.jpg)
Special case of 3rd segment
Read & Encode DataSegment (until 2*segment_size)
Segment 1 + 2
CompressSegment 1
Read & Encode DataSegment 3
CompressSegment 2
Split
First segment can “stretch” to be twice as large Optimizes for smaller lookup tables
Segment 3
CompressSegment 3
t
![Page 21: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/21.jpg)
Processing - Memory & CPU usage
Network
Memory
CPU
Segment 1 + 2
Compress Segment 1
Compress Segment 2
Read & Encode DataSegment (until 2*segment_size)
Read & Encode DataSegment 3
CompressSegment 3
Calc cols, hier, rels.
Segment 3Split
![Page 22: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/22.jpg)
Controlling Segment Size and CompressionDefaultSegmentRowCount – # of rows in the segment
0 – default: 8M for Analysis Services, 1M for PowerPivotValue must be power of 2, should be at least 1MLarger => generally better compression, faster queries with lower overheadSmaller => smaller working set during processing
ProcessingTimeboxSecPerMRow-1 – default: 10 sec Smaller => greedy algorithm gives most gains in the beginningLarger => almost always better compression, higher query performanceIncrease for large number of columns (~>200)Reported in profiler & DISCOVER_STORAGE_TABLE_COLUMN_SEGMENTS
![Page 23: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/23.jpg)
Processing options
DataT1Data
Hierarchies
CC1
Relationships
Recalc
Full: Data and RecalcDefault: Data and Recalc if neededDefrag: defragment dictionaries for a tableClear: remove dataAdd: creates new partition and merges
![Page 24: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/24.jpg)
Incremental Processing
TypicalCreate new partitions for new dataProcessData on new partition or to reload into existing tablesProcess Recalc to rebuild calc columns, hierarchies, relationships
AvoidMultiple ProcessFull – each causes a Recalc, unless in a single transactional batch
![Page 25: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/25.jpg)
Advanced ProcessingParallel processing
Use single transaction batch for processing tables in parallelNo parallel processing of a table’s partitions in SQL Server 2012
Use ProcessDefrag periodicallyDeleting partitions may leave outdated values in dictionaries
Merge partitions for reducing metadata overhead
Error handling RI violations assigned a blank value, no error during processing
![Page 26: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/26.jpg)
Server Memory Map
VertiPaqPagingMode enables use of system pagingDictionaries need to remain in memory
Server memory
DB1 FE VertiPaq
caches
DB2 DB1~New, not
committed part of the database
VertiPaqMemoryLimit – 60 %
TotalMemoryLimit – 80 %
![Page 27: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/27.jpg)
demo
Inspecting Processing Sequence
![Page 28: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/28.jpg)
Query Processing
![Page 29: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/29.jpg)
Querying
Two formula engines: MDX, DAX
FE calls into VertiPaq to retrieve data Query logic pushed to VertiPaq where possibleA VertiPaq query executes in parallel, one core per segmentOptimized for execution on compressed formatVertiPaq-level caching for chatty FE/VertiPaq communicationEvents in profiler
VertiPaq
MDX DAX
Queries
![Page 30: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/30.jpg)
VertiPaq Query Performance
Scans @ 5B+ rows/s 1B+ rows of data
Scans @ 20B+ rows/s10B+ rows of data
![Page 31: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/31.jpg)
demo
Inspecting Query Evaluation
![Page 32: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/32.jpg)
Rich & Fast
RichSingle threaded per queryDesigned for expressivity
SimpleOne core per segmentOptimized for speed
DAX/MDX VertiPaq Query
More pushed down over time
![Page 33: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/33.jpg)
Why raw speed counts . . .Vertipaq Performance, Performance, Performance
Amdahl's law: Establishes themaximum expected improvementto an overall system when onlypart of the system is improved.
The speedup of a program usingmultiple processors in parallelcomputing is limited by the timeneeded for the sequential fractionof the program.
![Page 34: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/34.jpg)
Session Objectives and Takeaways
Session Objectives: Understand the architecture of Analysis Services in tabular modeUnderstand and optimize processing performanceUnderstand query processing architecture
Takeaways:Factors to think through for capacity planning
![Page 35: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/35.jpg)
Related Content
DBI61-HOL: Developing a Tabular BISM Using SQL Server Data Tools
Microsoft SQL Server: Breakthrough Insights - Credible, Consistent Data
![Page 36: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/36.jpg)
Workbook shown in talk:
Available here:• http://www.powerpivotblog.nl/
/what-is-using-all-that-memory-on-my-analysis-server-instance /bismservermemoryreport
• http://www.powerpivotblog.nl/wp-content /uploads/2012/02/BISMServerMemoryReport.xlsx
• http://www.powerpivotblog.nl/?s=BISMServerMemory
![Page 37: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/37.jpg)
Track Resources
@sqlserver@ms_teched
mvaMicrosoft Virtual Academy
SQL Server 2012 Eval Copy
Get Certified!
Hands-On Labs
![Page 38: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/38.jpg)
Resources
Connect. Share. Discuss.
http://europe.msteched.com
Learning
Microsoft Certification & Training Resources
www.microsoft.com/learning
TechNet
Resources for IT Professionals
http://microsoft.com/technet
Resources for Developers
http://microsoft.com/msdn
![Page 39: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/39.jpg)
Evaluations
http://europe.msteched.com/sessions
Submit your evals online
![Page 40: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/40.jpg)
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to
be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS
PRESENTATION.
![Page 41: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/41.jpg)
Best Practices for Tabular ModelsWorkspace DB for data-driven design experience
BI Dev Studio
Analysis Services
Workspace DB
Process Tables/Partitions
Analysis Services
Deploy project
DB
![Page 42: Optimizing Your BI Semantic Model for Performance and Scale Dave Wickert (AE7TD) dwickert@microsoft.com Principal Program Manager SQL Server Business Intelligence](https://reader036.vdocuments.net/reader036/viewer/2022062321/56649eb75503460f94bc1354/html5/thumbnails/42.jpg)
Options for working with large datasets
1. Point to smaller/empty DB2. Filter large tables to reduce
rows