u-sql query execution and performance tuning

40
Michael Rys Principal Program Manager Microsoft Big Data @MikeDoesBigData, [email protected] U-SQL Query Execution and Performance Tuning

Upload: michael-rys

Post on 08-Jan-2017

1.681 views

Category:

Data & Analytics


3 download

TRANSCRIPT

Page 1: U-SQL Query Execution and Performance Tuning

Michael RysPrincipal Program ManagerMicrosoft Big Data@MikeDoesBigData, [email protected]

U-SQL Query Execution and Performance Tuning

Page 2: U-SQL Query Execution and Performance Tuning

Session Objectives And TakeawaysSession Objective(s): • Understand the U-SQL Query execution• Be able to understand and improve the U-SQL query plan• Know how to write more efficient U-SQL scripts

Key Takeaways:• U-SQL is designed for scale-out• U-SQL provides scalable execution of user code• U-SQL has a tool set that can help you analyze and improve your scalability and performance

Page 3: U-SQL Query Execution and Performance Tuning

Agenda• Job Execution Experience and Investigations

Query ExecutionStage GraphDryad crash courseJob MetricsResource Planning

• Partitioning AnalysisAnalyze the critical pathHeat MapCritical PathData Skew

• Tuning / OptimizationsData PartitioningPartition EliminationPredicate PushingColumn PruningSome Data HintsUDOs can be evilINSERT optimizations

U-SQL Query Execution and Performance Tuning

Page 4: U-SQL Query Execution and Performance Tuning

Job Execution Experience and Investigations

Page 5: U-SQL Query Execution and Performance Tuning

12Expression-flow Programming Style

Automatic "in-lining" of U-SQL expressions – whole script leads to a single execution model.

Execution plan that is optimized out-of-the-box and w/o user intervention.

Per job and user driven level of parallelization.

Detail visibility into execution steps, for debugging.

Heatmap like functionality to identify performance bottlenecks.

Page 6: U-SQL Query Execution and Performance Tuning

Job Scheduling: Job States

Preparing

Queued

Running

Finalizing

Ended(Succeeded, Failed,

Cancelled)

NewCompiling

QueuedSchedulin

gStarting

Running

Ended

What you see in the UX

Underlying Job State

The script is being compiled by the Compiler Service

All jobs enter the queue.

Are there enough ADLAUs to start the job?

If yes, then allocate those ADLAUs for the job

The U-SQL runtime is now executing the code on 1 or more ADLAUs or finalizing the outputs

The job has concluded.

Page 7: U-SQL Query Execution and Performance Tuning

The USQL Compilation

U-SQL C# user code

C++ system code

Algebra

other files(system files, deployed

resources)

managed dll

Unmanaged dll

Input script

Compilation output (in job folder)

Compiler & Optimizer

Files

Meta Data

Service

Deployed to vertices

Page 8: U-SQL Query Execution and Performance Tuning

Distributing Work into VertexesSome fixed amount of

work

Each square is called a “vertex”

Each vertex represents a fraction of the work

Page 9: U-SQL Query Execution and Performance Tuning

U-SQL Query Execution Physical plans vs. Dryad stage graph…

Page 10: U-SQL Query Execution and Performance Tuning

Super Vertex Stage Details252 Pieces of work

AVG Vertex execution time

4.3 Billion rows

Data Read & Written

Page 11: U-SQL Query Execution and Performance Tuning

U-SQL Query ExecutionDryad as an art form…

Page 12: U-SQL Query Execution and Performance Tuning

13

U-SQL Query ExecutionRedefinition of big-data…

Page 13: U-SQL Query Execution and Performance Tuning

14

U-SQL Query ExecutionRedefinition of big-data…

Page 14: U-SQL Query Execution and Performance Tuning

Performance

Analyzing the Critical Path

Page 15: U-SQL Query Execution and Performance Tuning

16

U-SQL Performance AnalysisAnalyze the critical path, heat maps, playback, and runtime metrics on every vertex…

Page 16: U-SQL Query Execution and Performance Tuning

Partitioning during Processing

Analyzing Data Skew

Page 17: U-SQL Query Execution and Performance Tuning

What is Data Skew? Some data points

are much more common than others

data may be distributed such that all rows that match a certain key go to a single vertex

imbalanced execution, vertex time out.

Californ

iaFlo

rida

Ohio

North C

arolina

Washing

ton

Indian

a

Marylan

d

Colorad

o

Louisia

na

Oklaho

ma

Mississ

ippi

Utah

Nebras

kaHaw

aii

Rhode

Islan

d

South

Dakota

District

of Colu

mbia0

5,000,000

10,000,000

15,000,000

20,000,000

25,000,000

30,000,000

35,000,000

40,000,000Population by State

Page 18: U-SQL Query Execution and Performance Tuning

Low Distinctiveness Keys Keys with small

selectivity can lead to large vertices even without skew

Solutions: Find more distinctive keys Implement

aggregation/reducer recursively if possible

@rows = SELECT Gender, AGG<MyAgg>(…) AS Result

FROM @HugeInputGROUP BY Gender;

Gender==Male Gender==Female

@HugeInput

Vertex 0 Vertex 1

Page 19: U-SQL Query Execution and Performance Tuning

Vertex 4

Vertex 3 Vertex 2 Vertex 1

Vertex 1

Non-Recursive vs Recursive SUM20

1 2 3 4 5 6 7 8 36

1 2 3 4 5 6 7 8

6 15 15

36

Page 20: U-SQL Query Execution and Performance Tuning

U-SQL Partitioning during Processing Data Skew

Page 21: U-SQL Query Execution and Performance Tuning

U-SQL PartitioningData Skew – Recursive Reducer

// Metrics per domain@Metric = REDUCE @Impressions ON UrlDomain USING new Bing.TopNReducer(count:10) ;

// …

Inherent Data Skew

[SqlUserDefinedReducer(IsRecursive = true)]public class TopNReducer : IReducer{ public override IEnumerable<IRow> Reduce(IRowset input, IUpdatableRow output) { // Compute TOP(N) per group // … }}

Recursive• Allow multi-stage aggregation trees• Requires same schema (input => output) • Requires associativity:• R(x, y) = R( R(x), R(y) )

• Default = non-recursive• User code has to honor recursive

semantics

www.bing.combrought to a single vertex

Page 22: U-SQL Query Execution and Performance Tuning

U-SQL Partitioning during ProcessingPartitioning – Combiner Modes

// Existing Watson hits@DeDup = COMBINE @Exceptions AS L WITH @WatsonBuckets AS R ON L.AppId WITH R.AppId USING new Windows.WatsonDedupCombiner() ;

// …

[SqlUserDefinedCombiner(Mode = CombinerMode.Right)]public class WatsonDedupCombiner : ICombiner{ public override IEnumerable<IRow> Combine(IRowset left, IRowset right, IUpdatableRow output) { // DeDup against existing Call Stacks // … }}

CombinerMode•Allow parallelism even within a partition public enum CombinerMode { Inner, /// Inner join - both row level Left, /// Left Group Join - left row level Right, /// Right Group Join - right row level Full /// Full Group Join - none row level

• Default = Full Group Join• User code has to honor row level semantics

Row Level Combiner

X

IDother columns

XQ

QFQ

XXZ

FFX

LEFT

X

ID other columns

Q

QF

Q

X

Z

FFX

RIGHT

F

Z

M1

M2

M3

M4

Enables Broadcast JOIN

Page 23: U-SQL Query Execution and Performance Tuning

Data Storage Files Tables

Unstructured Data in files Row-oriented files are split into 250MB

extents Should be uploaded with row-boundary alignment

(CR/LF as boundary) Allows parallel extraction

Binary files (incl XML/JSON) will still be in extents Have to be processed in single vertex extractor with

atomicFileProcessing=true. Use File Sets to provide semantic partition

pruning Tables

Clustered Index (row-oriented) storage Vertical and horizontal partitioning Statistics for the optimizer (CREATE

STATISTICS) Native scalar value serialization

Page 24: U-SQL Query Execution and Performance Tuning

Data PartitioningFilesTables

Partitioning of unstructured data Use File Sets to provide semantic partition pruning

Table Partitioning Fine grained (horizontal) partitioning

Partitions within a file (together with clustering) to keep same data values close

Choose for: Join alignment, partition size, filter selectivity

Coarse grained (vertical) partitioning Partition is addressable in language Query predicates will allow partition pruning

Partition Scheme

When to use?

HASH(keys)DIRECT HASH Exact control of hash bucketRANGE(keys) Keeps ranges togetherROUND ROBIN To get equal distribution (if others give skew)

Page 25: U-SQL Query Execution and Performance Tuning

Optimizations

Data Partitioning and the optimizer

Page 26: U-SQL Query Execution and Performance Tuning

// Unstructured Files (24 hours daily log impressions)@Impressions = EXTRACT ClientId int, Market string, OS string, ... FROM @"wasb://ads@wcentralus/2015/10/30/{*}.nif" FROM @"wasb://ads@wcentralus/2015/10/30/{Market:*}_{*}.nif" ;

// …

// Filter to by Market@US = SELECT * FROM @Impressions WHERE Market == "en" ;

U-SQL OptimizationsPartition Elimination – Unstructured Files

Partition Elimination• Even with unstructured files!• Leverage Virtual Columns (Named)• Avoid unnamed {*}

• WHERE predicates on named virtual columns• That binds the PE range during compilation time• Named virtual columns without predicate = error

• Design directories/files with PE in mind• Design for elimination early in the tree, not in the leaves

Extracts all files in the folder

Post filter = pay I/O cost to drop most data

PE pushes this predicate to the EXTRACT

EXTRACT now only reads “en” files!

en_10.0.nif

en_8.1.nif

de_10.0.nif

jp_7.0.nif

de_8.1.nif

../2015/10/30/

Page 27: U-SQL Query Execution and Performance Tuning

// TABLE(s) - Structured Files (24 hours daily log impressions)CREATE TABLE Impressions (Day DateTime, Market string, ClientId int, ... INDEX IX CLUSTERED(Market, ClientId) PARTITIONED BY BUCKETS (Day) HASH(Market, ClientId) INTO 100 );

DECLARE @today DateTime = DateTime.Parse("2015/10/30");

// Market = Vertical PartitioningALTER TABLE Impressions ADD PARTITION (@today);

// …

// Daily INSERT(s)INSERT INTO Impressions(Market, ClientId) PARTITION(@today) SELECT * FROM @Q ;

// …

// Both levels are elimination (H+V)@Impressions = SELECT * FROM dbo.Impressions WHERE Market == "en" AND Day == @today ;

U-SQL OptimizationsPartition Elimination – TABLE(s) Partition Elimination

• Horizontal and vertical partitioning• Horizontal is traditional within file (range, hash, robin)• Vertical is across files (bucketing)

• Immutable file system• Design according to your access patterns

Enumerate all partitions filtering for today

30.ss

30.1.ss

29.ss28.ss

29.1.ss

Impressions

deen

jp

de

PE across files + within each file

Page 28: U-SQL Query Execution and Performance Tuning

@Inpressions = SELECT * FROM searchDM.SML.PageView(@start, @end) AS PageView OPTION(LOWDISTINCTNESS=Query) ;

// Q1(A,B)@Sessions = SELECT ClientId, Query, SUM(PageClicks) AS Clicks FROM @Impressions GROUP BY Query, ClientId ;

// Q2(B)@Display = SELECT * FROM @Sessions INNER JOIN @Campaigns ON @Sessions.Query == @Campaigns.Query ;

U-SQL OptimizationsPartitioning – Minimize (re)partitions

Input must be partitioned on: (Query)

Input must be partitioned on:(Query) or (ClientId) or (Query,

ClientId)

Optimizer wants to partition only onceBut Query could be skewed

Data Partitioning• Re-Partitioning is very expensive• Many U-SQL operators can handle multiple partitioning

choices• Optimizer bases decision upon estimations

Wrong statistics may result in worse query performance

Page 29: U-SQL Query Execution and Performance Tuning

// Unstructured (24 hours daily log impressions)@Huge = EXTRACT ClientId int, ... FROM @"wasb://ads@wcentralus/2015/10/30/{*}.nif" ;

// Small subset (ie: ForgetMe opt out)@Small = SELECT * FROM @Huge WHERE Bing.ForgetMe(x,y,z) OPTION(ROWCOUNT=500) ;

// Result (not enough info to determine simple Broadcast join)@Remove = SELECT * FROM Bing.Sessions INNER JOIN @Small ON Sessions.Client == @Small.Client ;

U-SQL OptimizationsPartitioning - Cardinality

Broadcast JOIN right?

Broadcast is now a candidate.

Wrong statistics may result in worse query performance

Optimizer has no stats this is small...

Page 30: U-SQL Query Execution and Performance Tuning

Optimizations

Predicate pushing and user code

Page 31: U-SQL Query Execution and Performance Tuning

// Bing impressions@Impressions = SELECT * FROM searchDM.SML.PageView(@start, @end) AS PageView ;

// Compute sessions@Sessions = REDUCE @Impressions ON Client, Market READONLY Market USING new Bing.SessionReducer(range : 30) ;

// Users metrics@Metrics = SELECT * FROM @Sessions WHERE Market == "en-us" ;

// …

Microsoft Confidential

U-SQL OptimizationsPredicate pushing – UDO pass-through columns

Page 32: U-SQL Query Execution and Performance Tuning

// Bing impressions@Impressions = SELECT * FROM searchDM.SML.PageView(@start, @end) AS PageView ;

// Compute page views@Impressions = PROCESS @Impressions READONLY Market PRODUCE Client, Market, Header string USING new Bing.HtmlProcessor() ;

@Sessions = REDUCE @Impressions ON Client, Market READONLY Market USING new Bing.SessionReducer(range : 30) ;

// Users metrics@Metrics = SELECT * FROM @Sessions WHERE Market == "en-us" ;

Microsoft Confidential

U-SQL OptimizationsPredicate pushing – UDO row level processors

public abstract class IProcessor : IUserDefinedOperator{ /// <summary/> public abstract IRow Process(IRow input, IUpdatableRow output);}

public abstract class IReducer : IUserDefinedOperator{ /// <summary/> public abstract IEnumerable<IRow> Reduce(IRowset input, IUpdatableRow output);}

Page 33: U-SQL Query Execution and Performance Tuning

// Bing impressions@Impressions = SELECT Client, Market, Html FROM searchDM.SML.PageView(@start, @end) AS PageView ;

// Compute page views@Impressions = PROCESS @Impressions PRODUCE Client, Market, Header string USING new Bing.HtmlProcessor() ;

// Users metrics@Metrics = SELECT * FROM @Sessions WHERE Market == "en-us" && Header.Contains("microsoft.com") AND Header.Contains("microsoft.com") ;

U-SQL OptimizationsPredicate pushing – relational vs. C# semantics

Page 34: U-SQL Query Execution and Performance Tuning

Optimizations

Column pruning

Page 35: U-SQL Query Execution and Performance Tuning

// Bing impressions@Impressions = SELECT * FROM searchDM.SML.PageView(@start, @end) AS PageView ;

// Compute page views@Impressions = PROCESS @Impressions PRODUCE * REQUIRED ClientId, HtmlContent(Header, Footer) USING new Bing.HtmlProcessor() ;

// Users metrics@Metrics = SELECT ClientId, Market, Header FROM @Sessions WHERE Market == "en-us" ;

U-SQL OptimizationsColumn Pruning and dependencies

C H M

C H M

C H M

Column Pruning• Minimize I/O (data shuffling)• Minimize CPU (complex processing, html)• Requires dependency knowledge:• R(D*) = Input ( Output )

• Default no pruning• User code has to honor reduced columns

A B C D E F G J KH I … M … 1000

Page 36: U-SQL Query Execution and Performance Tuning

General UDO Warning and other Tips • Use SELECT with UDFs instead of PROCESS• Use User-defined Aggregators instead of REDUCE• Hint Cardinality if you use CROSS APPLY and it

does chose the wrong plan• Avoid ORDER BY unless needed (OUTPUT, “Top N

Rows”)• Learn to use Windowing Functions (OVER

expression)• Use SQL.MAP and SQL.ARRAY instead of C#

Dictionary and array

Page 37: U-SQL Query Execution and Performance Tuning

INSERT Multiple INSERTs into same table

• Generates separate file per insert in physical storage:• Can lead to performance degradation

• Recommendations:• Try to avoid small inserts• Rebuild table after frequent insertions

with:ALTER TABLE T REBUILD;

Page 38: U-SQL Query Execution and Performance Tuning

In Review: Session Objectives & TakeawaysSession Objective(s): • Understand the U-SQL Query execution• Be able to understand and improve the U-SQL query plan• Know how to write more efficient U-SQL scripts

Key Takeaways:• U-SQL is designed for scale-out• U-SQL provides scalable execution of user code• U-SQL has a tool set that can help you analyze and improve your scalability and performance

Page 39: U-SQL Query Execution and Performance Tuning

Additional Resources

Blogs and community page: http://usql.io http://blogs.msdn.com/b/visualstudio/ http://azure.microsoft.com/en-us/blog/topics/big-data/ https://channel9.msdn.com/Search?term=U-SQL#ch9Search

Documentation and articles: http://aka.ms/usql_reference https://azure.microsoft.com/en-us/documentation/services/dat

a-lake-analytics/

https://msdn.microsoft.com/en-us/magazine/mt614251

ADL forums and feedback http://aka.ms/adlfeedback https://

social.msdn.microsoft.com/Forums/azure/en-US/home?forum=AzureDataLake

http://stackoverflow.com/questions/tagged/u-sql

Page 40: U-SQL Query Execution and Performance Tuning

© 2016 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.