extra performance out of thin air
TRANSCRIPT
![Page 1: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/1.jpg)
Konstantine Krutiy Principal Engineer, Crew Lead
![Page 2: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/2.jpg)
PATH TO EXTRA PERFORMANCE
Eliminate unneeded work § Choose data types wisely
Eliminate unneeded waits § Reduce number of locks
Make system operate in more efficient way § Optimize BIOS settings § Stay in same “technology slice” § Make sure you have enough RAM
![Page 3: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/3.jpg)
ART OF CHOOSING DATATYPES
![Page 4: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/4.jpg)
WHY DATA TYPE MATTERS ?
![Page 5: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/5.jpg)
WHY DATA TYPE MATTERS ?
Fastest CPU today is 3.7 GHz
It takes 1 / 3,700,000,000 of a second
to do single operation
![Page 6: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/6.jpg)
WHY DATA TYPE MATTERS ?
Fastest CPU today is 3.7 GHz
It takes 1 / 3,700,000,000 of a second
to do single operation
“BIG DATA” record set starts from
100 billion records
![Page 7: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/7.jpg)
WHY DATA TYPE MATTERS ?
Fastest CPU today is 3.7 GHz
It takes 1 / 3,700,000,000 of a second
to do single operation
“BIG DATA” record set starts from
100 billion records
Processing time
1 / 3,700,000,000 sec X 100,000,000,000 = 27 sec
![Page 8: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/8.jpg)
DO YOU NEED TO STORE DATA SAME WAY IT IS PRESENTED ?
![Page 9: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/9.jpg)
DO YOU NEED TO STORE DATA SAME WAY IT IS PRESENTED ?
Presentation: $395.17
![Page 10: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/10.jpg)
DO YOU NEED TO STORE DATA SAME WAY IT IS PRESENTED ?
Presentation: $395.17
Data: 395.17
![Page 11: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/11.jpg)
DO YOU NEED TO STORE DATA SAME WAY IT IS PRESENTED ?
Presentation: $395.17
Data: 395.17
Storage: Store as Money Data type: MONEY Internal data type: NUMERIC(18,4)
Storage: Store as numeric Data type: NUMERIC Internal data type: NUMERIC(37,15)
Storage: Store as integer Data type: INT Internal data type: INT
![Page 12: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/12.jpg)
DO YOU NEED TO STORE DATA SAME WAY IT IS PRESENTED ?
Presentation: $395.17
Data: 395.17
Storage: Store as Money Data type: MONEY Internal data type: NUMERIC(18,4)
Storage: Store as numeric Data type: NUMERIC Internal data type: NUMERIC(37,15)
Storage: Store as integer Data type: INT Internal data type: INT
![Page 13: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/13.jpg)
DATA TYPE BENCHMARK DATA
![Page 14: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/14.jpg)
DATA TYPE BENCHMARK AVERAGES IN SEC
27.2 29.7
37
0
5
10
15
20
25
30
35
40
INT NUMERIC(18,5) NUMERIC(37,15)
![Page 15: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/15.jpg)
MAKING RIGHT CHOICES
• If you can store data as INTEGER • Choose INTEGER
• If your data fits into 18 digits of PRECISION • Choose NUMERIC(18)
• If your data larger then 18 digits of PRECISION • Choose NUMERIC(your-desired-precision)
Vertica default for NUMERIC is NUMERIC(37,15)
![Page 16: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/16.jpg)
ELIMINATING UNNECESSARY LOCKING
![Page 17: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/17.jpg)
LOCKING BEHAVIOR
AUTOCOMMIT = ON (jdbc driver default) § Each statement treated as complete transaction § When statement completes changes automatically
committed to database
AUTOCOMMIT = OFF § Transaction continue until manually run COMMIT or
ROLLBACK § Locks kept on objects for transaction duration
![Page 18: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/18.jpg)
CONTROLLING AUTOCOMMIT STATE
JAVA: conn = DriverManager.getConnection("jdbc:vertica://DBHost:5433/MyDB", myProperties); // get the state of the auto commit parameter System.out.println("Autocommit state: " + conn.getAutoCommit()); // Change the auto commit state to false conn.setAutoCommit(false);
SQL:
![Page 19: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/19.jpg)
IMPACT ON LOCK COUNTS BY CHANGING AUTOCOMMIT SETTING TO OFF
![Page 20: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/20.jpg)
HOW TO DISABLE – OBVIOUS METHOD
![Page 21: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/21.jpg)
HOW TO DISABLE – BETTER METHOD
![Page 22: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/22.jpg)
BIOS SETTINGS OPTIMIZATIONS
![Page 23: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/23.jpg)
WHAT IS TUNABLE IN BIOS?
![Page 24: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/24.jpg)
HOW TO TUNE ?
http://h10032.www1.hp.com/ctg/Manual/c01804533.pdf
![Page 25: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/25.jpg)
DOES IT REALLY MATTER ?
0
100
200
300
400
500
600
700
800
900
1000
DSS BIOS settings with 1x DRAM refresh rate
DSS BIOS settings with 4x DRAM refresh rate
HPC BIOS settings with 4x DRAM refresh rate
HPC + HyperThreading BIOS settings with 4x DRAM
refresh rate
HPC - NO TurboBoost BIOS settings with 4x DRAM
refresh rate
Sec
DSS BIOS se)ngs with 1x DRAM refresh rate 738.949439 DSS BIOS se)ngs with 4x DRAM refresh rate 745.111176 HPC BIOS se)ngs with 4x DRAM refresh rate 552.148285 HPC + HyperThreading BIOS se)ngs with 4x DRAM refresh rate 877.838469 HPC -‐ NO TurboBoost BIOS se)ngs with 4x DRAM refresh rate 561.260084
Performance increase potential
about 40%
![Page 26: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/26.jpg)
WHAT TUNING DOC SAYS ?
![Page 27: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/27.jpg)
STAYING IN THE SAME “TECHNOLOGY SLICE”
![Page 28: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/28.jpg)
WHAT I WILL BE SLICING THROUGH ???
CPU and chipset
Hardware
Operating System (OS)
Database Management System (DBMS)
![Page 29: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/29.jpg)
WHAT IS “TECHNOLOGY SLICE” ANYWAY ???
CPU Gen3
CPU Gen4
Server Gen-B
OS v. 36
DBMS v. 6
Server Gen-C
OS v. 37
DBMS v. 7
CPU Gen5
Server Gen-D
CPU Gen6
CPU Gen7
Server Gen-E
Srv GenF
OS v. 38
Server Gen-A
OS v. 35
OS v. 34
DBMS v. 5
DBMS v. 4
DBMS v. 3
![Page 30: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/30.jpg)
WHAT IS “TECHNOLOGY SLICE” ANYWAY ???
CPU Gen3
CPU Gen4
Server Gen-B
OS v. 36
DBMS v. 6
Server Gen-C
OS v. 37
DBMS v. 7
CPU Gen5
Server Gen-D
CPU Gen6
CPU Gen7
Server Gen-E
Srv GenF
OS v. 38
Server Gen-A
OS v. 35
OS v. 34
DBMS v. 5
DBMS v. 4
DBMS v. 3
![Page 31: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/31.jpg)
COMMON “TECHNOLOGY SLICE” TRAP
CPU Gen3
CPU Gen4
✔
Server Gen-B
OS v. 36
DBMS v. 6
Server Gen-C
✔
OS v. 37
✔
DBMS v. 7
✔
CPU Gen5
Server Gen-D
CPU Gen6
CPU Gen7
Server Gen-E
Srv GenF
OS v. 38
Server Gen-A
OS v. 35
OS v. 34
DBMS v. 5
DBMS v. 4
DBMS v. 3
![Page 32: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/32.jpg)
COMMON “TECHNOLOGY SLICE” TRAP
CPU Gen3
CPU Gen4
✔
Server Gen-B
OS v. 36
DBMS v. 6
Server Gen-C
✔
OS v. 37
✔
DBMS v. 7
✔
CPU Gen5
Server Gen-D
CPU Gen6
CPU Gen7
Server Gen-E
Srv GenF
OS v. 38
Server Gen-A
OS v. 35
OS v. 34
DBMS v. 5
DBMS v. 4
DBMS v. 3
?
?
![Page 33: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/33.jpg)
SYMPTOMS OF “TECHNOLOGY SLICE” ISSUES
System AVG: 57.90 Nice AVG: 46.56 System AVG > Nice AVG System AVG / Nice AVG = 1.24
System AVG: 11.19 Nice AVG: 57.38 System AVG < Nice AVG System AVG / Nice AVG = 0.19
![Page 34: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/34.jpg)
“TECHNOLOGY SLICE” PERFORMANCE IMPACT
0
20
40
60
80
100
120
140
different “TECHNOLOGY SLICE” kernel proper “TECHNOLOGY SLICE” kernel
Sec
![Page 35: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/35.jpg)
SUFFICIENT RAM CALCULATIONS
![Page 36: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/36.jpg)
DO I REALLY NEED MORE RAM ?
select event_type, count(1) from query_events group by event_type order by 2 desc;
Spilled events are very good indication of queries not fitting in
RAM
![Page 37: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/37.jpg)
HOW I CAN QUANTIFY IMPACT ?
select 'event_timestamp' as timestamp_type, min(event_timestamp) as min_timestamp, max(event_timestamp) as max_timestamp from query_events union select 'query_timestamp' as timestamp_type, min(start_timestamp) as min_timestamp, max(start_timestamp) as max_timestamp from query_requests;
System tables in Vertica have individual rolling window. Make sure you understand relation of
histories available.
![Page 38: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/38.jpg)
HOW I CAN QUANTIFY IMPACT ? CONT.
select spilled_queries, total_qieries, round( spilled_queries / total_qieries * 100 , 2 ) as spilled_queries_percent from (select count(1) as total_qieries from query_requests where request_type = 'QUERY' and start_timestamp > (select min(event_timestamp) from query_events)) query_data, (select count(1) as spilled_queries from (select session_id, transaction_id, statement_id from query_events where event_type ilike '%SPILLED%' group by session_id, transaction_id, statement_id) spill_data) spill_data2;
Amount of spilled queries in relation to entire query volume.
![Page 39: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/39.jpg)
CAN MY SPILLED DATA FIT IN TO RAM ?
select min(counter_value) as min_bytes_spilled, max(counter_value) as max_bytes_spilled, avg(counter_value) as avg_bytes_spilled from execution_engine_profiles where counter_name = 'bytes spilled' and counter_value > 0;
Understanding size of spillage to disk.
![Page 40: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/40.jpg)
WHO CAUSING SPILLS ?
select user_name, count(1) as spill_event_count from query_events where event_type ilike '%SPILLED%' group by user_name order by 2 desc;
In Vertica RAM allocated to queries through resource pools. Resource
pools connected to users. Knowing user will point us to resource pool,
which needs tuning.
![Page 41: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/41.jpg)
WHAT I SHOULD TUNE ?
select distinct resource_pool from users where user_name in ('peter', 'john');
Identified resource pool with spilled queries. Now we know
what to tune.
![Page 42: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/42.jpg)
The resource pool parameters of MEMORYSIZE and
PLANNEDCONCURRENCY provide the options that let you tune the target
memory allocated to queries.
WHAT I SHOULD CHANGE ?
HP Vertica Analytics Platform Version 7.1.x Documentation Administrator's Guide Managing the Database Managing Workloads Resource Pool Architecture Target Memory Determination for Queries in Concurrent Environments
![Page 43: Extra performance out of thin air](https://reader033.vdocuments.net/reader033/viewer/2022051520/58ef02c21a28ab4f0b8b4655/html5/thumbnails/43.jpg)
Q & A