the (un)loved child of generation cloud - oracle on vmware...the (un)loved child of generation cloud...

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 2/50

The (Un)loved Child of Generation Cloud - Oracle on VMware

Martin Klier

Performing Databases GmbHMitterteich / Germany

Das (un)geliebte Kind der Generation Cloud - Oracle auf VMware

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 3/50

Basics

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 4/50

a Type-I-Hypervisor

VMware vSphere

CPUExtensions(VT-x,SIMD, etc.)

Para-VirtualizedController

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 5/50

TEMP

Oracle Architecture (simplified)

Listener:1521

PMON

SGA

ServerServer

Buffer Cache

Shared Pool

Log Buffer

ARCnSMON

LGWR

DBWn

PGAs

CKPT

Server

Online

Redo

Logs

ArchivedRedoLogs

Tablespace Undo Tbs.

Blocks

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 6/50

NUMA Architecture

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 7/50

Core(s)+SharedCache, LLC

Core(s)+SharedCache, LLC

QPI-C

Core(s)+SharedCache, LLC

Core(s)+SharedCache, LLC

PCIePCIe

IMC IMCQPI-C

IMCQPI-CQPI-CIMC

NonUnifiedMemoryAccess

NUMA

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 8/50

Oracle & NUMA

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 9/50

_enable_NUMA_support = TRUE

MOS Doc ID 864633.1

• Multiple Buffer Caches

• Striped pools

=> cross context :((

=> pool access :(

Oracle & NUMA

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 10/50

13GB+13GB=26 GB

One buffer cache for each node

Oracle & NUMA

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 11/50

Oracle & NUMA

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 12/50

Suggestions

• Useful in big environments only (think: DB consolidation)

• Test thoroughly and quantify use vs. effort (think: bugs)

• Use only if necessary to achieve your goals!

Oracle NUMA aware

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 13/50

Hyperthreading

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 14/50

HyperThreading (HTT)

CoreProcessing big workloads= large input queue

Performance =Throughput

125% :)Disclaimer:Very simplified example!

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 15/50

OLTP & HTT

A B

B

A

Core

Tim

e

x

x

x

OLTP workloads= empty input queue

Performance =Response Time

50% :(

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 16/50

OLTP & HTT

Core

Tim

e

A

A

DisableHyperThreading for OLTP systems!

- in hardware BIOS- on VMware level („Latency Sensitivity“ high)

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 17/50

VMware vSphere (ESXi)CPU Usage

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 18/50

My Test Setup

• HP ProLiant DL380 G6• 2x 64GB RAM• 2 CPUs, 2x 6 cores = 2 NUMA nodes

Intel(R) Xeon(R) CPU X5670 @ 2.93GHz boost 3.33GHz„Westmere EP“ 32nm technology• HyperThreading disabled (BIOS)

• VMware vSphere ESXi 6.0 U2• 2 VMs / Oracle Linux 7.4 w/ UEK Kernel

VMware CPU Usage

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 19/50

VMware CPU Usage

1 Worker6 vCPU 22% each

OS Metric / Linux top20% CPU

2w / 44%

3w / 65%

4w / 85%4w / 85%

5w / 95%6w / 100%

OS: 40%

OS: 50%

OS: 65%

OS: 80%

OS: 95%

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 20/50

VMware CPU Usage

6 cores active

6 cores idle~0% „CPU Ready“

100% CPU

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 21/50

CPU Ready=

Wait for free CPU time

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 22/50

VMware CPU Usage

6 cores 100%

Not in graph:0% „CPU Ready“

12 cores 100%

6 cores idle

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 23/50

Green IT :)

6 cores 100% 12 cores 100%

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 24/50

VMware CPU Usage8 vCores

13 workers to fill up to 100% in top

Each worker:

while true( bc < 5.5*7.7 etc.

)

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 25/50

VMware CPU UsageVM w/ 8 vCores13 workers

> 1 NUMA nodestarts switchingcontexts :(

6 cores idle

VM w/ 6 vCores6 workers

runs on oneNUMA nodeclean & tidy

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 26/50

Oracle Workload on ESXi

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 27/50

Oracle Stress in VM

loop( select 1 row;

update this row;commit)

6 workerseach on a different set of 10k rows

runs ~20min until Free Buffer Waits

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 28/50

Anti-Load in VM

14 workers (bc)55% avg. loadover 12 vCoreswe know:Will heavily switchCPU context!

Not nearly full power(CPU load 55%)

Owner will feel no guilt

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 29/50

Oracle vs. Anti-LoadAnti-Load VM CPU Oracle VM CPU

Not nearly full power(CPU load 55%)

But „steals“ 40%CPU time

>30%„CPU Ready“

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 30/50

So far, much wasmeasured w/ help of the

VMware Admin:(

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 31/50

SMS=

Surrogate Measure

Sucks

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 32/50

Let's do it ourselvesLet's do what we do best

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 33/50

Use Oracle

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 34/50

AWR Diff Report

Period 1 Period 2

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 35/50

AWR Diff Report

409.888,3 BG / 5,9 s = 69.473 BG/s234.796.8 BG / 5,5 s = 42.690 BG/s = 61%

39% less buffer getswith SAME CPU TIMEand WORKLOAD

(Linux sees 100% all times)

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 36/50

and use your brain!

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 37/50

Simple Benchmark

Guess,what happenedhere? :(

First value: ID of test iteration### bar: VisualizationDecimal value: Seconds for

10.000 x SELECT+UPDATEof different single rows(Avg. per iteration over all workers,all workers on different rows)

Simulated User Experience of aDB on VMware (if built poorly)

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 38/50

Knowledge is importantProof is better

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 39/50

Prepare to Prove

What you need in your VM (Linux)

• VMware tools (VMware recommends package from your distributor)• Phython environment (usually present)• git (or copy yourself)• build environment like make (Oracle needs that anyway)• vmguestlib with vmguest-stats (made by Dag Wieers)

uses an API provided by vSphere (via VMW tools)

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 40/50

vmguest-stats

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 41/50

vmguest-stats

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 42/50

Countermeasures

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 43/50

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 44/50

Ten Commandmentsfor OLTP on VMware

And the Lord said:

1) Thou shalt not overcommit your machine!2) Thou shalt not use Hyperthreading,

because HTT is CPU overcommitment3) Thou shalt make 100% CPU + RAM reservations4) Thou shalt disable CPU Hot Add (enable breaks vNUMA)5) Thou shalt disable RAM Hot Add (enable breaks vNUMA)6) Thou shalt not use more vCores in your VM than a NUMA node

of the host has7) Thou shalt not use more RAM in your VM than a NUMA node

of the host has8) Thou shalt configure VMs to Latency Sensitivity „HIGH“9) Thou shalt disable all BIOS based Energy Saving Options10) Thou shalt use Paravirtual Drivers at any cost

And peace be on earth.

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 45/50

(More) RessourceConsumption

Makes you (more)vulnerable

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 46/50

Speaker

● Martin Klier● Solution Architect and

Database Expert

● My focus– Performance Optimization– High Availability– Architecture DBMS

● Linux since 1997● Oracle Database since 2003

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 47/50

Speaker

● Meet & Greet

● Contact: martin.klier@performing-db.com

● Weblog: http://www.usn-it.de (English)

RegionalgruppenFachkonferenzen

Las Vegas, April 2018Tel Aviv, January 2018

Frankfurt, December 2018

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 48/50

Performing Databases

● Experts for Database Technology

– Concept– Planning & Sizing– Licensing– Implementation and Troubleshooting

● Get in touch

– Performing Databases GmbHWiesauer Straße 2795666 Mitterteich, GERMANY

– Web: http://www.performing-databases.com– Twitter: @PerformingDB

@MartinKlierDBA (Un)loved Child of Gen. Cloud - Oracle on VMware 50/50

Q & A

Download my Presentations and Whitepapershttp://www.performing-databases.com