best practices for deploying mysql on the solaris platform presentation 1

8/14/2019 Best Practices for Deploying MySQL on the Solaris Platform Presentation 1

1/49

Presented by,

MySQL AB &OReilly Media, Inc.

Best Practices For

Deploying MySQL on

Solaris

Ritu KambojJenny Chen


2/49

Agenda

MySQL -Solaris Integration MySQL High Availability Data Service

Consolidate MySQL Deployment Using SolarisContainers

DTrace (Jenny Chen)


3/49

MySQL-Solaris Integration


4/49

Optimized MySQL on Open Solaris

MySQL 5.0.45 (32bit) integrated with OpenSolaris build 76

SXDE 01/08 MySQL 5.0.45 (64bit) integrated with Open

Solaris build 87 SXCE


5/49

Layout of MySQL on Open Solaris

MySQL 5.0.45 packagesDefault data directory

/var/mysql/5.0/data Default configuration directory

/etc/mysql/5.0 Installation directory /usr/mysql/5.0/

Latest version accessible from /usr/mysql/bin Symbolic link from all directories involved in 5.0.45release to corresponding /usr/mysql


6/49

Optimization Of MySQL On Open Solaris

Compiled with optimal compiler options usingSun Studio compilers

Added SMF support for MySQL

Easily accommodates varying configurations Initializes MySQL database thereby enhancing userexperience

Mysql user can manage MySQL database via SMF

Support for MySQL cluster engine (ndb)


7/49

Compiler Optimization

Enable in-lining

Change header file univ.i to add Sun Studio If (!defined(SUNPRO_C)

#define UNIV_MUST_NOT_INLINE

Enable pre-fetching

-xprefetch=auto and -xprefetch_level=3

Set optimization level -xO4


8/49

Compiler Optimization (Contd)

Feedback optimization (not yet implemented)

Workload is key

-xprofile -xipo

Interprocess optimization

About 10% improvement

Link with libmtmalloc

Library for threaded application About 8% improvement


9/49

Service Management Facility (SMF)

Makes Solaris Services Self-Healing Services automatically restart in dependency

order

Misbehaving and mis-configured services areeasier to debugLog files for each service

Administrators can securely delegate tasks to

non-root users


10/49

MySQL-Service Management Facility

Dynamically configured propertiesmysql/binInstallation , default : /usr/mysql/5.0/bin

mysql/dataData Directory , Default: /var/mysql/5.0/data

mysql/enable_64bitFlag to select 32bit or 64bit , Default : false

First time installationsCreates system tables (mysql_install_db.sh )


11/49

Starting MySQL on Open Solaris

Older Open Solaris Build (Build 76- Build-87)32Bit MySQL integratedCreate mysql user Enable MySQL SMF service

Latest Open Solaris Build (Build 87 onwards)

32Bit and 64bit MySQL integratedDefault mode : 32bit service

For starting 64bitSet enable_64bit == true


12/49

Optimized MySQL on Solaris 10

CoolStackSAMP stack + more

Optimized MySQL download Standalone package SAMP stack component

Available at Sun Download center:http://cooltools.sunsource.net/coolstack

Version MySQL 5.0.45

Similar optimizations as in Open Solaris
http://cooltools.sunsource.net/coolstackhttp://cooltools.sunsource.net/coolstack


13/49

MySQL-High Availability Data Service


14/49

Solaris Cluster Overview

Provides general purpose HA platform

Fifty percent ofenterprises that lack a

recovery plan go out ofbusiness withinone year of a significant

disasterGartner Group

Availability is ourcustomers'most criticalrequirementSun Cluster VOC Survey


15/49

Solaris Cluster : Hardware Components

Servers with local storageCan have up to 16 nodes

Shared storageTolerates single-node failures

Centralizes configuration files Cluster interconnect

At least two redundant networks

Public network interfacesSpreads outbound packets


16/49

Solaris Cluster Algorithms

Cluster membership monitor Ensures data integrity

Determines cluster membership

Cluster configuration repository

Global repository Ensures consistent view

Disk Fencing Fences off non-cluster nodes

Prevents Partition Quorum Uses a majority voting schema


17/49

MySQL High Availability Data Service

HA-MySQL is a failover data service

Node

1

Node

2

Node

3

Node

4DD

BB


18/49

MySQL High Availability Data Service

Supported configurationsStandalone MySQL server MySQL replication server

Single/Multiple MySQL instances in master configuration Single/Multiple MySQL instances in slave configuration

Solaris containers supportGlobal zone

Non-global failover zone

Non-global zone


19/49

HA-MySQL Service Deployment

Node1 Node2:Zone2Node1:Zone1

Node2

MySQL

Stor Host

MySQL

Stor Host


20/49

MySQL-Solaris Cluster Benefits

Enhanced end-to-end infrastructure availability Continuous MySQL Availability

Automatic failover if master node fails

Low cost solution

Software is free and open sourced Efficient Resource Utilization

Multiple applications can be consolidated

Ease of operationsSC enables clustered systems to be managed as ifthey were on a single system


21/49

Additional information

Step by step deployment guidehttp://docs.sun.com/app/docs/doc/819-3059

Failover study of HA-MySQL

http://blogs.sun.com/krishs/date/200804 Solaris Clusterhttp://www.sun.com/software/solaris/cluster/index.xmlhttp://opensolaris.org/os/community/ha-clusters/
http://docs.sun.com/app/docs/doc/819-3059http://blogs.sun.com/krishs/date/200804http://www.sun.com/software/solaris/cluster/index.xmlhttp://www.sun.com/software/solaris/cluster/index.xmlhttp://blogs.sun.com/krishs/date/200804http://docs.sun.com/app/docs/doc/819-3059


22/49

Consolidate MySQL installations using SolarisContainers


23/49

Solaris Containers

Containers : Zones + Resource Management Zones: isolated virtual application environments

Resource management resource control (CPU,

Memory)

Achieving Consolidation GoalsReduce Hardware

Combine low utilization systemsIsolate applications from faultsMaintain Service Levels

Fine tune response times


24/49

MySQL Consolidation Study

System ConfigurationSun Fire X4100 (4 CPU, 8 GB Memory)

Local container configuration ( 1 CPU , 1 GB

Memory

Sysbench Read-only Sysbench read-write

32 64 128 256

1300

1400

1500

1600

1700

1800

1900

Sysbench read-only

No ContainersWith Containers

Number of threads

Throug

hput

32 64 128 256

425

450

475

500

525550

575

600

625

650

675Sysbench read-write

No ContainersWith Containers

Number of threads

Throug

hput


25/49

Dtrace (Jenny Chen)


26/49

DTrace

Use DTrace with MySQL to drill downMonitoring MySQL Performance

Examples & Solutions

Easy Steps To add DTrace probes intoMySQL core server and storage Engines

Easy Steps to display MySQL DTraceprobes into Chime visualization Tool forDTrace


27/49

Why DTrace

Solaris 10 Dynamic Tracing Facility to providecomprehensive view of operating system andapplication behaviour

> DTrace to examine particular system areas: disk I/O, CPU,

Memory> Process Tracing and Debugging

USDT(User-level statically defined tracing) place custom probes inapplication code

Add USDT into MySQL source to monitor MySQL and gatherthe useful data missing by the current MySQLmonitor tools:


28/49

DTrace: Moni to r I/Os

Exam I/O wait time by filename and mysqld(Available at DTraceToolkit)#!/usr/sbin/dtrace s

#pragma D option quiet

io:::wait-start

/ execname == mysqld /

{ self->start = timestamp; }

io:::wait-done

/ execname == mysqld && self->start /

{ this->elapsed = timestamp - self->start;

@files[pid,args[1]->dev_pathname, args[2]->fi_pathname] = sum(this->elapsed);

self->start = 0;

}

profile:::tick-5s

{ printf(-------------------------------------------------\n);

printf(%6s %8s %20s %50s\n, PID, TIME, DEVICE, FILE);

printa(%6d %@8d %20s %8s \n, @files);

printf (------------------------------------------------\n); }

#./mysqliowait.d

---------------------------------------------------

PID TIME DEVICE FILE

113 234 /dev/dsk/c0t2d0s0 /usr/local/mysql/data/ibdata1


29/49

DTrace: Monitor CPU classic performance problem

#!/usr/sbin/dtrace -s

#pragma D option quiet

syscall:::entry

{

self->ts=vtimestamp;

}

syscall:::return

/self->ts/{

@a[execname, probefunc] = count();

@b[execname, probefunc] = sum (vtimestamp - self->ts);

self->ts=0;

}

END

{

printf("%-16s %-16s %-8s\n","EXEC","SYSCALL","COUNT");

printa("%-16s %-16s %-@8d\n",@a);

printf("%-16s %-16s %-8s\n","EXEC","SYSCALL","TIME");

printa("%-16s %-16s %-@8d\n",@b);

}


30/49

# ./syscall.d

EXEC SYSCALL COUNT

devfsadm lwp_park 1

dtrace fstat 1

...

mysqld read 106542

mysqld gtime 109613

mysqld pread 1181669

---------------------------------------------------

EXEC SYSCALL TIME

dtrace lwp_sigmask 218pkill getpid 302

...

mysqld read 259284183

mysqld write 267556239

mysqld pread 4650457224


31/49

Agrregated user stack backtrace to understand of the nature of pread() inMySQL source code useful for mysql developers

# dtrace -n 'syscall::pread:entry / execname == "mysqld" / { @[ustack()]=count() }

dtrace: description 'syscall::pread:entry ' matched 1 probe...

libc.so.1`_pread+0xa

mysqld`my_pread+0x54

mysqld`_mi_read_static_record+0x67

mysqld`mi_rnext+0x1fe

...

mysqld`handle_one_connection+0x855

libc.so.1`_thr_setup+0x67

1564811

....

Replace high cost pread with

--myisam_use_mmap=1- useful for mysql DBA

A 94% performanceimprovement !


32/49

Dtrace: Memory Analysis Check which process causes anonymous page in

# dtrace -n anonpgin '{@[execname] = count()}'

dtrace: description anonpgin matched 1 probe

sshd 2

vmstat 23

mysqld 673

Use Dtrace to measure waiting for paging in

# ./ whospaging.d available at Solaris Internals(http://www.solarisinternals.com/si/dtrace/)Who's on cpu (milliseconds):

sshd 1

vmstat 3

mysqld 120

sched 43210

Who's waiting for pagin (milliseconds):

mysqld 239082
http://www.solarisinternals.com/si/dtrace/http://www.solarisinternals.com/si/dtrace/


33/49

Dtrace Probes In MySQL

Provide deep view of internal MySQL core server and storage engines'operation & behaviour

>Database information

>Query execution latency

>Index & table scan cost

>Wait events inside MyISAM & Innodb

>Deadlock information

>Query cache hit/miss

>And many more...

Speed resolution of performance bottlenecks with in database designand MySQL server configurations

Negligible performance overhead

Easy steps to create & insert your own Dtrace probes into MySQL

MySQL DTrace GUI Monitor Tool - Chime


34/49

Query execution Time Enable Slow query log with log-slow-queries requires re-

start MySQL server

SQL statements with query execution time longer thanlong_query_time second in the log file

SQL statements generating most loads on the application maynot in slow query log

Replication query statements are not available in slow query log

Time spending by the query optimizer to generate query plan isnot available in slow query log

Using DTrace can get mising query execution information online


35/49

Insert DTrace Probes In to MySQL Step 1: Figure out what probes are needed to insert into thesource code

Step 2: Define MySQL Provider and probes

# cat mysql_dtrace.d

provider mysql

{

probe query__execute__start(void *, char *, char *, const char *, char *);

probe query__execute__finish(void *, char *, char *, const char *, char *,int);

}

>Two Probes defined in the mysql provider>Note to use two underscore(__) translated to hypen automatically


36/49

Step 3: Define a header file mysql_dtrace.h with definitions forprobes

dtrace -h -s mysql_dtrace.d

#ifndef _MYSQL_H

#define _MYSQL_H

#define DTRACE_QUERY_EXECUTE_START(arg0, arg1, arg2, arg3, arg4) \

__dtrace_mysql___query__execute__start(arg0, arg1, arg2, arg3, arg4)

#define DTRACE_QUERY_EXECUTE_START_ENABLED() \

__dtraceenabled_mysql___query__execute__start()

extern void __dtrace_mysql___query__execute__start(void *, char *, char *, char*, char *);

extern int __dtraceenabled_mysql___query__execute__start(void);

#endif


37/49

Step 4: Insert the probes into source code #include

...

bool mysql_execute_command(THD *thd)

{

DTRACE_QUERY_EXECUTE_START((void *)thd, thd->db,

thd->security_ctx->user, (char *)thd->security_ctx->host_or_ip,thd->query);

...

DTRACE_QUERY_EXECUTE_FINISH((void*)thd, thd->db,

thd->security_ctx->user, (char *)thd->security_ctx->host_or_ip,

thd->query, res ==0 ? 0: -1);

go to end;

...

}


38/49

Step 5: Build MySQL with DTrace

In the Makefile.in, compile 64-bit MySQL with Dtrace

mysqld_OBJECTS = $(am_mysqld_OBJECTS) mysql_dtrace.$(OBJEXT)

mysql_dtrace.o:$(top_srcdir)/include/mysql_dtrace.d $(am_mysqld_OBJECTS)

dtrace -G 64 -s $(top_srcdir)/include/mysql_dtrace.d $(am_mysqld_OBJECTS)

Inserting DTrace probes comleted, DTrace probes are ready to use!


39/49

Step 6: Use inserted DTrace probes to measure query executiontime with other database information

#!/usr/sbin/dtrace -s

mysql*:::query-execute-start

{ self->start = timestamp; }

mysql*:::query-execute-finish

/self->start/

{ this->query = copyinstr(arg4); }

mysql*:::query-execute-finish

/self->start/

{ this->elapsed = (timestamp - self->start) / 1000000;

this->who = strjoin(copyinstr(arg2), strjoin("@", copyinstr(arg3)));

printf(" %-16.16s %-18.18s %5d %3d %-32.32s\n", arg1 ? copyinstr(arg1)

: ".", this->who, this->elapsed, (int)arg5, this->query); self->start = 0;

}


40/49

# ./mysqld_qestat.d

DATABASE USER@HOST ms RET QUERY

sbtest root@localhost 0 0 show tablessbtest root@localhost 0 0 show databases

sbtest root@localhost 178 0 select * from sbtest

Use the same steps to insert Dtrace probes to/sql/sql_select.cc at the start and end ofchoose_plan()function to measure the time spent in query optimization

optimizer_prune_level=1 reduce query compilation time

Reduce optimizer_search_depth or optimizer_search_depth=0


41/49

Index & Table scan cost

Identify the places in MySQL source to handle scanning index, andtable

>Index-scan functions: index_next, index_next_same, index_prev,index_first, index_last

>Table-scan functions: rnd_init, rnd_end, rnd_next, rnd_pos

>Insert DTrace Probes at the start and before return from the functions

can measure the time spending on scanning table or index.mysql*:::innodb-index-next-start

{ @indexnext[args[0]] = count();

self->inext = timestamp; }

mysql*:::innodb-index-next-finish

/self->inext/

{ @indexnexttime[args[0]] = sum(timestamp - self->inext);

self->inext = 0; }

Expensive index-scan/table-scan report from Dtrace requires tooptimize schema accordingly


42/49

Buf fer wait in Innodb

Innodb buffer wait is common in I/O-bound MySQL system while readingpage synchronous from disk

Insert Dtrace probes at: innobase/buf/buf0rea.c,

ulint buf_read_page( ulint space, ulint offset)

{ ...

DTRACE_INNODB_BUFFER_WAIT_START();

/* We do the i/o in the synchronous aio mode to save threadswitches: hence TRUE */

count2 = buf_read_page_low(&err, TRUE, BUF_READ_ANY_PAGE, space,

tablespace_version, offset);

DTRACE_INNODB_BUFFER_WAIT_FINISH();

...

}

Increase innodb_buffer_pool size Tune SQL to access rows with fewer block reads(i.e. By adding indexes)


43/49

Query cache probes

Use DTrace to measure query cache hit and query cache miss todetermine how well the query cache is performing

> Insert DTrace Probes at: sql/sql_cache.cc function: send_result_to_client

> DTrace Test script to report query cache hit and miss counts by query string, andtotals

mysql*:::query-cache-hit,

mysql*:::query-cache-miss

{ this->query = copyinstr(arg4); }

mysql*:::query-cache-hit

{ @elapsed[this->query, "hit"] = count();

hits++; }

mysql*:::query-cache-miss

{ @elapsed[this->query, "miss"] = count();

misses++; }


44/49

# ./mysqld_qchit.d

Tracing... Hit Ctrl-C to end.

^C

QUERY RESULT COUNT

select * from months miss 1

select * from months where num > 3 hit 1

select * from months where num > 3 and num < 9 miss 1

show databases miss 1

show tables miss 1

select * from months hit 9

Hits : 10

Misses : 4

Hit Rate : 71%

Tuning query_cache_size variable according to the hit/miss rate


45/49

DTrace Perf or mance Impact Inserting DTrace Probes into MySQL source code are useful forMySQL DBA, MySQL & application developers. The

performance impact of adding DTrace probes is critical forenterprise environment.

Cost of inserting USDT probe can be basically negligible:

Each probe inserted into the source code can be enabled by adding the

code like:

if (PROVIDER_PROBE_ENABLED()

{

PROVIDER_PROBE(arg0,...);

}


46/49

DTrace Probes In MySQL 6.0

probe insert_row_start();probe insert_row_end();probe filesort_start();probe filesort_end();probe delete_start();probe delete_end();probe select_start()probe select_end();probe update_start();

probe update_end(); compile with --enable-dtrace configure option to usethe Dtrace probes


47/49


48/49


49/49

Resources

> http://www.opensolaris.org/os/community/dtrace

- OpenSolaris Community: Dtrace

>http://developers.sun.com/solaris/articles/solaris_perftools.html- Solaris Performance and Tools

> http://docs.sun.com/app/docs/doc/817-6223/6mlkidlms?a=view

- Statically Defined Tracing for User Applications chapter of DTrace

Manual

>http://www.brendangregg.com

- DTrace toolkits >http://www.solarisinternals.com/wiki/index.php/DTrace_Topics_Databases

- DTrace Topics Databases

Acknowledgements

Brandan Gregg Sun Microsystems Engineer in Advanced ProductsGroup
http://www.opensolaris.org/http://docs.sun.com/app/docs/doc/817-6223/6mlkidlms?a=viewhttp://docs.sun.com/app/docs/doc/817-6223/6mlkidlms?a=viewhttp://www.opensolaris.org/

best practices for deploying mysql on the solaris platform presentation 1

Documents