tuning tips for linux & aix to run a database - ukuug linux and unix support and tuning, ......
TRANSCRIPT
v
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG
Tuning Tips for Linux & AIX to run a Database
Nigel GriffithsLinux on Power & VirtualisationTechnical Leader, IBM Europe
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG2
Abstract:►Sitting behind any reasonable size website is some sort of
database to hold the data involved. ►This can be used to control the web output, user records or
profiles or regular database contents which are made available or manipulated via web applications.
► It is key then for the database to perform well in its support of the user experience.
►This session looks at the common causes of database performance problems with the aim to make sure you don't fall into these common traps.
►This is practical advice from the presenter's wide experience of Linux and UNIX support and tuning, and not database vendor specific.“
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG3
Agenda1.Scene setting and Assumptions 2.Configuration and Change control3.Use the latest OS version4.Burn in the box5.Spotting single-threadedness6.Balanced Systems7.Waiting is not a problem8.Disks * four …9.Memory – Use it or lose it!10.The problem with paging11.Understand your virtualisation tools & effects12.Classic bad thinking13.Know Your tools and nmon (blatant advert)
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG4
Assumptions
Database is on a server (not a PC) Multiple CPUs, Many GB of RAM, Many 100’s GB of disks and Many Processes / Users
You have many machines to look after Datacentre
I am biased toward what I know and have used► I work for IBM (15 years plus 10 before that)► Big UNIX systems: RS6000 / pSeries / System p / … ► AIX & Linux on POWER then Linux on x86 – Benchmarking + Sizing► RDBMS Oracle, then some DB2, MySQL, Sybase, Informix► I will try to keep the IBM content to a minimum – but it is hard for me to do
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG5
Scene Setting
Movie
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG6
Scene Setting
From my personal experience (your mileage may vary) Performance problems reported to IBM
► 50% poor disk layout + mgmt some disks 90%+ busy, while 50% not used at all► 10% poor setup of RDBMS tuning parameters relating to memory use► 10% single threaded batch applications► 10% poorly written customer extensions to standard applications► 5% system running with errors in the [errpt] log file (including CPU failures!!)► 5% paging on large RAM (>2 GB) systems [vmtune not use to set min/maxperm]► 5% OS problems already discovered and fixed but OS was not up to date► 4% badly ported App = not compiled with optimization or on old OS version► 1% genuine bugs in OS, RDBMS or commands
every single one of these was reported as a problem with the hardware!!!
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG7
What is “Quality”?
CPU► good 70%► bad 85%► ugly 90%+ (except batch)
Disk► good <30%► bad <40%► ugly <50%+
Memory► good zero paging► bad 10 pages per second per CPU► ugly any more paging+
Network ► good <30% bandwidth► Hubs bad >70%► Switched bad is MB/s near Mb /10
Run Queue► good < 2 * CPU's
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG8
Configuration and Change Control
Script to document the whole config1. CPU – number, type and speed2. Memory – size3. Adapters – number & type4. Network - number & type5. Disks – number & type6. Disk layout & file systems layout7. OS version & tuning options8. RDBMS versions & tuning options9. App (s) version & tuning options
Off machine – you need this anyway for disaster recover – right!
Do you know when the machine. OS, App, RDBMS was last changed?► By whom and why?► Exactly what they did?
Save config & performance data Before Make the change Save config & performance data After Makes problem determination
10 times easier and faster
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG9
Use the latest OS version Linux ► For advanced kernel features use 2.6 kernel and the latest rev► 100’s of man hour gone into the OS performance tuning ● why not take it for free?
► In 2.6.16 alone:● gcc, glibc, libhugetlbfs, 64KB pages = scaling (SLES10, RHEL5)
► No excuse for the 2.4 kernel now
AIX6► Major improvements and Power6 support at 4.7GHz CPU► Mobility to move a CPU partition between machines while live► Workload Partitions, JFS2 encryption & snapshots, RBAC
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG10
Burn in the box Burn in – first used in mainframe
► Larger the machine the more parts and larger every year
Hardware failures happen► Most failures are in the first two weeks► Then fairly random► Some occasions after 1 2 years in small batches
Therefore smart to burn in test for 24 hours Before you add your data► CPUs – just run something on each of them► Memory – get it hot► Disks – get them all spinning► Adapters – get them all busy
After this Disk protection & HA will then catch the remaining failures
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG11
Single Threaded Applications
Still finding applications like this!
The IT Industry has been SMP for over a decade ► There are no excuses► Its just lazy programming and no longer acceptable
If you have no choice …get the fastest CPUs in the industry … POWER6for you single process
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG12
Balanced Systems
Meaning a sensible ratio of CPU Memory Disks Adapters
For example my rule of thumb for POWER6► 1 CPU at 4.7GHz► 8 GB Memory► 1 TB Disk on 10 disks► 1 FC SAN adapter► 1 GB Ethernet adapter
Decide your own ratios for your platform If you are too far away from balanced you are
“stuffed”!
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG13
Waiting is not a problem CPU Wait – worries many people CPU Wait = CPU in Idle state
► But a flag is set indicating outstanding Disk I/O
20 years ago indicated ► Disks are holding back the machine
Now indicates► CPUs process the data faster than the
disks can supply it► Many applications just work this way
● pulls in lots of disk blocks but does not do much processing on the data
► Like BI, SAP RDBMS, SQL table scans
A benchmarker sees this as a opportunity to start more work !!
Is this machine 80 or 60% Busy?
Answer: 60% Busy with 40% unused CPU
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG14
Disks – the biggest problem 50% of performance problems are Disk Related
Going to assume you are running disk protection of some sort► Software Mirror► Hardware Mirror► Hardware RAID5► Fibre-Channel SAN sub-system based protection► Even iSCSI or NFS
If not you deserve every problem you get!
Also don’t forget to protect OS and paging space► If you lose these … you panic!
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG15
Disks – uneven use
Very often find this … Some times this … But never this …
Lots of times: Half the disks are doing nothing!!
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG16
Disks - Layout Old School►Separate Data, Indexes & Logs disks (+separate OS, App, paging)►This worked well when we had few concurrent SQL statements►Problem is we now have 1000’s of “in flight” transactions►Generates what looks like random disk I/O● So zero disk optimisation is possible
►We end up with some hot & some cold disks
New School►Make every disk participate – the “hose it all around” approach►Zero hot spots using your hardware to do the work (not people)► If extreme logging and 30 disks+ ● May have Dedicated Log disks 4 is the minimum
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG17
Disks – Too many disks also an issue
How many LUN’s can you manage?► 10► 100► 500► 1000► 2000► 5000
IMHO Too many LUN’s is a big mistake, that we will have to correct in the next 10 years.
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG18
Disks - Missing adapter stats No UNIX that I know does adapter stats They add the underlying disks
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG19
Memory – Use it or lose it!
If you buy 8 GB of memory use 8GB of memory!► Free memory (above a few 10MB) is a waste of money
Free memory is a tuning opportunity AIX will use free RAM in the filesystem cache
► Grow the RDBMS disk cache for faster SQL with less Disk I/O► Or larger in memory sorts
Tests (a few years ago) showed in memory data runs 10 times faster
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG20
The problem with Paging UNIX folklore Paging – just don’t do it. With high numbers of user systems, they page all the time►Asked my mainframe colleagues …
We have to live with it► Just make sure you spread out paging ►On to lots and lots of disks - what the back half of large disks is for !!
Paging many types
/bin/ksh
Memory Paging spaceFilesystem
Process
Code (ro)
Data, heap, stack
MemoryMapped file Shared
memory
Libraries
File
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG21
Understand your virtualisation tools and effects Virtualisation► Like: Xen, VMware, Advanced POWER Virtualisation (APV)
If you are in these environments you have to understand them well Can get very misleading in performance numbers Slow down might be due to other workloads not your tuning Do your homework► Read the manuals, articles, books and websites► Take the education
Run experiments► Generate workload► Can your tools clear tell you what is going on?
In APV, we now monitor Physical CPU use (not Utilisation)
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG22
Classic Bad Thinking 1 Filesystems are as fast as raw disks RAID5 is fast (the clue is in the I = inexpensive) RAID5 write will not be too bad DB does not do writes Using new and untried features Misreading I/O wait CPU statistics (covered before) Simplistic test that prove nothing on larger machines Assuming the defaults will be fine JFS raw + not adjusting RDBMS buffer pool size Using the tool you know, rather than the right tool Random OS parameters changes and forgetting them
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG23
Classic Bad Thinking 2 Running more than one database on a machine► expecting the OS to work out which is more important
Working out transaction rate from a yearly total and assuming 365 * 8 even hours of flat workload► Especially not true in the web business► Also basing a transaction rate on marketing estimates
Incremental upgrades & changes allow the quickest diagnosis► Do not upgrade OS, database and the application all together► Do not change software and H/W (like disk layout)► Do not change software or DB parameters and data placement
Parallel databases are easy (and cheap)► More complex need balancing more training, experience & management
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG24
Know your tools Regular UNIX/Linux Tuning tools
vmstat iostat, top (Linux) or topas (AIX)
Do your home work read those manuals► I regularly have to tell people what the columns really mean!
Not so good for larger machines = 100’s of disks or processes etc. Do get boring after an hour We need something better and graphs (not numbers)
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG25
nmon basics Freely available Performance monitor for
► AIX 5 & 6► AIX 4 (via older nmon version) ► Linux (POWER, x86 and mainframe)
Near zero installation Two modes
► Online see ► Save data to file then either:● nmon2rrd & rrdtool for .gifs/website● nmon Analyser Excel spreadsheet
– From Stephen Atkins– IBM, UK.– Requires Excel 2000– Very good chap.
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG26
Know your Profile (processes, CPU, RAM & Disk) Nmon Analyser default graphs Then you can understand when its wrong
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG27
nmon
Screennmonfile
0
50
100
150
200
Excel graphs
Stephen’s nmonAnalysernmon2rrd
C filter
Java Dynamic Graphs
rrdtoolopen source
Scripts - Create rrd - Load rrd - Graph rrd index.html
-f or –F optionnmon
Federico’spGraph
Website +.jpg graphs
Bruce’snmon2web
perl
rrdtoolopen source
Scriptsrrdtool scripts+ CPU & RAM aggregation
Stephen’snmon
Consolidator
0
50
100
150
200
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG28
nmon References and Download
Forum Performance Tools Forum
Mostly about nmon For questions & answers
Wiki AIX 5L Wiki “Other Performance Tools” nmon
Download FAQ Manual and lots more
http://www.ibm.com/systems/p/community/
Tuning Tips - Databases
UK UNIX User Group – Databases and the Web
© 2007 IBM Corporation & UKUUG29
Whitepaper: Oracle 10g Release 2 Performance and Tuning on an IBM System p™ Linux on POWER Platform► Chij-Mehn Chang, Ric Hendrickson, Michel Riviere, Arun Sar► June 30, 2007
IBM Redbook: Tuning Linux OS on System p► SG24-7338
Questions
Other References