This is the 1st article in our new series on Linux performance monitoring and tuning.
Linux system administrators should be proficient in Linux performance monitoring and tuning. This article gives a high level overview on how we should approach performance monitoring and tuning in Linux, and the various subsystems (and performance metrics) that needs to be monitored.
To identify system bottlenecks and come up with solutions to fix it, you should understand how various components of Linux works. For example, how the kernel gives preference to one Linux process over others using nice values, how I/O interrupts are handled, how the memory management works, how the Linux file system works, how the network layer is implemented in Linux, etc.,
Please note that understanding how various components (or subsystems) works is not the same as knowing what command to execute to get certain output. For example, you might know that “uptime” or “top” command gives the “load average”. But, if you don’t know what it means, and how the CPU (or process) subsystem works, you might not be able to understand it properly. Understanding the subsystems is an on-going task, which you’ll be constantly learning all the time.
On a very high level, following are the four subsystems that needs to be monitored.
- CPU
- Memory
- I/O
- Network
1. CPU
You should understand the four critical performance metrics for CPU — context switch, run queue, cpu utilization, and load average.
Context Switch
- When CPU switches from one process (or thread) to another, it is called as context switch.
- When a process switch happens, kernel stores the current state of the CPU (of a process or thread) in the memory.
- Kernel also retrieves the previously stored state (of a process or thread) from the memory and puts it in the CPU.
- Context switching is very essential for multitasking of the CPU.
- However, a higher level of context switching can cause performance issues.
Run Queue
- Run queue indicates the total number of active processes in the current queue for CPU.
- When CPU is ready to execute a process, it picks it up from the run queue based on the priority of the process.
- Please note that processes that are in sleep state, or i/o wait state are not in the run queue.
- So, a higher number of processes in the run queue can cause performance issues.
Cpu Utilization
- This indicates how much of the CPU is currently getting used.
- This is fairly straight forward, and you can view the CPU utilization from the top command.
- 100% CPU utilization means the system is fully loaded.
- So, a higher %age of CPU utilization will cause performance issues.
Load Average
- This indicates the average CPU load over a specific time period.
- On Linux, load average is displayed for the last 1 minute, 5 minutes, and 15 minutes. This is helpful to see whether the overall load on the system is going up or down.
- For example, a load average of “0.75 1.70 2.10” indicates that the load on the system is coming down. 0.75 is the load average in the last 1 minute. 1.70 is the load average in the last 5 minutes. 2.10 is the load average in the last 15 minutes.
- Please note that this load average is calculated by combining both the total number of process in the queue, and the total number of processes in the uninterruptable task status.
2. Network
- A good understanding of TCP/IP concepts is helpful while analyzing any network issues. We’ll discuss more about this in future articles.
- For network interfaces, you should monitor total number of packets (and bytes) received/sent through the interface, number of packets dropped, etc.,
3. I/O
- I/O wait is the amount of time CPU is waiting for I/O. If you see consistent high i/o wait on you system, it indicates a problem in the disk subsystem.
- You should also monitor reads/second, and writes/second. This is measured in blocks. i.e number of blocks read/write per second. These are also referred as bi and bo (block in and block out).
- tps indicates total transactions per seconds, which is sum of rtps (read transactions per second) and wtps (write transactions per seconds).
4. Memory
- As you know, RAM is your physical memory. If you have 4GB RAM installed on your system, you have 4GB of physical memory.
- Virtual memory = Swap space available on the disk + Physical memory. The virtual memory contains both user space and kernel space.
- Using either 32-bit or 64-bit system makes a big difference in determining how much memory a process can utilize.
- On a 32-bit system a process can only access a maximum of 4GB virtual memory. On a 64-bit system there is no such limitation.
- The unused RAM will be used as file system cache by the kernel.
- The Linux system will swap when it needs more memory. i.e when it needs more memory than the physical memory. When it swaps, it writes the least used memory pages from the physical memory to the swap space on the disk.
- Lot of swapping can cause performance issues, as the disk is much slower than the physical memory, and it takes time to swap the memory pages from RAM to disk.
All of the above 4 subsystems are interrelated. Just because you see a high reads/second, or writes/second, or I/O wait doesn’t mean the issue is there with the I/O sub-system. It also depends on what the application is doing. In most cases, the performance issue might be caused by the application that is running on the Linux system.
Remember the 80/20 rule — 80% of the performance improvement comes from tuning the application, and the rest 20% comes from tuning the infrastructure components.
There are various tools available to monitor Linux system performance. For example: top, free, ps, iostat, vmstat, mpstat, sar, tcpump, netstat, iozone, etc., We’ll be discussing more about these tools and how to use them in the upcoming articles in this series.
Following is the 4 step approach to identify and solve a performance issue.
- Step 1 – Understand (and reproduce) the problem: Half of the problem is solved when you clearly understand what the problem is. Before trying to solve the performance issue, first work on clearly defining the problem. The more time you spend on understanding and defining the problem will give you enough details to look for the answers in the right place. If possible, try to reproduce the problem, or at least simulate a situation that you think closely resembles the problem. This will later help you to validate the solution you come up to fix the performance issue.
- Step 2 – Monitor and collect data: After defining the problem clearly, monitor the system and try to collect as much data as possible on various subsystems. Based on this data, come up list of potential issues.
- Step 3 – Eliminate and narrow down issues: After having a list of potential issues, dive into each one of them and eliminate any non issues. Narrow it down further to see whether it is an application issue, or an infrastructure issue. Drill down further and narrow it down to a specific component. For example, if it is an infrastructure issue, narrow it down and identify the subsystem that is causing the issue. If it is an I/O subsystem issue, narrow it down to a specific partition, or raid group, or LUN, or disk. Basically, keep drilling down until you put your finger on the root cause of the issue.
- Step 4 – One change at a time: Once you’ve narrowed down to a small list of potential issues, don’t try to make multiple changes at one time. If you make multiple changes, you wouldn’t know which one fixed the original issue. Multiple changes at one time might also cause new issues, which you’ll be chasing after instead of fixing the original issue. So, make one change at a time, and see if it fixes the original problem.
In the upcoming articles of the performance series, we’ll discuss more about how to monitor and address performance issues on CPU, Memory, I/O and Network subsystem using various Linux performance monitoring tools.
My name is Ramesh Natarajan. I will be posting instruction guides, how-to, troubleshooting tips and tricks on Linux, database, hardware, security and web. My focus is to write articles that will either teach you or help you resolve a problem. Read more about
Comments on this entry are closed.
Great piece of article..
Thanks a lot.
Thanks good information on monitoring
Nice and concise article….. I am looking forward for the information on the commands and tools. Way to Go Ramesh…… By the way congrats on the birth of your child…… She going be the Kernel of your life now!!!!!!! 🙂
Such a nice and Interesting topic … Pls keep posting …tks…
Thanks for the great article, Ramesh. New subscriber here. 🙂
Hi Ramesh.
I’ve been a subscriber for a fair while now & would like to compliment you
on the quality of your tutorials. They’re refreshingly ‘to the point’ !!
Very good article. Waiting for rest of the articles on this topic.
What about PAE kernels ? Aren’t they able to address beyond 4G ?
Thanks for the series, I’ve learned quite a bit following your blog.
Very good article!
Waiting for the next parts!
great piece of information…excellent
Good artical
Excellent “get you started” article – its easy to forget those starting out on the performance tuning journey after you have been doing this for a few years. I hope you are going to include “nmon” in the list of useful commands in the next article in the series as a one stop shop.
The load average also includes network activity; not cpu processes alone. It is possible to have a high load average without dramatic cpu usage.
… The load average also includes disk usage…
Good article.very useful.Thank u Ramesh..
Really very good information and it helps us to increase our knowledge in more details.
Looking forward for next articles.
Great serie! I’m anxious to the next articles.
Regards,
Júlio.
Detailed information. No need to refer any other materials on the internet.
Eagerly waiting for more information on this topic.
The first thing that jumped out at me is your comment about a high i/o wait being a bad thing. This is totally false and a huge source for confusion. All this means is that the CPU was busy while I/O was in progress. If you look at any busy nfs server you’ll see a high I/O wait percentage. If you see a process doing a large burst of I/O you’ll see a high burst of I/O wait. If a compute bound job happens to be running at the same time you’ll see a low I/O wait because the CPU is busy doing other things at the same time.
btw – when you talk about performance monitoring, be sure to pay attention to the open source tool collectl, which essentially replaces most other tools, collecting all the data in one place and providing an easy-to-read simple display. Text based of course, but if you really want plots, its companion utility colplot can do that too.
-mark
Very good article.
However in first 2 paragraphs (and some other places) CPU and kernel have been used interchangeably , which could be confusing to people new to operating system. Please update it, if you like.
Mark, l think Ramesh’s comment about I/O is not confusing at all, but your’s is.
In general, CPU might be waiting for I/O to finish, which is a typical case for busy waiting, how ever if CPU is doing DMA or any other interrupt based data transfer then CPU is free to perform any other task, and should not be considered as busy. However, some I/O driver (e.g. some disk driver) does do busy wait while performing I/O, because there wait period is much smaller (if we compare this with interrupt service and rescheduling the ongoing task).
Eagerly waiting for next series of yours.
Lookin forward for the next article! Great job..!
/Cy
Thanks for the explanation, I have followed your blog for some time and I have become a recurring reader, thanks and hope you continue so for long.
Thanks..it is nice to know these concepts.thank you for your valuable informations..
when will the nest post come??
Great article… Thanks
As part of my job, I am required to analyze performance of Unix, linux, and Windows server infrastructure. I could not agree more with this entire article, especially the 80/20 rule. It is one thing to see that the system has memory pressure, or CPU pressure. It is another thing entirely to figure out how to fix it without having to add physical memory or processors.
Great Article..at really works..! Thanks..
Thank you so much for writing and posting this article, Ramesh. A great overview for newbies in performance tuning.
Hi Ramesh,
Good one for Beginners:) keep up the good work
Awesome article! Thanks a lot:)
Hi Ramesh,
Really Great article!
Thanks a lot for your good work.
Thanks a lot
linux [cpu+memory+i/o+n/w] performance tuning & monitoring nice article
Very useful information in simple words …
Thank you for posting such a nice article
Nice article… on performance. Given good stuff..
Very useful information.
and i want to know the cpu and memory utilization of each process and services running separately. can any body help in answering my question.