This article provides a total of 24 examples on iostat, vmstat, and mpstat commands.
- iostat reports CPU, disk I/O, and NFS statistics.
- vmstat reports virtual memory statistics.
- mpstat reports processors statictics.
This article is part of our ongoing Linux performance monitoring series.
Please note that iostat and vmstat are part of the sar utility. You should install sysstat package as explained in our sar (sysstat) article to get iostat and vmstat working.
IOSTAT EXAMPLES
1. iostat – Basic example
Iostat without any argument displays information about the CPU usage, and I/O statistics about all the partitions on the system as shown below.
$ iostat
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011
avg-cpu: %user %nice %system %iowait %steal %idle
5.68 0.00 0.52 2.03 0.00 91.76
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 194.72 1096.66 1598.70 2719068704 3963827344
sda1 178.20 773.45 1329.09 1917686794 3295354888
sda2 16.51 323.19 269.61 801326686 668472456
sdb 371.31 945.97 1073.33 2345452365 2661206408
sdb1 371.31 945.95 1073.33 2345396901 2661206408
sdc 408.03 207.05 972.42 513364213 2411023092
sdc1 408.03 207.03 972.42 513308749 2411023092
2. iostat – Display only cpu statistics
iostat option -c, displays only the CPU usage statistics as shown below.
$ iostat -c
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011
avg-cpu: %user %nice %system %iowait %steal %idle
5.68 0.00 0.52 2.03 0.00 91.76
3. iostat – Display only disk I/O statistics
iostat option -d, displays only the disk I/O statistics as shown below.
$ iostat -d Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 194.71 1096.61 1598.63 2719068720 3963827704 sda1 178.20 773.41 1329.03 1917686810 3295355248 sda2 16.51 323.18 269.60 801326686 668472456 sdb 371.29 945.93 1073.28 2345452365 2661209192 sdb1 371.29 945.91 1073.28 2345396901 2661209192 sdc 408.01 207.04 972.38 513364213 2411024484 sdc1 408.01 207.02 972.38 513308749 2411024484
4. iostat – Display only network statistics
iostat option -n, displays only the device and NFS statistics as shown below.
$ iostat -n
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011
avg-cpu: %user %nice %sys %iowait %idle
4.33 0.01 1.16 0.31 94.19
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 2.83 0.35 5.39 29817402 457360056
sda1 3.32 50.18 4.57 4259963994 387641400
sda2 0.20 0.76 0.82 64685128 69718576
sdb 6.59 15.53 42.98 1318931178 3649084113
sdb1 11.80 15.53 42.98 1318713382 3649012985
Device: rBlk_nor/s wBlk_nor/s rBlk_dir/s wBlk_dir/s rBlk_svr/s wBlk_svr/s
192.168.1.4:/home/data 90.67 0.00 0.00 0.00 5.33 0.00
192.168.1.4:/backup 8.74 0.00 0.00 0.00 8.74 0.00
192.168.1.8:/media 0.02 0.00 0.00 0.00 0.01 0.00
5. iostat – Display I/O data in MB/second
By default iostat, displays the device I/O statistics in Blocks. To change it to MB, use -m as shown below.
$ iostat -m
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011
avg-cpu: %user %nice %system %iowait %steal %idle
5.68 0.00 0.52 2.03 0.00 91.76
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 194.70 0.54 0.78 1327670 1935463
sda1 178.19 0.38 0.65 936370 1609060
sda2 16.51 0.16 0.13 391272 326402
sdb 371.27 0.46 0.52 1145240 1299425
sdb1 371.27 0.46 0.52 1145213 1299425
sdc 407.99 0.10 0.47 250666 1177259
sdc1 407.99 0.10 0.47 250639 1177259
6. iostat – Display I/O statistics only for a device
By default iostat displays I/O data for all the disks available in the system. To view statistics for a specific device (For example, /dev/sda), use the option -p as shown below.
$ iostat -p sda
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011
avg-cpu: %user %nice %system %iowait %steal %idle
5.68 0.00 0.52 2.03 0.00 91.76
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 194.69 1096.51 1598.48 2719069928 3963829584
sda2 336.38 27.17 54.00 67365064 133905080
sda1 821.89 0.69 243.53 1720833 603892838
7. iostat – Display timestamp information
By default iostat displays only the current date. To display the current time, use the option -t as shown below.
$ iostat -t
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011
Time: 08:57:52 AM
avg-cpu: %user %nice %system %iowait %steal %idle
5.68 0.00 0.52 2.03 0.00 91.76
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 194.69 1096.49 1598.45 2719070384 3963829704
sda1 178.18 773.32 1328.88 1917688474 3295357248
sda2 16.51 323.14 269.57 801326686 668472456
sdb 371.25 945.82 1073.16 2345452741 2661228872
sdb1 371.25 945.80 1073.16 2345397277 2661228872
sdc 407.97 207.02 972.27 513364233 2411030200
sdc1 407.97 207.00 972.27 513308769 2411030200
8. iostat – Display Extended status
Use option -x, which will displays extended disk I/O statistics information as shown below.
$ iostat -x
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011
avg-cpu: %user %nice %system %iowait %steal %idle
5.68 0.00 0.52 2.03 0.00 91.76
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 27.86 63.53 61.77 132.91 1096.46 1598.40 13.84 0.21 1.06 2.28 44.45
sda1 0.69 33.22 48.54 129.63 773.30 1328.84 11.80 1.39 7.82 2.28 40.57
sda2 27.16 30.32 13.23 3.28 323.13 269.56 35.90 0.55 32.96 3.44 5.68
sdb 39.15 215.16 202.20 169.04 945.80 1073.13 5.44 1.05 2.78 1.64 60.91
sdb1 39.15 215.16 202.20 169.04 945.77 1073.13 5.44 1.05 2.78 1.64 60.91
sdc 8.90 3.63 356.56 51.40 207.01 972.24 2.89 1.04 2.56 1.55 63.30
sdc1 8.90 3.63 356.55 51.40 206.99 972.24 2.89 1.04 2.56 1.55 63.30
To display extended information for a specific partition (For example, /dev/sda1), do the following.
$ iostat -x sda1
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011
avg-cpu: %user %nice %system %iowait %steal %idle
5.68 0.00 0.52 2.03 0.00 91.76
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda1 0.69 33.21 48.54 129.62 773.23 1328.76 11.80 1.39 7.82 2.28 40.56
9. iostat – Execute Every x seconds (for y number of times)
To execute iostat every 2 seconds (until you press Ctl-C), do the following.
$ iostat 2
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011
avg-cpu: %user %nice %system %iowait %steal %idle
5.68 0.00 0.52 2.03 0.00 91.76
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 194.67 1096.39 1598.33 2719070584 3963891256
sda1 178.16 773.26 1328.79 1917688482 3295418672
sda2 16.51 323.11 269.54 801326878 668472584
sdb 371.22 945.74 1073.08 2345454041 2661251200
sdb1 371.22 945.72 1073.08 2345398577 2661251200
sdc 407.93 207.00 972.19 513366813 2411036564
sdc1 407.93 206.98 972.19 513311349 2411036564
..
To execute every 2 seconds for a total of 3 times, do the following.
$ iostat 2 3
10. iostat – Display LVM statistic (and version)
To display the LVM statistics use option -N as shown below.
$ iostat -N
To display the version of iostat, use -V. This will really display the version information of sysstat, as iostat is part of sysstat package.
$ iostat -V sysstat version 7.0.2 (C) Sebastien Godard
VMSTAT EXAMPLES
11. vmstat – Basic example
vmstat by default will display the memory usage (including swap) as shown below.
$ vmstat procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 305416 260688 29160 2356920 2 2 4 1 0 0 6 1 92 2 0
vmstat output contains the following fields:
- Procs – r: Total number of processes waiting to run
- Procs – b: Total number of busy processes
- Memory – swpd: Used virtual memory
- Memory – free: Free virtual memory
- Memory – buff: Memory used as buffers
- Memory – cache: Memory used as cache.
- Swap – si: Memory swapped from disk (for every second)
- Swap – so: Memory swapped to disk (for every second)
- IO – bi: Blocks in. i.e blocks received from device (for every second)
- IO – bo: Blocks out. i.e blocks sent to the device (for every second)
- System – in: Interrupts per second
- System – cs: Context switches
- CPU – us, sy, id, wa, st: CPU user time, system time, idle time, wait time
12. vmstat – Display active and inactive memory
By default vmstat doesn’t display this information. Use option -a, to display active and inactive memory information as shown below.
$ vmstat -a procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free inact active si so bi bo in cs us sy id wa st 0 0 305416 253820 1052680 2688928 2 2 4 1 0 0 6 1 92 2 0
13. vmstat – Display number of forks since last boot
This displays all the fork system calls made by the system since the last boot. This displays all fork, vfork, and clone system call counts.
$ vmstat -f
81651975 forks
14. vmstat – Execute Every x seconds (for y number of times)
To execute every 2 seconds, do the following. You have to press Ctrl-C to stop this.
$ vmstat 2 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 537144 182736 6789320 0 0 0 0 1 1 0 0 100 0 0 0 0 0 537004 182736 6789320 0 0 0 0 50 32 0 0 100 0 0 ..
To execute every 2 seconds for 10 times, do the following. You don’t need to press Ctrl-C in this case. After executing 10 times, it will stop automatically.
$ vmstat 2 10 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 537144 182736 6789320 0 0 0 0 1 1 0 0 100 0 0 0 0 0 537004 182736 6789320 0 0 0 0 50 32 0 0 100 0 0 ..
15. vmstat – Display timestamp
When you use vmstat to monitor the memory usage repeately, it would be nice to see the timestap along with every line item. Use option -t to display the time stamp as shown below.
$ vmstat -t 1 100 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ ---timestamp--- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 3608728 148368 3898200 0 0 0 0 1 1 0 0 100 0 0 2011-07-09 21:16:28 PDT 0 0 0 3608728 148368 3898200 0 0 0 0 60 15 0 0 100 0 0 2011-07-09 21:16:29 PDT 0 0 0 3608712 148368 3898200 0 0 0 0 32 28 0 0 100 0 0 2011-07-09 21:16:30 PDT
For me, the timestamp option worked in the following version.
$ vmstat -V procps version 3.2.8
Note: If you use a older version of vmstat, option -t might not be available. In that case, use the method we suggested earlier to display timestamp in vmstat output.
16. vmstat – Display slab info
Use option -m, to display the slab info as shown below.
$ vmstat -m Cache Num Total Size Pages fib6_nodes 5 113 32 113 ip6_dst_cache 4 15 256 15 ndisc_cache 1 15 256 15 RAWv6 7 10 768 5 UDPv6 0 0 640 6 tw_sock_TCPv6 0 0 128 30 ...
17. vmstat – Display statistics in a table format
Instead of displays the values in the record format, you can display the output of vmstat in table format using option -s as shown below.
$ vmstat -s
4149928 total memory
3864824 used memory
2606664 active memory
1098180 inactive memory
285104 free memory
19264 buffer memory
2326692 swap cache
4192956 total swap
274872 used swap
3918084 free swap
1032454000 non-nice user cpu ticks
14568 nice user cpu ticks
89482270 system cpu ticks
16674327143 idle cpu ticks
368965706 IO-wait cpu ticks
1180468 IRQ cpu ticks
..
18. vmstat – Display disk statistics
Use option -d to display the disk statistics as shown below. This displays the reads, writes, and I/O statistics of the disk.
$ vmstat -d
disk- ------------reads------------ ------------writes----------- -----IO------
total merged sectors ms total merged sectors ms cur sec
sda 153189971 69093708 2719150864 737822879 329617713 157559204 3965687592 4068577985 0 1102243
sdb 501426305 97099356 2345472425 731613156 419220973 533565961 2661869460 1825174087 0 1510434
sdc 884213459 22078974 513390701 452540172 127474901 8993357 2411187300 2133226954 0 1569758
19. vmstat – Increase the width of the display
The default output without increasing the width is shown below.
$ vmstat 1 3 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 3608688 148368 3898204 0 0 0 0 1 1 0 0 100 0 0 0 0 0 3608804 148368 3898204 0 0 0 0 72 30 0 0 100 0 0 0 0 0 3608804 148368 3898204 0 0 0 0 60 27 0 0 100 0 0
Use option -w to increase the width of the output columns as shown below. This give better readability.
$ vmstat -w 1 3 procs -------------------memory------------------ ---swap-- -----io---- --system-- -----cpu------- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 3608712 148368 3898204 0 0 0 0 1 1 0 0 100 0 0 0 0 0 3608712 148368 3898204 0 0 0 0 93 23 0 0 100 0 0 0 0 0 3608696 148368 3898204 0 0 0 0 35 34 0 0 100 0 0
20. vmstat – Display statistics for a partition
To display the disk I/O statistics of a specific disk partition use option -p as shown below.
$ vmstat -p sdb1
sdb1 reads read sectors writes requested writes
501423248 2345417917 419221612 2661885948
21. vmstat – Display in MB
By default vmstat displays the memory information in kb. To disply in MB, use the option “-S m” as shown below.
$ vmstat -S m procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 281 288 19 2386 0 0 4 1 0 0 6 1 92 2 0
MPSTAT EXAMPLES
22. mpstat – Display basic info
By default mpstat displays CPU statistics as shown below.
$ mpstat Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 10:25:32 PM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s 10:25:32 PM all 5.68 0.00 0.49 2.03 0.01 0.02 0.00 91.77 146.55
23. mpstat – Display all information
Option -A, displays all the information that can be displayed by the mpstat command as shown below. This is really equalivalent to “mpstat -I ALL -u -P ALL” command.
$ mpstat -A Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 _x86_64_ (4 CPU) 10:26:34 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 10:26:34 PM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.99 10:26:34 PM 0 0.01 0.00 0.01 0.01 0.00 0.00 0.00 0.00 99.98 10:26:34 PM 1 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 99.98 10:26:34 PM 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 10:26:34 PM 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 10:26:34 PM CPU intr/s 10:26:34 PM all 36.51 10:26:34 PM 0 0.00 10:26:34 PM 1 0.00 10:26:34 PM 2 0.04 10:26:34 PM 3 0.00 10:26:34 PM CPU 0/s 1/s 8/s 9/s 12/s 14/s 15/s 16/s 19/s 20/s 21/s 33/s NMI/s LOC/s SPU/s PMI/s PND/s RES/s CAL/s TLB/s TRM/s THR/s MCE/s MCP/s ERR/s MIS/s 10:26:34 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 7.47 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:26:34 PM 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 4.90 0.00 0.00 0.00 0.00 0.03 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:26:34 PM 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.04 0.00 0.00 0.00 0.00 0.00 3.32 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:26:34 PM 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 4.17 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00
24. mpstat – Display CPU statistics of individual CPU (or) Core
Option -P ALL, displays all the individual CPUs (or Cores) along with its statistics as shown below.
$ mpstat -P ALL Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 _x86_64_ (4 CPU) 10:28:04 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 10:28:04 PM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.99 10:28:04 PM 0 0.01 0.00 0.01 0.01 0.00 0.00 0.00 0.00 99.98 10:28:04 PM 1 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 99.98 10:28:04 PM 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 10:28:04 PM 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
To display statistics information of a particular CPU (or core), use option -P as shown below.
$ mpstat -P 0 Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 _x86_64_ (8 CPU) 10:28:53 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 10:28:53 PM 0 0.01 0.00 0.01 0.01 0.00 0.00 0.00 0.00 99.98 $ mpstat -P 1 Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 _x86_64_ (8 CPU) 10:28:55 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 10:28:55 PM 1 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 99.98
Finally, as we mentioned earlier mpstat is part of the sysstat package. When you do mpstat -V, it will really display the version number of the systat package as shown below.
$ mpstat -V sysstat version 9.0.4 (C) Sebastien Godard (sysstat orange.fr)
Previous articles in the Linux performance monitoring and tuning series:
My name is Ramesh Natarajan. I will be posting instruction guides, how-to, troubleshooting tips and tricks on Linux, database, hardware, security and web. My focus is to write articles that will either teach you or help you resolve a problem. Read more about
Comments on this entry are closed.
Hi Ramesh,
Nice article, Loved it, I bought your book on Nagios, Found a little command typos and some info update would be grate will drop a mail with the info
The vmstat -w and vmstat -t commands return:
# vmstat -w
usage: vmstat [-V] [-n] [delay [count]]
-V prints version.
-n causes the headers not to be reprinted regularly.
-a print inactive/active page stats.
-d prints disk statistics
-D prints disk table
-p prints disk partition statistics
-s prints vm table
-m prints slabinfo
-S unit size
delay is the delay between updates in seconds.
unit size k:1000 K:1024 m:1000000 M:1048576 (default is K)
count is the number of updates.
obs: SLES 11 SP1
sysstat package, no systat.
Hi Brother,
This is an another great stuff by you. I enjoyed it. Am looking forward to your next article.
Thanks for your knowledge sharing!!!!!!!
These are all very useful, but they leave out one important bit of info about a specific need. What if you have a process doing high I/O, but you don’t know which process it is? The tools listed above will help you narrow down which disk/partition might be the busiest but will not help you find the process.
Enter, iotop.
iotop is a “top” like tool that list the busiest process by I/O.
bb
These are indeed useful tool but also contribute to what I think is an big problem with many utilities. They only provide realtime results. If you want historical data you need to run other tools, like sar. BUT by the time you realize you need that data you weren’t running sar. And if you were you were probably doing it at the default interval of 10 minutes which I find pretty useless for intermittent problems.
I wanted one tool that I could keep going back to both for real-time as well as historical reporting. I wanted a consistent output format which none of the tools seems to use, epecially since they were all written by different people. I do realize that sometimes you need different formats so my ideal tool should do that as well. And of course if I have a lot of data I want to plot it. I don’t want a tool that will plot disk data and a different tool that will plot memory usage. In other words I want it all!
That’s one of the reasons I wrote collectl, which has been around for close to 10 years and is now part of fedora, suse and most recently debian. It can do everything on my wish list and more. Basically every system I work on I just start it and leave it running and it takes samples every 10 seconds – usually sufficient enough for most diagnostic situations yet light weight enough (<0.1% of a cpu) that you don't mind leaving it running continuously.
But now that you have access to a light-weight data collector that has the ability to communicate with other tools, I was able to write colmux, which can communicate with multiple instances of collectl across hundreds or even thousands of node! Now you have the ability to run a top-like utility across a cluster that not only sorts on cpu load, it can sort on anything, be it disks, memory, slabs, process, etc. I wrote a little more about it here – if you want to hear more.
-mark
Hi Ramesh,
Good article with enough examples. Basically I am DBA and I was just trying to understand these OS commands. Thanks, Umasrinivas.
Hello Ramesh , I am using the following command to check my RAID0 drives iostats – “iostat -dx /dev/md4 5” , but avgqu-sz , await , svctm and %util
columns are always zero .
And that is the only drive that my server is actually using , since its serving files from md4 RAID array , but still i am not getting information from the iostat command.
I read this at the iostat man page
“%util
But for devices serving requests in parallel, such as
RAID arrays and modern SSDs, this number does not
reflect their performance limits.”
Do you know of any tools or command that i can use to monitor the disk I/O for my RAID0 drives ?
Great job, Ramesh!
Can we get iostat for a custom time? Say for example, Can I get iostat for previous day?