Introduction to Linux Interrupts and CPU SMP Affinity

Interrupts are signal that are sent across IRQ (Interrupt Request Line) by a hardware or software.

Interrupts allow devices like keyboard, serial cards and parallel ports to indicate that it needs CPU attention.

Once the CPU receives the Interrupt Request, CPU will temporarily stop execution of running program and invoke a special program called Interrupt Handler or ISR (Interrupt Service Routine).

The Interrupt Service or Interrupt Handler Routine can be found in Interrupt Vector table that is located at fixed address in the memory. After the interrupt is handled CPU resumes the interrupted program.

At boot time, system identifies all devices, and appropriate interrupt handlers are loaded into the interrupt table.

The following are two ways of requesting CPU attention:

1. Interrupt based
2. Polling based

All Linux based OS are interrupt driven.

When we press a key on keyboard, keyboards says to CPU that a key has been pressed. But CPU can be busy processing some stuff from RAM, System Clock, NIC card, may be video or PCI bus. In that case Keyboard places a voltage on IRQ line assigned to that hardware, here in this case [Keyboard]. This change in voltage serves as request from device saying that device has a request that needs processing.

/proc/interrupts File

On a Linux machine, the file /proc/interrupts contains information about the interrupts in use and how many times processor has been interrupted

# cat /proc/interrupts
           CPU0   CPU1  CPU2  CPU3
  0: 3710374484      0     0     0  IO-APIC-edge  timer
  1:         20      0     0     0  IO-APIC-edge  i8042
  6:          5      0     0     0  IO-APIC-edge  floppy
  7:          0      0     0     0  IO-APIC-edge  parport0
  8:          0      0     0     0  IO-APIC-edge  rtc
  9:          0      0     0     0  IO-APIC-level  acpi
 12:        240      0     0     0  IO-APIC-edge  i8042
 14:   11200026      0     0     0  IO-APIC-edge  ide0
 51:   61281329      0     0     0  IO-APIC-level  ioc0
 59:          1      0     0     0  IO-APIC-level  vmci
 67:   19386473      0     0     0  IO-APIC-level  eth0
 75:   94595340      0     0     0  IO-APIC-level  eth1
NMI:          0      0     0     0
LOC: 3737150067 3737142382 3737145101 3737144204
ERR:          0
MIS:          0

In the above file:

• The first Column is the IRQ number.
• The Second column says how many times the CPU core has been interrupted. In the above example timer is interrupt name [System clock] and 3710374484 is the number of times CPU0 has been interrupted. I8042 is Keyboard controller that controls PS/2 keyboards and mouse in Pc’s.
• For interrupt like rtc [Real time clock] CPU has not being interrupted. RTC are present in electronic devices to keep track of time.
• NMI and LOC are drivers used on system that are not accessible/configured by user.

IRQ number determines the priority of the interrupt that needs to be handled by the CPU.

A small IRQ number value means higher priority.

For example if CPU receives interrupt from Keyboard and system clock simultaneously. CPU will serve System Clock first since it has IRQ number 0.

• IRQ 0 — system timer (cannot be changed);
• IRQ 1 — keyboard controller (cannot be changed)
• IRQ 3 — serial port controller for serial port 2 (shared with serial port 4, if present);
• IRQ 4 — serial port controller for serial port 1 (shared with serial port 3, if present);
• IRQ 5 — parallel port 2 and 3 or sound card;
• IRQ 6 — floppy disk controller;
• IRQ 7 — parallel port 1. It is used for printers or for any parallel port if a printer is not present.

For devices like joystick CPU doesn’t wait for the device to send interrupt. Since Joystick used for gaming and the movement of joystick will be fast it will be ideal to use polling and check whether device needs attention. The disadvantage behind this method is CPU can get into busy wait, checking the device many times.

On a related note, it is also essential to handle the signals properly in Linux.

Hardware Interrupts

All of the above discussed scenarios are example of Hardware interrupts.

Hardware interrupts are further classified into two major categories:

1. Non-maskable interrupts [NMI]: As the name suggests these types of interrupts cannot be ignored or suppressed by the CPU. MNI’s are send over separate interrupt line and it’s generally used for critical hardware errors like memory error, Hardware traps indicating Fan failure, Temperature Sensor failure etc.
2. Maskable interrupts: These interrupts can be ignored or delayed by CPU. The Interrupt Mask Register masks the interrupts being triggered on external pins of cache controller. Setting a bit by writing a 0, disables the interrupt triggering on the pin

Software Interrupts

These interrupts are generated when the CPU executes an instruction which can cause an exception condition in the CPU [ALU unit] itself.

For example, divide a number by zero which is not possible, it will lead to divide-by-zero exception, causing the computer to abandon the calculation or display an error message.

The file /proc/stat is also a file part of the /proc filesystem, which has information about system kernel statistics, also holds some interrupt information.

# cat /proc/stat
cpu  17028082 5536753 5081493 1735530500 42592308 90006 479750 0
cpu0 5769176 1170683 1495750 403368354 39406374 90006 284864 0
cpu1 3714389 1451937 1186134 444082258 1084780 0 64876 0
cpu2 3791544 1471013 1211868 443988514 1056981 0 64764 0
cpu3 3752971 1443119 1187740 444091373 1044172 0 65244 0
intr 417756956 --- Output Truncated

The line intr shows the count of the interrupt serviced since boot time. The first column is total of all interrupts serviced. Each subsequent column is the total for a particular interrupt.

SMP_AFFINITY

Symmetric multiprocessing is the processing of programs by multiple processors.

smp_affinity file holds interrupt affinity value for a IRQ number. The smp_affinity file associated with each IRQ number is stored in /proc/irq/IRQ_NUMBER/smp_affinity file. The value in the file is stored in hexadecimal bit-mask representing all CPU cores in the system. smp_affinity works for device that has IO-APIC enabled device drivers.

For example, smp_affinity entry for Ethernet driver is shown below:

grep eth0 /proc/interrupts
67: 23834931 0 0 0 IO-APIC-level eth0

IRQ number for eth0 is 67 and corresponding smp_affinity file is located at:

cat /proc/irq/67/smp_affinity
00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001

The decimal equivalent for value ‘000000001’ is ‘1’. ie All the interrupt related to Ethernet driver will be serviced by CPU0.

We can manually change the processor affinity by changing values in smp_affinity file for a particular controller or by using irqbalance.

IRQ Balance

Irqbalance is a Linux utility that distributes interrupts over the processor cores in your computer system which helps to improve performance.

Irqbalance goal is to find a balance between power saving and optimal performance.

If irqbalance is not installed on your system, install it using yum as shown below.

# rpm -qa | grep irqbalance
irqbalance-0.55-15.el5

# yum search irqbalance

# yum install irqbalance.x86_64

Start the irqbalance service:

service irqbalance start

The following is a sample output from a Linux machine where irqbalance is installed. We could see that interrupts are now being distributed between CPUs.

# cat /proc/interrupts
           CPU0     CPU1      CPU2       CPU3
  0:  950901695        0         0          0  IO-APIC-edge  timer
  1:         13        0         0          0  IO-APIC-edge  i8042
  6:         96    10989       470          0  IO-APIC-edge  floppy
  7:          0        0         0          0  IO-APIC-edge  parport0
  8:          1        0         0          0  IO-APIC-edge  rtc
  9:          0        0         0          0  IO-APIC-level  acpi
 12:        109     1787         0          0  IO-APIC-edge  i8042
 15:         99 84813914         0          0  IO-APIC-edge  ide1
 51:      17371        0  46689970          0  IO-APIC-level  ioc0
 67:       1741        0         0  225409160  PCI-MSI  eth0
 83:          0        0         0          0  PCI-MSI  vmci
NMI:          0        0         0          0
LOC:  950902917  950903742  950901202  950901400
ERR:          0
MIS:          0

Irqbalance is especially useful on systems with multi-core processors, as interrupts will typically only be serviced by the first core.