CPU Hotplug

Linux’s CPU-Hotplug facility was originally designed to allow failing hardware to be removed from a running system. Hardware fails quite infrequently, so CPU-hotplug performance (much less real-time response) was not a major consideration. However, CPU hotplug is now used for energy management and (believe it or not!) realtime response, both of which have unsurprisingly exposed some shortcomings in CPU hotplug. This document reviews a number of these shortcomings, and then proposes an alternative CPU-hotplug approach that we believe will address these shortcomings.

Linux’s CPU-hotplug implementation is based on notifiers, which are callbacks into the subsystems that need to be aware of CPUs coming and going. These noti- fiers are invoked repeatedly in multiple phases, so that when a CPU is coming online, they are invoked with CPU UP PREPARE (which runs on some other CPU), then CPU STARTING (which runs with interrupts disabled on the CPU coming online), and finally CPU ONLINE (which might run on any CPU, but after the CPU has come online). CPUs going offline have four notification phases: CPU DOWN PREPARE (which might run on any CPU), CPU DYING (which runs with interrupts disabled on the offlining CPU while all other CPUs are spinning waiting), CPU DEAD (which runs on some other CPU after the CPU has gone offline), and CPU POST DEAD (which runs on some other CPU after some of the CPU-hotplug locks have been dropped). The CPU UP PREPARE and CPU DOWN PREPARE notifiers are permitted to “fail”, in other words, to refuse to allow the hotplug operation to proceed.