0% scroll to read · click to mark done
← all posts ◈ Linux 100 · #03

System Monitoring Deep Dive: What top Doesn't Tell You (and What Does)

Apr 17, 2026· 8 min read· #linux#monitoring#performance#iostat#vmstat
PrerequisitesBasic Linux CLItop/htop usagesudo access

top is where everyone starts and where most people stop. But the most common "the server is slow and top looks fine" incidents are invisible to top's default view. Here's the rest of the stack.

I/O wait — the slowdown top hides

eknatha@prod ~
$ vmstat 2 5
r b swpd free buff cache wa st 4 8 0 12000 2000 400000 68 0 ← wa=68% I/O wait!

wa is the percentage of time the CPU sat idle waiting for disk. 68% means two-thirds of your "CPU time" is the processor twiddling its thumbs waiting for I/O. top's load average looks high, but the CPU isn't the bottleneck — the disk is. The b column (processes blocked on I/O) confirms it.

Confirm at the device level

eknatha@prod ~
$ iostat -x 2 | grep -v "^{{BODY}}quot;
sda r/s: 0.0 w/s: 842.0 await: 180ms ← saturated

await is average latency per I/O in milliseconds. Healthy SSD await is single digits. 180ms means the disk is saturated — likely a runaway writer or a failing drive. %util near 100% confirms saturation.

CPU steal — the noisy-neighbour tax

The st column is steal time — CPU cycles your VM wanted but the hypervisor gave to another tenant:

eknatha@prod ~
$ vmstat 2
... st 14 ← 14% stolen by neighbours

If st is consistently high on a cloud VM, your instance is being starved by co-tenants. No amount of in-guest tuning fixes it — you resize or move.

Memory pressure vs swap usage

People panic at any swap usage. That's wrong. The question is whether you're actively swapping:

eknatha@prod ~
$ vmstat 2
si so 0 4200 ← so = pages swapping OUT per second

si/so (swap in/out) near zero with swap used is fine — the kernel parked idle pages. so continuously above zero means you're thrashing, and that's the emergency.

The decision tree

SymptomLook atCause
High load, low CPUwa in vmstatI/O bottleneck
High load on cloud VMstnoisy neighbour
Slow despite free RAMsi/soactive swapping
One core pinnedtop then -1single-threaded hog
top tells you something is wrong. vmstat and iostat tell you what.
// written from production experience — by Eknatha