LatencyTOP

[lightweight]

A coworker pointed me to this today: http://www.latencytop.org/ It is described as

About LatencyTOP
Skipping audio, slower servers, everyone knows the symptoms of latency. But to know what's going on in the system, what's causing the latency, how to fix it... that's a hard question without good answers right now.

LatencyTOP is a Linux* tool for software developers (both kernel and userspace), aimed at identifying where in the system latency is happening, and what kind of operation/action is causing the latency to happen so that the code can be changed to avoid the worst latency hiccups.

About latency
There are many types and causes of latency. LatencyTOP focuses on the type of latency that causes skips in audio, stutters in your desktop experience or that overloads your server (while you have plenty of CPU power left).

LatencyTOP focuses on the cases where the applications want to run and execute useful code, but there's some resource that's not currently available (and the kernel then blocks the process). This is done both on a system level and on a per process level, so that you can see what's happening to the system, and which process is suffering and/or causing the delays.

From the PowerTOP/lesswatts.org guy(s?). He describes it for developers but any curious Linux user will enjoy this program. One can use it as a tool to optimize their machine or find out what exactly is hanging or whatever. Its just a great tool.

Minus:
Needs KConfig
Needs a kernel built with
CONFIG_LATENCYTOP=y
and
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
so you may need to recompile. (Those using Debian repositories will.) <3