How to read a flamegraph: a 10-minute guide for production debugging

A flamegraph is the single most useful thing I look at when a service is mysteriously slow. It tells you, in one picture, where the CPU is actually going — not where you think it’s going.

This post explains how to read one without having to be a Linux performance wizard. Three worked examples, four patterns you’ll see repeatedly, and a short list of how to capture flamegraphs from the runtimes most teams actually use.

What a flamegraph actually is

Take this tiny program:

def serve_request():
    log_request()                # quick — barely any CPU
    return parse_payload()       # expensive — most of the CPU

def parse_payload():
    return json_decode(raw_body) # expensive part of parse_payload

You can capture a flamegraph of it yourself in two commands. Save this complete version as serve.py:

# serve.py
import json

raw_body = '{"items":[' + ','.join('{"id":%d}' % i for i in range(500)) + ']}'

def log_request():
    pass

def json_decode(b):
    return json.loads(b)

def parse_payload():
    return json_decode(raw_body)

def serve_request():
    log_request()
    return parse_payload()

if __name__ == "__main__":
    for _ in range(500_000):
        serve_request()

Then run it under py-spy — a sampling profiler that needs no code changes:

pip install py-spy
py-spy record -o flame.svg --rate 99 -- python serve.py

Open flame.svg in a browser. You’ll see exactly the shape described below — serve_request at the bottom, parse_payload wide above it, log_request narrow next to it, json_decode on top.

A flamegraph samples what your program is doing 99 times per second. Each sample records the chain of function calls that was running at that instant — the call stack. After 30 seconds you have ~3000 samples like:

sample 1:   serve_request → parse_payload → json_decode
sample 2:   serve_request → parse_payload → json_decode
sample 3:   serve_request → log_request
sample 4:   serve_request → parse_payload → json_decode
...

Then you draw it. Each function becomes a rectangle. The rectangle’s width is how many samples saw that function on the stack. Stack the rectangles vertically: the function that called a child sits below the child.

That’s the picture. For our example it looks like this (rough ASCII rendering — real flamegraphs are SVGs but the shape is identical):

                ┌──────────────────────────┐
top of flame    │  json_decode (70%)       │   ← deepest call
                ├──────────────────────────┼───────────┐
                │  parse_payload (85%)     │ log_req(5%)│
                ├──────────────────────────┴───────────┤
bottom of flame │  serve_request (100%)                │   ← entry point
                └──────────────────────────────────────┘

Three things to notice:

1. serve_request is the widest rectangle, at the very bottom — it was on the stack for every single sample (100%) because the program is serve_request.
2. parse_payload (85%) and log_request (5%) sit side by side above their parent. Their widths together can’t exceed the parent’s width, because they were both called from inside serve_request. They’re 90% combined; the missing 10% is samples that landed in serve_request’s own body before either child was called.
3. json_decode is narrower than parse_payload because some of parse_payload’s time was spent in its own code, not yet in json_decode.

The visual ends up looking like flames — wide base, narrowing peaks. That’s where the name comes from.

What the actual flamegraph looks like

Here’s the real flamegraph from running the serve.py and py-spy record commands above — not an illustration, the actual SVG output (10,304 samples in a 30s capture):

A few things to notice on the real picture vs the ASCII version above:

• The log_request rectangle is tiny. It’s there — about 0.2% of samples — but at 0.2% wide it’s barely a vertical line on the right side of the flame. The ASCII version drew it 5% wide for clarity; the real one shows that “cheap function” really means barely-visible-on-the-flamegraph cheap.
• json_decode is just barely narrower than parse_payload — 90.48% vs 90.61%. The ASCII drew them visibly different. In real life, when one function does almost nothing except call another, parent and child bars are nearly identical width — and that’s a useful signal (“parse_payload is almost pure overhead; the real work is in the child”).
• A whole stack of json library frames stacks above json_decode — loads, decode, raw_decode, down to the C-extension scanner. The ASCII version stopped at json_decode; the real picture shows that the cost is deep inside the json library, not in the user code that called it.
• Total sample count (10,304) appears at the implicit “all” frame at the very top of the SVG. That’s the number to compare across captures, not just the per-frame percentages — covered later in “Common mistakes.”

Click any rectangle in the SVG (open it in a new tab via the link above) to zoom into that subtree. That interactivity is one of the reasons flamegraphs ship as SVGs and not PNGs.

The four reading rules

• Width = CPU time. A 50%-wide rectangle was on the stack for 50% of the samples. That’s the only thing width means.
• Height = how deep the call stack went. Tall doesn’t mean slow. A function called from 20 layers deep can be a 1% rectangle if it ran briefly.
• The horizontal order is NOT time. The leftmost rectangle didn’t happen first. Real flamegraphs sort siblings alphabetically (or by stack identity) so the same function lands in the same horizontal position across captures, making before/after comparisons easier. Don’t try to read it as “what happened next.”
• Color usually means nothing. Most renderers color rectangles randomly so adjacent ones are distinguishable. Some renderers color by language (your code vs library vs kernel), which is useful — but treat color as decoration unless you’ve checked what your renderer assigns.

If you remember nothing else: wider = more CPU. The rectangles you care about are the wide ones near the top of the flame (the ones at the bottom are usually main / serve_request / equivalent — they’re 100% wide by definition because something in your program is always running). What matters is which child of the entry point is wide, and which child of that is wide.

What to look for: four patterns

Once you’ve read a flamegraph or two, the same shapes start showing up.

Pattern 1: a wide plateau

┌────────────────────────────────────────────────────────┐
│  parse_internal (1370 samples · 52.69%)                │
└────────────────────────────────────────────────────────┘

A single frame that’s eating half the CPU. Almost always either an unintended hot loop, a function being called more often than you realized, or a known-expensive call (regex compile, format spec parse, JSON decode) on a per-message path. This is the easy case — the flamegraph hands you the answer.

Pattern 2: a tall thin tower

                                         ╱╲
                                        ╱  ╲
                                       ╱    ╲
                              recursive_walk (8% total)

Deep stack, small width. Usually recursion or a deeply layered abstraction. Tall isn’t slow on its own. But if a tall tower is also wide at the base, you might have over-recursion or an N+1 pattern (calling a function once per record, where the function is itself doing N work).

Pattern 3: many narrow tips at the same level

fn_A (1%)  fn_B (1%)  fn_C (1%)  fn_D (2%)  fn_E (1%)  ...
└────────── all called from dispatch_message ──────────┘

The cost is real but spread across many functions, each cheap. Often shows up in dispatch loops or registries with many handlers. The fix is rarely “speed up fn_A” — it’s “do less dispatching” (batch, gate, reduce fan-out).

Pattern 4: the hot frame hiding under another hot frame

This one bites engineers who declare victory too early. You fix the visible top frame, redeploy, and the next flamegraph shows… a different frame at 50%. The new frame was always there. It just sat in the second half of the previous flamegraph, looking smaller in relative terms because the visible top frame was eating so much.

The trick is to read the total sample count alongside the percentages. If your post-fix flamegraph has 5× fewer samples in the same wall-clock window, you actually reduced CPU even if a single frame’s percentage looks unchanged.

Worked example 1: the obvious hot loop

The same bug shows the same flamegraph shape across languages. Two versions of an O(N) scan accidentally nested inside an O(N) iterator — once in Rust, once in Python:

// Rust
fn process_request(records: &[Record]) -> Vec<Output> {
    records.iter().map(|r| {
        // unintended O(N) scan inside an O(N) iterator
        let user = USERS.iter().find(|u| u.id == r.user_id).unwrap();
        Output::new(r, user)
    }).collect()
}

# Python — same bug
def process_request(records):
    return [
        Output(r, next(u for u in USERS if u.id == r.user_id))
        for r in records
    ]

If records has 1000 entries and USERS has 10,000, both versions do 10 million inner comparisons per call. The flamegraph (whether captured by pprof-rs for the Rust binary or py-spy for the Python one) will look almost identical:

process_request (98%) ──────────────────────────────
   └── <iterator next> (95%) ───────────────────────
       └── <linear find> (93%) ─────────────────────
           └── <id equality> (88%) ─────────────────

A wide tower from process_request down through the linear find to the equality comparison. The shape is the giveaway: 88% of samples are in an equality check inside a find/scan call.

The fix is structural in both languages — replace the linear scan with a hashmap lookup built once:

// Rust
let users_by_id: HashMap<UserId, &User> = USERS.iter().map(|u| (u.id, u)).collect();
let user = users_by_id.get(&r.user_id).unwrap();

# Python
users_by_id = {u.id: u for u in USERS}
user = users_by_id[r.user_id]

The next flamegraph would show process_request shrinking dramatically, with whatever was previously the second-widest frame now at the top. Same diagnosis, same fix shape, different syntax. Reading the flamegraph is a language-neutral skill.

Worked example 2: the misleading top frame

Sometimes the wide plateau is a symptom, not the cause. The expensive function isn’t slow on its own — it’s being called too often, doing setup work on every call that should happen once.

This pattern shows up in nearly every language. Two cases, same shape on the flamegraph.

Case A: Rust + chrono date format spec

chrono::format::parse::parse_internal (52.69%) ────────
   └── chrono::format::strftime::StrftimeItems::next (33.15%) ───
       └── <chrono::DateTime as serde::Serialize>::serialize (...)
           └── serde_json::value::to_value (...)
               └── build_metrics_jsonb (a small frame)
                   └── timescaledb_sink::write_event (...)
                       └── analytics_runtime::run (...)

Half the CPU is in chrono’s date-format parser. The reflex is “chrono is slow.” But chrono’s parser isn’t slow — it’s invoked from a per-message path many times a second, and it re-parses the format spec string on every single call. There’s no compile-time format folding.

The fix isn’t a faster date library. The fix is to call the parser once at startup and reuse the result:

use std::sync::OnceLock;
use chrono::format::{Item, StrftimeItems};

fn iso_format() -> &'static [Item<'static>] {
    static ITEMS: OnceLock<Vec<Item<'static>>> = OnceLock::new();
    ITEMS.get_or_init(|| StrftimeItems::new("%Y-%m-%dT%H:%M:%S%.fZ").collect())
}

Case B: Python + uncompiled regex

The exact same pattern is the most common Python perf pitfall:

re.search (45%) ───────────────────────────────────
   └── _compile (40%) ─────────────────────────────
       └── sre_compile.compile (38%) ──────────────
           └── _parse (35%) ───────────────────────
               └── parse_template (30%) ───────────

Python’s re.search(pattern, text) compiles the pattern on every call (with an internal LRU cache that helps for fixed patterns but thrashes when patterns vary). On a hot path, you’ll see frames from sre_compile, _parse, and friends adding up to half the CPU:

# Slow — recompiles on every call
def extract_id(line):
    return re.search(r"id=(\d+)", line).group(1)

# Fast — compile once at module load, reuse forever
_ID_PATTERN = re.compile(r"id=(\d+)")

def extract_id(line):
    return _ID_PATTERN.search(line).group(1)

Same lesson across both cases

The plateau frame (chrono’s parser, Python’s _compile) was running fast per call — the unit cost was tiny. But the function was reachable from a per-message stack and got called millions of times. Look for “expensive setup work being done per call instead of once” any time a parser/compiler/initializer frame shows up on a per-message hot path.

This is also why you read a flamegraph bottom-up to find what’s hot, then top-down from the hot frame to understand why. The top of the tower tells you what cost the CPU. The bottom of the tower (the call chain leading down to it) tells you who’s responsible for calling it that often.

Worked example 3: nothing is hot, but everything is slow

Sometimes the flamegraph is flat — no wide plateaus, just dozens of small frames at 2-5% each.

This usually means one of two things:

1. Allocator pressure. Your program is spending time on memory allocation across many call sites, none individually dominant. Frame names that show up:

   • Rust: __rust_alloc, _int_malloc, mmap, kernel page-fault frames. Symptom of frequent Vec::new(), String::from, Box::new in a loop. Fix: reuse buffers, intern strings, switch allocator (jemalloc, mimalloc).
   • Python: gc_collect_main, frame_alloc, tuple_alloc from CPython internals. Symptom of building lots of throwaway tuples / dicts / list comprehensions per request. Fix: use generators, reuse mutable structures, drop CPython object overhead via __slots__ or attrs/dataclass(slots=True).

2. You’re not actually CPU-bound. The program is waiting on I/O, locks, or external services. The CPU samples that did land are scattered because the program was off-CPU (sleeping, waiting on a socket) most of the time. Frame names that show up:

   • Rust: tokio::io::poll_read, epoll_wait, parking_lot::raw_mutex::RawMutex::lock. Use an off-CPU profiler or async-aware tool like tokio-console.
   • Python: socket.recv, select.poll, GIL-related frames like take_gil. Use py-spy --idle to include time-spent-waiting in the profile, or trace with austin.

If a flamegraph looks flat and your CPU usage is high, suspect (1) — allocator pressure. If flat and CPU is low, suspect (2) — wrong tool, the work isn’t on-CPU at all.

How to capture a flamegraph

The capture step is the bit that scares people, but the modern tools have made it easy. Pick one path:

Runtime	Easiest tool	Notes
Rust	pprof-rs HTTP endpoint, or cargo-flamegraph	pprof-rs lets you curl a flamegraph from a running container. cargo-flamegraph wraps a cargo run invocation.
Python	py-spy	py-spy record -o profile.svg --pid <PID>. No code changes, attaches to a running process. austin and pyinstrument are alternatives — austin handles native frames, pyinstrument is in-process and great for tests.
Node.js	--prof flag + flamegraph from 0x	npx 0x your-script.js.
Java	async-profiler	The standard. Works on JVM and HotSpot internals.
Go	Built-in net/http/pprof + go tool pprof with -svg	Same workflow as pprof-rs; same wire format.
Anything Linux	perf record + Brendan Gregg’s flamegraph.pl	The original tool. Works on any binary, including kernel frames. Some setup.

For long-running services, the production-friendly pattern is a pprof-style HTTP endpoint: it lets ops capture a profile from a running container without restart, redeploy, or even SSH access:

docker exec my_service curl -sf \
  'http://localhost:8001/debug/pprof/flamegraph?seconds=30' \
  -o /tmp/profile.svg

30 seconds of sampling, write to an SVG, open it in a browser. The pprof endpoint adds zero idle overhead — the SIGPROF handler is registered only inside the request handler.

What a good flamegraph workflow looks like

1. Capture a baseline while the symptom is happening. Don’t try to read a flamegraph from a healthy run hoping to see “what would be slow under load.” Capture under load.
2. Look for wide plateaus at the top of the tree (interpreted as you scroll the SVG vertically). Click in to see the full call stack.
3. Form one hypothesis about the mechanism. “This is hot because X is called per-message and X re-parses a format spec.” Don’t make three. One.
4. Fix one thing. Deploy. Capture another flamegraph.
5. Compare the new flamegraph against the old one. Always look at the total sample count, not just the per-frame percentages. Frames that look unchanged in percentage may have shrunk dramatically in absolute work.
6. Repeat until the wide plateaus are gone or are on legitimate work that you can’t reduce further.

The discipline that makes this work is never decide what to fix next without reading the latest flamegraph. Hypotheses last one deploy cycle. Then they’re confirmed by the data or replaced.

Common mistakes

• Reading the y-axis as time. It isn’t. Width is time; height is stack depth.
• Trusting color. Most renderers randomize it. Check your renderer’s docs before reading anything into it.
• Comparing percentages across captures without checking sample counts. A frame at 30% in a 1000-sample capture is far less work than a frame at 30% in a 5000-sample capture, even though they look identical on the SVG.
• Optimizing a tall tower because it looks impressive. Tall isn’t slow. A 10-level deep stack at 1% width is genuinely 1% of your CPU.
• Forgetting to capture under load. A flamegraph from a quiet test doesn’t predict where time goes when the system is busy.
• Trying to micro-optimize before reading the flamegraph. Don’t. The frame you’ll spend two days SIMD-tuning may be the third-widest frame, with the actual top-of-tree being a misplaced HashMap allocation.

What to read next

• Brendan Gregg’s flame graphs page — the canonical reference. Includes off-CPU flamegraphs (for I/O-bound profiling), differential flamegraphs (comparing two captures), and language-specific recipes.
• Speedscope — interactive flamegraph viewer that takes pprof / instruments / chrome devtools profiles. Side-by-side comparisons are much easier than diffing SVGs by eye.
• The Rust performance book — Rust-specific patterns once you’ve got the flamegraph reading down.

A flamegraph won’t tell you why something is slow. It’ll tell you where the CPU went. That’s the half of the problem most teams skip and try to solve by guessing. Don’t guess — read the picture.