What is the exact difference between On-CPU profiling and Off-CPU profiling in Linux systems?

On-CPU profiling measures where threads are spending active CPU execution cycles when executing instructions on a physical CPU core. It operates via timer interrupts (e.g. perf_event_open with PERF_COUNT_SW_CPU_CLOCK at 19-99 Hz) that periodically sample the instruction pointer and unwind the current call stack. On-CPU profiling is ideal for detecting algorithmic hot paths, CPU-heavy serialization, regex backtracking, and JIT compilation overhead. In contrast, Off-CPU profiling measures where threads are blocked and waiting while sleeping or descheduled. It operates by intercepting the kernel scheduler tracepoint (sched:sched_switch) to record the exact timestamp when a thread leaves TASK_RUNNING. When the thread is rescheduled, the duration of the sleep is computed and attributed to the call stack captured at deschedule time. Off-CPU profiling reveals lock contention (pthread_mutex_lock, futex), blocking disk I/O, socket read timeouts, and thread pool starvation that cause high tail latency without showing up on CPU utilization metrics.

Why does Frame Pointer unwinding (-fno-omit-frame-pointer) remain the gold standard for eBPF continuous profiling despite its 1-2% CPU overhead?

Frame Pointer unwinding relies on a dedicated CPU register (RBP on x86_64) that maintains a linked list of stack frames in memory: the memory at (RBP) stores the caller previous RBP, and (RBP + 8) stores the return instruction pointer. This allows the in-kernel eBPF helper bpf_get_stackid() to unwind 32 to 128 stack frames in roughly 10 to 30 nanoseconds with zero dynamic memory allocation and zero context switches. While reserving the RBP register reduces the number of general-purpose registers available to the compiler (causing a 1-2% throughput penalty on x86_64), it eliminates the need to transport raw stack memory to userspace. In contrast, DWARF unwinding requires evaluating complex Call Frame Information (CFI) bytecode, which the Linux kernel eBPF verifier cannot safely execute, necessitating either streaming entire 8KB-64KB raw stack memory chunks over ring buffers to userspace (crushing memory bandwidth) or compiling complex compact unwind tables.

How do eBPF compact unwind tables (the Parca / PolarSignals approach) achieve zero-overhead DWARF unwinding in the kernel?

Standard DWARF .eh_frame sections are large and use Turing-complete bytecode to compute stack offsets, making them impossible to evaluate within the strict constraints of the in-kernel eBPF verifier. The compact unwind table approach solves this by pre-processing the ELF binary .eh_frame sections in userspace into an ultra-compact binary format (typically 8 bytes per machine instruction range). This table maps any given instruction pointer (IP) to two numbers: (1) how to calculate the Canonical Frame Address (CFA) relative to the stack pointer or frame pointer, and (2) where the return address is stored relative to CFA. These compact tables are loaded into an eBPF BPF_MAP_TYPE_ARRAY or BPF_MAP_TYPE_HASH. When a timer interrupt fires, the eBPF program performs a binary search over the compact table in kernel space, unwinding stacks for binaries compiled with -fomit-frame-pointer at zero runtime CPU overhead on the application and without copying raw stack memory to userspace.

Why should continuous profiling sampling frequencies strictly avoid round numbers like 100 Hz, 500 Hz, or 1000 Hz?

Sampling profilers must strictly use prime numbers or frequencies that do not resonate with periodic system activities (such as 19 Hz, 47 Hz, or 97 Hz). Round numbers like 100 Hz or 1000 Hz synchronize with Linux kernel timer ticks (CONFIG_HZ = 100, 250, or 1000), scheduled cron jobs, application request poll loops, and periodic GC flushes. When the sampling frequency aligns with an application internal cadence (harmonic resonance / aliasing), the profiler repeatedly samples the same phase of the loop, producing heavily biased flame graphs that either drastically overestimate or completely hide recurring operations. A prime sampling frequency like 19 Hz or 47 Hz ensures a uniform, quasi-random temporal distribution across all application execution phases.

How does in-kernel aggregation via BPF_MAP_TYPE_STACK_TRACE differ from ring buffer event streaming in continuous profiling architecture?

In continuous profiling, millions of stack samples are captured every second across a fleet. In-kernel aggregation uses a specialized BPF map (BPF_MAP_TYPE_STACK_TRACE) paired with a BPF_MAP_TYPE_HASH: the eBPF program hashes the stack frame instruction pointers in the kernel and increments a 64-bit sample counter for that stack ID. Identical stack traces (e.g. 10,000 samples of the same event loop) collapse into a single 8-byte counter update in kernel RAM. Userspace collectors read this aggregated map once every 5 to 10 seconds, reducing CPU overhead to < 0.3% and bandwidth to a few kilobytes per second. Conversely, streaming raw stack frames over BPF_MAP_TYPE_RINGBUF emits an event for every single sample. While raw streaming provides exact timestamps for individual events (useful for distributed trace stitching), it generates hundreds of megabytes of telemetry per minute and can saturate the CPU cache and network egress.

What is the symbolization pipeline and why are debug symbols (.symtab) and DWARF info separated from production binaries?

The in-kernel eBPF profiler only records raw 64-bit instruction pointers (memory addresses such as 0x7fff8921a4bc). To produce human-readable flame graphs with function names and line numbers, these addresses must undergo symbolization: (1) The profiler reads /proc/ /maps to locate the virtual memory base address and offset of each loaded ELF executable and shared library (.so); (2) It calculates the relative file offset: File_Offset = Address - VMA_Base + ELF_Offset; (3) It queries the binary symbol table (.symtab) or dynamic symbol table (.dynsym). In modern cloud environments, debug symbols are stripped from production container images to keep sizes under 50MB and prevent reverse engineering. The profiler agent extracts the binary GNU Build ID (from .note.gnu.build-id) and fetches matching unstripped symbols from a central symbol server (debuginfod) or storage bucket to symbolize stacks asynchronously.

Verified Specification | Standardized Formulas | Instant Precision

Secure & Private (Zero Data Retention) Free Access • No Sign-Up

Linux eBPF Profiler On-CPU & Off-CPU DWARF & Frame Pointers Zero Overhead Target

Linux Continuous Profiling, Stack Unwinding & FlameGraph Studio

Architect enterprise-scale eBPF continuous profiling pipelines: compare in-kernel Frame Pointer vs compact DWARF vs ORC stack unwinding, calculate profiler CPU tax and map memory budgets, evaluate On-CPU cycles vs Off-CPU blocking stalls, explore interactive SVG FlameGraphs, and synthesize production C/libbpf and Go collectors.

1,520

Samples / Sec (Host)

0.28%

Profiler CPU Tax

4.2 MB

Kernel Map Memory

1.4 MB/s

Fleet Ingest Egress

Interactive SVG FlameGraph Explorer

Click any frame to zoom in; click Reset to restore root view. Hover to inspect sample count and %.

Matched: 0 frames (0.0%)

Palette:

Hover over any stack frame to inspect... -

Total Profile Weight: 2,450 samples FlameGraph Width = Relative Time Spent

How to Read a Production FlameGraph

▪ X-Axis (Width): Represents total sample count or time spent in that function and its descendants. It does NOT represent chronological execution time. A wider frame means more CPU cycles or blocking duration occurred within that call path.

▪ Y-Axis (Depth): Shows the call stack depth from root (bottom or top depending on inversion) to leaf. The top-most frames with wide plateaus are the actual execution bottlenecks (the "top of the stack").

▪ Plateau Hunting: Look for wide frames that have NO children or narrow children. These flat tops are where the CPU is actually executing instructions directly rather than delegating to child functions.

▪ Icicle vs FlameGraph: Classic FlameGraphs place root at the bottom, growing upward. Icicle mode inverts the graph so root is at the top, mimicking modern browser DevTools and IDE call trees.

Stack Unwinding Engine Comparison

Continuous profilers rely on low-level CPU registers and ELF binary sections to reconstruct the sequence of calling functions. Review the exact architectural trade-offs between unwinding mechanisms:

Unwinding Mechanism	Target Domain	eBPF Kernel Compatibility	Application CPU Tax	Symbol Quality & Limitations
Frame Pointer (RBP) `-fno-omit-frame-pointer`	Native C/C++, Go, Rust, JVM	Native bpf_get_stackid()	1.0% - 2.0% throughput penalty (loss of 1 register on x86)	Instant (~15ns). Breaks if any shared library (glibc) was compiled with frame pointer omission.
Compact DWARF Tables `Parca / PolarSignals BPF`	Stripped native binaries, Go, Rust	Kernel BPF Map Array Search	0.0% runtime penalty on target application	Evaluates CFA and return addresses via compact 8-byte eBPF tables. Requires ~10-20MB RAM per binary for tables.
ORC (Oops Rewind Capability) `CONFIG_UNWINDER_ORC`	Linux Kernel (`vmlinux`)	Built into Linux 4.14+	0.0% penalty	Deterministic, linear lookup. Completely immune to compiler reordering optimizations in kernel space.
JIT / Managed Maps `/tmp/perf-<pid>.map`	Node.js (V8), Java (JVM), Erlang	Userspace Symbol Bridge	0.5% - 1.5% (symbol map dumping)	Bridges dynamically emitted JIT memory blocks to method names. Requires Async-Profiler or V8 --perf-prof flag.

Fedora / Ubuntu Frame Pointer Revolution: Both Fedora (since release 38) and Ubuntu (since release 24.04 LTS) have officially changed their default distribution compiler flags to include -fno-omit-frame-pointer across all packages. This standardized choice unlocks zero-cost continuous profiling across entire cloud fleets without sacrificing debuggability.

The Anatomy of a Stack Frame on x86_64

+------------------------------------+ <-- High Memory Address | Caller Stack Frame Data | +------------------------------------+ | Return Instruction Pointer (RIP) | [RBP + 8] <-- Next instruction in caller +------------------------------------+ | Saved Frame Pointer (Previous RBP) | [RBP] <-- Points to caller's RBP +------------------------------------+ <-- Current RBP register value | Local Function Variables | | ... | +------------------------------------+ <-- Stack Pointer (RSP)

When an eBPF timer fires, the kernel reads the current RBP register, dereferences it to find the previous RBP, and reads (RBP + 8) to record the return address. Repeating this loop 32 times takes less than 300 CPU cycles—orders of magnitude faster than parsing DWARF bytecode.

Continuous Profiling Sizing & Overhead Calculator

Model fleet-wide resource utilization, eBPF map memory allocation, profiler CPU tax, and telemetry ingestion network bandwidth:

Fleet Size (Servers): 100

CPU Cores per Server: 16

Sampling Frequency: 19 Hz (Prime)

Max Stack Depth (Frames): 64 frames

Aggregation Strategy

Unwinding Strategy

Telemetry Ingestion Protocol

Sizing Architecture Calculation Results

Host Sample Rate

304 samples/s

Fleet-Wide Throughput

30,400 samples/s

Raw Telemetry Rate

15.2 MB/s

Zstd Ingest Bandwidth

450 KB/s

Kernel BPF Map RAM

3.2 MB / host

Estimated Host CPU Tax

0.18%

On-CPU vs Off-CPU Profiling Mechanics

A complete latency investigation requires measuring both where threads are executing and where threads are waiting:

On-CPU Profiling (Burned Cycles)

Active Execution

Trigger Mechanism: Timer interrupt via perf_event_open with PERF_COUNT_SW_CPU_CLOCK.
Thread State: TASK_RUNNING actively executing on a core.
What It Identifies:
- Excessive JSON serialization / deserialization
- Regex catastrophic backtracking
- Hot loop arithmetic & memory hashing
- JVM Garbage Collection scavenge phases
When To Use: System CPU utilization is high (70%-100%) and you need to optimize CPU capacity or reduce server count.

Off-CPU Profiling (Blocked Stalls)

Waiting / Sleeping

Trigger Mechanism: Kernel scheduler tracepoint sched:sched_switch.
Thread State: TASK_INTERRUPTIBLE or TASK_UNINTERRUPTIBLE.
What It Identifies:
- Lock contention (pthread_mutex_lock, futex)
- Synchronous storage writes (fsync, WAL commits)
- Database connection pool exhaustion
- Network socket reads & external API timeouts
When To Use: P99 latency is spiking, users experience sluggishness, but server CPU utilization remains deceptively low (< 20%).

Kernel Scheduler State Machine Flow (Off-CPU Duration Tracking)

[ Thread A: Running ] ---( Calls mutex_lock / read / nanosleep )---> | v [ sched:sched_switch ] Fires in Kernel: 1. eBPF checks: prev_state != 0 (Thread is going to sleep) 2. bpf_get_stackid(&stack_map, ctx, BPF_F_USER_STACK) captures Call Stack 3. bpf_ktime_get_ns() records start_time in BPF Hash Map keyed by TID | |--- ( Thread A is sleeping / blocked on disk / waiting on lock ) --- | v [ sched:sched_switch ] Fires when Thread A is rescheduled back in: 1. eBPF looks up TID in start_time map 2. delta_ns = current_ktime - start_time 3. Accumulates delta_ns into stack trace duration counter map 4. Deletes TID from in-flight start_time map

Production eBPF Profiler Source Code

// Select an artifact above

Linux Continuous Profiling, Stack Unwinding & FlameGraph Studio

Interactive SVG FlameGraph Explorer

How to Read a Production FlameGraph

Stack Unwinding Engine Comparison

The Anatomy of a Stack Frame on x86_64

Continuous Profiling Sizing & Overhead Calculator

Sizing Architecture Calculation Results

On-CPU vs Off-CPU Profiling Mechanics

On-CPU Profiling (Burned Cycles)

Off-CPU Profiling (Blocked Stalls)

Kernel Scheduler State Machine Flow (Off-CPU Duration Tracking)

Production eBPF Profiler Source Code

🔍 Companion Workbenches & Verification Tools

CSV to JSON Converter & Delimited Table Parser

Linux SLUB Allocator, Slab Caches & Object Caging Architecture Studio

Linux EEVDF Kernel Scheduler & Virtual Deadline Studio

Linux Futex2, futex_waitv & Kernel Wait Queue Studio

Why Smart Users Are Switching to This Free Toolkit

The Browser Feature Most People Never Use

Stop Paying for Software You Can Get Free

File Conversion Tricks That Save Hours Every Week

This Free Tool Replaced 5 Paid Apps for Our Team

What Your Browser Can Do Without Any Extensions