Why does memory fragmentation cause high-order page allocation failures even when free memory is abundant?

The Linux kernel's physical page allocator (the buddy allocator) manages memory in powers-of-two page orders (Order-0 = 4 KB, Order-9 = 2 MB huge page, Order-10 = 4 MB). A 2 MB Transparent Huge Page (THP) requires 512 physically contiguous 4 KB pages aligned on a 2 MB boundary. Over time, as processes allocate and free varying amounts of memory, unevictable kernel or slab caches become scattered across physical address space. Even if a system has 20 GB of "free" memory, if no single contiguous block of 512 pages exists without an unmovable page pinned inside it, the buddy allocator fails to satisfy the Order-9 allocation.

How does the Linux kernel memory compaction algorithm work?

Memory compaction uses two independent scanning cursors moving toward each other within a memory zone. The "migrate scanner" starts at the bottom of the zone and moves upward, identifying movable pages (such as anonymous process memory and page cache pages). The "free scanner" starts at the top of the zone and moves downward, searching for empty, unallocated page slots. When both scanners identify suitable candidates, the kernel physically copies the movable pages into the high-memory free slots and updates page table mappings. By packing pages into one end of the zone, contiguous runs of empty pages are created at the other end, yielding Order-9 (2 MB) page blocks.

What is the difference between background compaction (kcompactd) and direct compaction?

Background compaction (kcompactd) runs asynchronously as a kernel daemon when high-order page watermarks fall below thresholds, consolidating memory without stalling user processes. In contrast, "direct compaction" occurs synchronously in the context of the allocating process when an allocation request (such as a 2 MB THP) cannot be satisfied immediately. The user thread is forcibly paused while the kernel synchronously scans page frames, copies memory, and flushes TLBs. In latency-critical databases like Redis, MongoDB, and ScyllaDB, direct compaction causes catastrophic latency spikes ranging from 10 ms to over 500 ms.

Why do production databases recommend setting transparent_hugepage to "madvise" or "never"?

When transparent_hugepage is set to "always", the kernel attempts to allocate a 2 MB page for EVERY anonymous memory allocation. Under memory pressure, this triggers aggressive synchronous direct compaction, producing severe tail latency jitter and thread freezes. By switching to "madvise", the kernel only allocates 2 MB huge pages for memory regions where the application explicitly requested them via madvise(MADV_HUGEPAGE) (ideal for JVM heaps or database buffer pools), leaving general allocations on fast 4 KB pages. Setting it to "never" eliminates THP entirely, guaranteeing deterministic 4 KB allocation performance.

Linux Memory Compaction & Transparent Huge Pages Studio: Buddy Allocator & Fragmentation

Verified Specification | Standardized Formulas | Instant Precision

Secure & Private (Zero Data Retention) Free Access • No Sign-Up

Linux Memory Compaction & Transparent Huge Pages Studio

Architect Linux kernel memory subsystems: inspect buddy allocator order-0 to order-10 fragmentation, simulate memory compaction scanners, and eliminate direct compaction tail-latency stalls.

THP: madvise Stall Risk: Low

48 Blocks

Available Order-9 (2 MB Huge Pages)

34.2%

External Fragmentation Index (Order 9)

0.0 ms

Direct Compaction Thread Stall Duration

20 / 100

vm.compaction_proactiveness Index

1. Kernel THP Mode & Memory Fragmentation Setup

Transparent Huge Pages Mode

External Fragmentation Level

Workload Allocation Profile

2. Buddy Allocator Free Page Pool (/proc/buddyinfo)

Click "Trigger Manual Compaction" to simulate echo 1 > /proc/sys/vm/compact_memory.

3. Production Kernel sysctl & THP Hardening

// Generated sysctl config

Linux Memory Compaction & Transparent Huge Pages Studio

1. Kernel THP Mode & Memory Fragmentation Setup

2. Buddy Allocator Free Page Pool (/proc/buddyinfo)

3. Production Kernel sysctl & THP Hardening

🔍 Companion Workbenches & Verification Tools

Distributed Tracing: W3C Baggage, Span Links & Tail Sampling Studio

JSON Schema Generator & Validator Studio

Unix Timestamp & Epoch Converter

Base64 to Image Decoder & Data URI Viewer

Why Smart Users Are Switching to This Free Toolkit

The Browser Feature Most People Never Use

Stop Paying for Software You Can Get Free

File Conversion Tricks That Save Hours Every Week

This Free Tool Replaced 5 Paid Apps for Our Team

What Your Browser Can Do Without Any Extensions