Redis Memory Fragmentation & jemalloc Tuning: Diagnosing OOM Kills and Calibrating Active Defragmentation in Production

The Phantom OOM: Why Redis Nodes Die with Empty RAM

One of the most perplexing production failures in high-throughput caching tiers is the sudden death of a Redis instance via the Linux kernel's Out-Of-Memory (OOM) killer. When inspecting the Redis metrics right before the termination, developers often see that used_memory (the actual bytes consumed by keys and values) was reporting 12 GB on a 32 GB server. Why did the Linux kernel terminate a process using only 37% of its data allocation?

The culprit is Memory Fragmentation, quantified by the mem_fragmentation_ratio metric in INFO memory:

mem_fragmentation_ratio = used_memory_rss / used_memory

Where used_memory_rss is the Resident Set Size—the actual physical RAM pages the Linux kernel has allocated to the Redis process. When keys are rapidly written, mutated, expired, and evicted with variable payload sizes (such as JSON strings, session hashes, and token arrays), memory allocators cannot return sparsely occupied memory pages back to the OS. When mem_fragmentation_ratio climbs to 2.2+, a 12 GB dataset can consume over 28 GB of physical RAM, driving the host into swap death and triggering an unrecoverable SIGKILL.

Inside jemalloc: Arenas and Size Classes

Redis uses jemalloc as its default memory allocator. jemalloc organizes allocations into discrete size classes (e.g., 8, 16, 32, 48, 64, 80, 96, 112, 128 bytes). When an existing cached string expands by 5 bytes, jemalloc may allocate a new slot in a higher size class and leave the previous slot empty.

If your workload involves keys with constantly fluctuating sizes, physical memory pages become riddled with tiny "holes" of free memory that cannot be merged into contiguous pages. Because the OS kernel only deals in 4 KB memory pages, as long as even a single byte on a 4 KB page is still allocated, the entire page remains pinned in physical RAM.

Calibrating Active Defragmentation in Redis 7+

Starting in Redis 4 and substantially refined in versions 6 and 7, Redis includes an Active Memory Defragmenter. Operating within the main event loop, the defragmenter inspects memory pages, copies allocated keys into contiguous memory blocks, and frees empty pages back to jemalloc and the operating system.

However, running active defragmentation with default settings on high-throughput instances can induce severe CPU latency spikes. Below is our production-tested configuration for high-traffic workloads:

# 1. Enable Active Defragmentation
activedefrag yes

# 2. Minimum fragmentation byte threshold before defrag initiates
# Do not trigger on small 100MB fluctuations; wait until 500MB is wasted
active-defrag-ignore-bytes 500mb

# 3. Minimum fragmentation ratio threshold (1.30 = 30% wasted memory)
active-defrag-threshold-lower 30

# 4. Maximum fragmentation ratio threshold where CPU effort scales up
active-defrag-threshold-upper 60

# 5. Dynamic CPU effort allocation (percentage of Redis event loop)
# Min 5% CPU effort during minor fragmentation, up to max 25% under critical pressure
active-defrag-cycle-min 5
active-defrag-cycle-max 25

# 6. Maximum number of set/hash/list fields to process per scan step
active-defrag-max-scan-fields 1000

Manual jemalloc Purging and Diagnostics

When investigating active fragmentation in production without restarting the instance, you can inspect detailed allocator arenas and command jemalloc to release unused dirty pages directly back to the operating system:

# Connect via redis-cli
redis-cli -p 6379

# Inspect detailed allocator stats
127.0.0.1:6379> MEMORY STATS
127.0.0.1:6379> MEMORY PURGE
OK

The MEMORY PURGE command instructs jemalloc to scan its dirty page lists and trigger madvise(MADV_DONTNEED), instantly returning unused pages to the kernel. Incorporating these controls guarantees predictable memory limits across high-throughput caching layers. For comprehensive architecture reviews, explore our Architecture Review & Code Audit Services.