LSM-Tree Compaction Tuning in High-Velocity Write Engines: Leveled vs. Size-Tiered Sizing

The Rise of Log-Structured Merge (LSM) Trees

Traditional relational database storage engines (such as MySQL InnoDB or SQLite) rely on B-Tree data structures. In a B-Tree, updates and inserts overwrite physical disk pages in-place. While B-Trees provide stellar read performance (O(log N) point lookups), high-velocity random write workloads induce catastrophic random I/O thrashing and page fragmentation.

To service high-velocity write streams (such as time-series metrics, telemetry logs, financial trade books, and distributed metadata), modern data engines (RocksDB, Cassandra, ScyllaDB, LevelDB, and ClickHouse) employ Log-Structured Merge (LSM) Trees. LSM engines convert costly random writes into high-speed sequential disk I/O.

Anatomy of an LSM Storage Engine

An LSM tree organizes data across memory and disk tiers:

1. MemTable (RAM): Incoming writes are appended to an in-memory skip-list or lock-free concurrent vector. Concurrently, the write is appended to an on-disk Write-Ahead Log (WAL) for durability.
2. Immutable MemTable: When the MemTable reaches capacity (e.g., 64MB), it freezes and becomes read-only while a fresh MemTable takes over active writes.
3. Flush to Level 0 (L0 SSTables): A background thread flushes the immutable MemTable to disk as a sorted, immutable SSTable (Sorted String Table) in Level 0.
4. Background Compaction: Because keys overlap across different L0 SSTables, background compaction merges, deduplicates, and sorts SSTables into deeper levels (L1, L2, ... Ln).

The Fundamental Trade-off: RUM Conjecture in LSM Trees

Storage engine engineering is governed by the RUM Conjecture: you can optimize for two of the following dimensions, but never all three simultaneously:

• Write Amplification (WA): The ratio of total bytes written to physical storage compared to the logical bytes submitted by the application. High WA wears out NVMe SSD flash memory rapidly and starves disk bandwidth.
• Read Amplification (RA): The number of physical disk lookups required to resolve a single logical read query. High RA degrades point lookup latency.
• Space Amplification (SA): The ratio of physical disk space consumed compared to actual live data size. High SA results from deleted or superseded versions of keys lingering on disk.

Compaction Strategies: Leveled (LCS) vs. Size-Tiered (STCS)

The choice of compaction algorithm dictates your engine's operational profile:

1. Leveled Compaction Strategy (LCS - RocksDB Default)

In Leveled Compaction, each level has a strictly defined maximum capacity (e.g., L1 = 256MB, L2 = 2.5GB, L3 = 25GB, scaling by a 10x multiplier). Crucially, keys within any level above L0 never overlap. When L1 exceeds capacity, a background thread merges one L1 SSTable with all overlapping SSTables in L2.

• Pros: Minimal Read Amplification (point reads only check one SSTable per level); extremely low Space Amplification (typically 1.1x to 1.2x).
• Cons: Severe Write Amplification (WA typically ranges from 15x to 30x), leading to write stalls under heavy insert volume.

2. Size-Tiered Compaction Strategy (STCS - Cassandra / ScyllaDB Default)

In Size-Tiered Compaction, SSTables of similar sizes are grouped together into tiers. Compaction triggers only when a fixed number of similar-sized tables accumulate (e.g., 4 tables), sorting and merging them into a single larger SSTable.

• Pros: Ultra-low Write Amplification (WA typically 4x to 8x); supports blistering write throughput with minimal disk write overhead.
• Cons: Massive Space Amplification (can require up to 50% temporary disk headroom during major compactions); high Read Amplification (point reads must search across multiple SSTables).

Tuning RocksDB for High-Throughput Ingestion

Below is a production-hardened C++ / Python RocksDB configuration calibrated for write-intensive telemetry workloads (50,000+ writes/sec on NVMe storage):

# RocksDB Production Options File
[DBOptions]
  max_background_jobs=8
  bytes_per_sync=1048576             # 1MB direct I/O sync smoothing
  delayed_write_rate=33554432        # 32MB/s smooth rate limit during stalls

[CFOptions "default"]
  # MemTable Sizing (Avoid write stalls)
  write_buffer_size=134217728        # 128MB MemTable
  max_write_buffer_number=6          # Up to 6 MemTables in RAM
  min_write_buffer_number_to_merge=2 # Merge 2 MemTables before flushing

  # Level Sizing & Multipliers
  level0_file_num_compaction_trigger=4
  level0_slowdown_writes_trigger=20
  level0_stop_writes_trigger=36
  target_file_size_base=67108864     # 64MB SSTable size
  max_bytes_for_level_base=536870912 # 512MB Level 1
  max_bytes_for_level_multiplier=10  # 10x level expansion

  # Compaction Style
  compaction_style=kCompactionStyleLevel
  compaction_pri=kMinOverlappingRatio # Target SSTables with minimal overlap first

  # Block-Based Table Options (Bloom Filters for Read Speed)
  table_factory=BlockBasedTable
  block_based_table_factory={
    block_size=16384;                # 16KB blocks
    cache_index_and_filter_blocks=true;
    filter_policy=bloomfilter:10:false; # 10 bits/key Bloom Filter (1% false positive)
  }

Operational Benchmarks: Eliminating Write Stalls

Across an ingestion benchmark streaming 100,000 key-value updates/second (128-byte keys, 512-byte payloads) on an NVMe enterprise SSD:

• Default Settings: Experienced 18 write stalls per hour where application write latency spiked from 0.4ms to over 850ms due to L0 file backlog; NVMe Write Amplification: 28.4x.
• Tuned Level Compaction + Dynamic Sizing: Zero write stalls; Write latency p99 remained stable at 1.8ms; NVMe Write Amplification dropped to 14.1x; SSD write life extended by 2x.

For organizations operating high-velocity data platforms, our Database Design & Performance Services provide end-to-end tuning across LSM engines, PostgreSQL, and distributed storage tiers.