Multithreading vs. Multiprocessing in Production Python: Memory Footprints, Copy-on-Write Thrashing, and CPU-Bound Sizing

The Concurrency Divide in CPython

Architecting scalable Python backends requires understanding the physical boundaries between operating system threads and operating system processes. While CPython provides both threading and multiprocessing abstractions, their execution models interact drastically differently with the CPU scheduler, memory allocator, and CPython's Global Interpreter Lock (GIL).

Threads share the exact same virtual memory space within a single process. However, because CPython's internal memory management is not thread-safe, the GIL mandates that only one native OS thread can execute Python bytecode at any given moment. Consequently, multithreading in Python provides zero concurrency for CPU-bound computations (such as cryptography, image resizing, or tabular analytics). Conversely, multiprocessing sidesteps the GIL entirely by spawning discrete OS processes with isolated Python runtimes, achieving true multi-core parallel execution at the expense of substantial memory overhead and Inter-Process Communication (IPC) serialization penalties.

The Copy-on-Write (CoW) Fallacy in Preforked Workers

In standard production deployments (e.g., Gunicorn prefork workers or Celery task executors), the master process loads the application code, Django models, ORM metadata, and configuration into memory before calling os.fork() to spawn child worker processes. Under Unix memory management, child processes inherit the memory pages of the parent via Copy-on-Write (CoW): physical RAM is shared between processes until one process writes to a page, at which point the Linux kernel allocates a private physical copy of that 4KB page for the writing process.

In theory, preforking should allow 10 worker processes to share 90% of an application's 400MB base memory footprint. In CPython, however, Copy-on-Write is rapidly destroyed by normal execution. The root culprit is reference counting and generational garbage collection:

1. Reference Counting Mutation: Every time Python reads an immutable object (e.g., inspecting a loaded Django model class or accessing a cached dictionary key), CPython increments that object's ob_refcnt field in memory. Modifying an integer in the object header dirties the underlying 4KB virtual memory page, forcing the Linux kernel to duplicate the page immediately.
2. Generational GC Pointer Relinking: CPython's generational garbage collector maintains doubly-linked lists tracking container objects (tuples, lists, dicts). During every GC collection sweep, the pointers in object headers are mutated, dirtied, and copied across all worker processes.

Within minutes of boot under production traffic, a worker pool that initially consumed 450MB of physical RAM balloons to over 3.8GB of Private Resident Set Size (RSS), triggering Linux kernel out-of-memory (OOM) kills.

Measuring and Preserving CoW with gc.freeze()

To prevent CoW degradation, modern CPython runtimes provide gc.freeze(). By invoking gc.freeze() in the master process immediately before forking, all currently loaded objects are moved into a permanent, frozen space that the garbage collector completely ignores during subsequent sweeps.

Here is an empirical Python script to measure private vs. shared memory dirty pages using /proc/self/smaps_rollup:

import os
import gc
import multiprocessing as mp

def read_memory_stats(label: str):
    '''Parses Linux smaps_rollup to extract exact physical memory distribution.'''
    with open("/proc/self/smaps_rollup", "r") as f:
        lines = f.readlines()
    
    stats = {}
    for line in lines:
        parts = line.split()
        if len(parts) >= 2:
            key = parts[0].rstrip(":")
            stats[key] = int(parts[1])
            
    print(f"[{label}] PID {os.getpid()} -> "
          f"PSS: {stats.get('Pss', 0)/1024:.1f}MB | "
          f"Shared Dirty: {stats.get('Shared_Dirty', 0)/1024:.1f}MB | "
          f"Private Dirty: {stats.get('Private_Dirty', 0)/1024:.1f}MB")

def worker_task(barrier):
    # Wait for all workers to align
    barrier.wait()
    # Read stats right after fork
    read_memory_stats("Child Immediately After Fork")
    
    # Simulate read-heavy web traffic inspecting shared objects
    import django
    # Inspecting loaded classes causes refcount mutations unless frozen
    read_memory_stats("Child After Read-Heavy Operations")

if __name__ == "__main__":
    read_memory_stats("Parent Before Preloading")
    # Simulate heavy Django application load
    dummy_cache = [f"preloaded_data_string_{i}" for i in range(1_500_000)]
    
    # CRITICAL: Freeze objects into immortal memory before fork
    gc.collect()
    gc.freeze()
    read_memory_stats("Parent After gc.freeze()")
    
    barrier = mp.Barrier(2)
    p = mp.Process(target=worker_task, args=(barrier,))
    p.start()
    p.join()

Decision Matrix: ThreadPool vs. ProcessPool Sizing

When engineering high-throughput microservices, choosing between multithreading and multiprocessing hinges on the bottleneck profile of the workload:

For systems handling intensive real-time workflows and distributed job scheduling, exploring our Distributed Task Orchestrator Architecture details how we calibrate worker pools and memory bounds across multi-node Linux clusters.