Chaos Engineering on a Shoestring Budget: Simulating Packet Loss, Disk Exhaustion, and OOM in Staging

The Enterprise Myth of Chaos Engineering

Chaos engineering—the discipline of experimenting on software systems to build confidence in their capability to withstand turbulent production conditions—is often perceived as the exclusive domain of tech giants with massive tooling budgets. Commercial SaaS platforms charge tens of thousands of dollars annually for automated chaos injectors and failure orchestration dashboards.

However, the Linux operating system kernel already provides battle-tested, kernel-level fault injection mechanisms out of the box. By leveraging native utilities like Traffic Control (tc), fallocate, and stress-ng within Dockerized staging and CI/CD pipelines, engineering teams can simulate catastrophic networking failures, disk saturation, and out-of-memory crashes for zero software cost.

1. Simulating Flaky Networks with Linux Traffic Control (tc)

In distributed microservice architectures, the most destructive failures are not complete server crashes, but partial network degradations: high packet latency, intermittent drops, and packet reordering. These cause connection pool starvation, thread blocking, and cascading retry storms.

Using the Linux kernel's Network Emulation (netem) queueing discipline via tc, you can inject precise network chaos on any network interface or inside a running Docker container:

# Introduce 250ms (+/- 50ms jitter) artificial network latency
tc qdisc add dev eth0 root netem delay 250ms 50ms distribution normal

# Simulate 15% random packet drop on downstream API connections
tc qdisc change dev eth0 root netem loss 15%

# Simulate packet duplication and corruption
tc qdisc change dev eth0 root netem duplicate 5% corrupt 2%

# Reset and remove all artificial network degradation rules
tc qdisc del dev eth0 root

2. Simulating Disk Exhaustion and Write Failures

When an application's root filesystem runs out of disk space (errno 28: ENOSPC), error-handling paths frequently fail because logging frameworks themselves attempt to write error messages to the exhausted disk, entering recursive crash loops.

To safely test how your application handles disk exhaustion in staging, use fallocate to instantly allocate an enormous file that consumes all remaining free blocks on the storage volume:

# Instantly allocate a 20GB dummy file to saturate disk capacity
fallocate -l 20G /var/lib/docker/volumes/test_volume/_disk_saturation.tmp

# Verify disk usage reports 100% full
df -h /var/lib/docker/volumes/test_volume/

# Execute your test suite: verify services gracefully reject writes with HTTP 503

# Cleanup: release disk space immediately
rm -f /var/lib/docker/volumes/test_volume/_disk_saturation.tmp

3. Inducing Controlled Out-of-Memory (OOM) Pressure

When a cloud VPS or Docker container runs low on physical memory, the Linux kernel invokes the OOM Killer (Out-Of-Memory Killer) to sacrifice resident processes and save the kernel from panicking. Without proper process prioritization, the OOM Killer may terminate PostgreSQL or PgBouncer while leaving a rogue worker script running.

You can simulate severe memory exhaustion using stress-ng:

# Spawn 2 memory stress workers allocating 85% of physical RAM for 60 seconds
stress-ng --vm 2 --vm-bytes 85% --vm-hang 5 --timeout 60s

To protect critical database processes from being terminated by the kernel during memory emergencies, adjust their oom_score_adj:

# Ensure PostgreSQL is never killed by OOM (Score range: -1000 to 1000)
# -1000 indicates total immunity from the OOM Killer
echo -1000 > /proc/$(pgrep -u postgres -f "postgres -D")/oom_score_adj

4. Automating Chaos in GitHub Actions & CI Pipelines

Resilience testing yields the greatest returns when executed continuously within automated CI pipelines. By wrapping Linux chaos commands inside ephemeral Docker containers, you can verify that your circuit breakers and retry policies function properly before deploying code to production:

# .github/workflows/chaos-resilience.yml
name: Staging Chaos Resilience Suite
on: [workflow_dispatch]

jobs:
  network-chaos:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Launch Application Containers
        run: docker compose up -d

      - name: Inject 300ms Network Delay into Database Container
        run: |
          docker compose exec --privileged db-service             tc qdisc add dev eth0 root netem delay 300ms

      - name: Execute End-to-End Resilience Suite
        run: |
          # Verify application gracefully enters degraded mode without 500 errors
          curl -f http://localhost:8000/health/ || exit 1

Combine chaos engineering with Linux Kernel TCP/IP Hardening to build truly resilient production systems. Learn more in our DevOps & Infrastructure Services.