eBPF-Powered Kernel Observability: Profiling Socket Drops, TCP Retransmits, and TLS Handshake Latency in Linux

The APM Blindspot: When Application Logs Lie

Modern microservice observability stacks rely heavily on application-level Application Performance Monitoring (APM) agents (OpenTelemetry, Datadog, New Relic) and reverse proxy access logs (Nginx, Envoy). When an edge proxy occasionally records an HTTP 502 Bad Gateway or a 504 Gateway Timeout connecting to an upstream backend, developers instinctively check the application logs.

All too often, the application logs reveal nothing: no errors, no exceptions, and average request handler runtimes well under 20ms. The failure is happening in the dark space between user space and the physical wire: inside the Linux kernel networking subsystem. Issues like SYN backlog overflows, listen queue drops, TCP window stalls, and ephemeral port exhaustion drop connections before the application runtime's accept() system call ever registers their existence.

eBPF: Safe, In-Kernel Programmability

Extended Berkeley Packet Filter (eBPF) revolutionizes Linux system observability. Rather than modifying application source code, injecting heavy tracing middleware, or capturing gigabytes of raw PCAP dumps with tcpdump, eBPF allows engineers to attach sandboxed, JIT-compiled bytecode programs directly to kernel tracepoints and kprobes.

The Linux kernel's BPF verifier guarantees that eBPF programs cannot crash the operating system, enter infinite loops, or violate memory bounds. By hooking into kernel socket functions, eBPF captures kernel events with sub-microsecond overhead.

Tracing Silent TCP Drops with bpftrace

When an upstream service experiences sudden traffic spikes, the TCP listen backlog can overflow, causing the kernel to silently discard incoming SYN packets. Below is an efficient bpftrace one-liner to detect exactly which process and port is dropping connections in real time:

# Monitor kernel function kfree_skb (where packets are dropped)
sudo bpftrace -e '
kprobe:tcp_drop {
    $sk = (struct sock *)arg0;
    $daddr = ntop($sk->__sk_common.skc_daddr);
    $saddr = ntop($sk->__sk_common.skc_rcv_saddr);
    $dport = $sk->__sk_common.skc_dport;
    printf("TCP DROP: %s -> %s (dport: %d) | Comm: %s
", 
           $saddr, $daddr, bswap($dport), comm);
}'

Tracing TLS Handshake Latency in Production

Another frequent cause of intermittent latency is cryptographic negotiation stalls during upstream TLS handshakes. By attaching an uprobe to the OpenSSL/BoringSSL shared library (libssl.so), eBPF can measure the precise duration of every SSL handshake across the entire node without touching application code:

# ebpf_ssl_tracer.py using python-bcc
from bcc import BPF

bpf_source = """
#include 

BPF_HASH(start_time, u32, u64);

int probe_SSL_connect_enter(struct pt_regs *ctx) {
    u32 pid = bpf_get_current_pid_tgid();
    u64 ts = bpf_ktime_get_ns();
    start_time.update(&pid, &ts);
    return 0;
}

int probe_SSL_connect_exit(struct pt_regs *ctx) {
    u32 pid = bpf_get_current_pid_tgid();
    u64 *tsp = start_time.lookup(&pid);
    if (tsp != 0) {
        u64 delta_us = (bpf_ktime_get_ns() - *tsp) / 1000;
        if (delta_us > 50000) { // Log handshakes exceeding 50ms
            bpf_trace_printk("Slow TLS Handshake: %llu us on PID %d\n", delta_us, pid);
        }
        start_time.delete(&pid);
    }
    return 0;
}
"""

b = BPF(text=bpf_source)
b.attach_uprobe(name="ssl", sym="SSL_connect", fn_name="probe_SSL_connect_enter")
b.attach_uretprobe(name="ssl", sym="SSL_connect", fn_name="probe_SSL_connect_exit")
print("eBPF TLS Handshake Monitor active. Press Ctrl+C to stop.")
b.trace_print()

Kernel Tuning Based on eBPF Insights

When eBPF traces confirm listen queue overflows during traffic surges, tune the corresponding Linux kernel networking parameters in /etc/sysctl.conf:

# Increase maximum socket listen backlog (default is often 128 or 4096)
net.core.somaxconn = 65535

# Increase maximum queued SYN requests for half-open connections
net.ipv4.tcp_max_syn_backlog = 3240000

# Enable TCP SYN cookies to survive sudden burst floods
net.ipv4.tcp_syncookies = 1

For engineering teams designing zero-downtime microservice clusters, explore our Cloud Deployment & DevOps Architecture Services.