Defending Against Low-and-Slow Rotating Proxy Attacks: Honeypot Interceptors, 24-Hour Sliding Windows, and Crawler Whitelisting

The Evolution of Web Probes: From Volumetric Bursts to Distributed Evasion

Modern web security models were historically designed to counter high-velocity volumetric attacks: SYN floods, application-layer HTTP floods, and rapid-fire credential stuffing attempting dozens of requests per second. Traditional defenses—such as Web Application Firewall (WAF) rate limits and Fail2ban jails configured with standard 10-minute time windows (findtime = 600)—excel at dropping bursty traffic. However, automated threat actors have evolved.

Sophisticated adversaries deploying web scraping bots, form injection tools, and API fuzzers now operate via low-and-slow rotating proxy networks. By routing requests through residential proxy services, commercial VPN egress pools, or consumer proxy tunnels (such as Cloudflare WARP egress ranges), an attacker distributes a probe campaign across hundreds of distinct IP addresses. Crucially, they introduce deliberate pacing: each individual IP submits only one request every 15 to 45 minutes.

Under a classic 10-minute Fail2ban configuration with maxretry = 3, an IP that fails authentication or reCAPTCHA validation resets its internal failure counter before the next request arrives. The attacker can sustain automated attacks indefinitely while remaining entirely invisible to short-horizon rate limiters.

1. Architectural Danger: Over-Aggressive Honeypots and Crawler Collateral Damage

To detect reconnaissance probes before they reach backend application servers, engineers frequently deploy honeypot interceptors inside Nginx. A honeypot matches common exploit paths (such as .env, .git, /wp-admin, .yaml, or .json) and immediately terminates the connection with an HTTP 444 response code while writing to an isolated security log:

# Dangerous naive honeypot rule:
location ~* \.json(/|$|\?) {
    access_log /var/log/nginx/vulnerability_probes.log combined;
    return 444;
}

While effective against scripts hunting for /config.json or /swagger.json, this blanket regex rule introduces a critical vulnerability: search engine and crawler false positives. Legitimate crawlers, including Googlebot, Bingbot, and Applebot, routinely fetch standard RFC-compliant metadata endpoints:

• /.well-known/assetlinks.json: Digital Asset Links verification for Android deep links and Google Association Services.
• /.well-known/apple-app-site-association: Apple Universal Link validation.
• /.well-known/traffic-advice: Google Chrome prefetching directives.
• /.well-known/security.txt: RFC 9116 security disclosure guidelines.
• /manifest.json: Web application manifest files for mobile and desktop browsers.

If an over-aggressive honeypot returns HTTP 444 on /.well-known/assetlinks.json and Fail2ban bans the requesting IP with maxretry = 1, genuine Googlebot crawler nodes get banned from the entire server, causing catastrophic drop-offs in search indexation and SEO visibility.

2. Nginx Prefix Termination (^~) for RFC Metadata Whitelisting

Nginx evaluates location blocks using strict precedence rules. While standard prefix locations (location /path/) are superseded by regular expression locations (location ~* regex), Nginx provides the prefix termination modifier (^~). When a request matches a location ^~ block, Nginx immediately halts the search for matching regular expressions.

By declaring an explicit prefix termination block for /.well-known/ prior to including the honeypot rules, any standard metadata request completely bypasses the honeypot regex evaluation:

# Main HTTPS Server Block
server {
    listen 443 ssl http2;
    server_name example.com;

    # 1. Legitimate RFC/Well-Known Metadata (AssetLinks, Security.txt, ACME)
    # '^~' guarantees regex locations in honeypot.conf are NEVER evaluated
    location ^~ /.well-known/ {
        root /var/www/certbot;
        try_files $uri =404;
        access_log off;
    }

    # 2. Web App Manifest exact match
    location = /manifest.json {
        alias /var/www/app/staticfiles/site.webmanifest;
        access_log off;
    }

    # 3. Comprehensive Zero-Tolerance Honeypot
    include /etc/nginx/snippets/honeypot.conf;

    # Application Proxy...
}

3. Implementing 24-Hour Sliding Aggregation in Fail2ban

To eliminate the low-and-slow proxy evasion loophole on sensitive endpoints (such as contact forms, legal aid intake, or authentication APIs), Fail2ban must expand its detection window. For form submissions where users are verified via invisible reCAPTCHA v3 or Turnstile, a legitimate human almost never fails verification twice in a day. An automated bot, however, will fail consistently.

Configure a dedicated jail with a 24-hour sliding aggregation window (findtime = 86400) and a strict threshold (maxretry = 2):

# /etc/fail2ban/jail.d/nginx-recaptcha-abuse.local
[nginx-recaptcha-abuse]
enabled  = true
port     = http,https
filter   = nginx-recaptcha-abuse
logpath  = /var/log/nginx/access.log
action   = %(banaction)s[port="%(port)s"]
           nginx-blocklist
maxretry = 2
findtime = 86400
bantime  = 604800
ignoreip = 127.0.0.1/8 ::1 66.249.64.0/19 142.250.0.0/15

4. Dedicated Edge Rate Limiting with Nginx

In addition to log-parsing intrusion prevention, Nginx should enforce strict request rate constraints directly at the network boundary. General web pages might allow 15 requests per second, but API form submission endpoints should be throttled to human-scale interaction rates (e.g. 6 requests per minute):

# In http context:
limit_req_zone $binary_remote_addr zone=api_form_rate:10m rate=6r/m;

# In server context:
location ~ ^/api/(pro-bono|contact|booking)/ {
    limit_req zone=api_form_rate burst=3 nodelay;
    limit_req_status 429;

    proxy_pass http://upstream_app;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
}

When an automated bot attempts a rapid burst, Nginx responds instantly with 429 Too Many Requests at the socket level, preventing CPU starvation in Gunicorn/Django and conserving paid third-party verification API quotas.