web / bot prevention

updated October 21, 2026

Automated bots and scanners probe public web servers for sensitive paths: .env, .git/config, database credentials, and admin consoles:

GET /.env
GET /docker-compose.yml
GET /proc/self/environ
GET /.git/config

Each probe returns a 404 response. Those requests consume worker processes, database connections, and logs.

Why default CDN rules allow path scans

Cloudflare Managed Rulesets target exploit payloads such as SQL injection. They permit plain GET requests for paths such as /.env to reach the origin server. The origin server must evaluate routing and respond with a 404.

Targeted path filtering

I block path scans at the edge with custom WAF rules:

(http.request.uri.path contains "/." and not http.request.uri.path contains "/.well-known/") or
http.request.uri.path matches "(?i)\.(env|yml|yaml|tfstate|sql|pem|key|bak|conf)$" or
http.request.uri.path contains "proc/self"

I exclude /.well-known/ paths so OAuth discovery and RFC metadata resolve.

WAF attack score mitigation

Cloudflare evaluates incoming requests with machine-learning models. The model assigns an attack score from 1 to 99, where lower scores indicate attacks.

I block requests with an attack score of 20 or lower:

(cf.waf.score ge 1 and cf.waf.score le 20)

This rule drops malicious requests at the edge before they touch the origin server.

Response-based rate limiting

Scanners send many invalid requests in seconds. I configure a rate limit rule that tracks origin HTTP 404 responses:

http.response.code eq 404

When an IP address triggers 15 404 responses in one minute, Cloudflare blocks that IP address for 10 minutes. Normal requests receive 200 responses and bypass the limit.

Protecting non-browser clients

Standard bot defenses challenge automated clients with JavaScript. My servers receive non-browser traffic from CLI tools, Git over HTTPS, and webhooks. These clients cannot solve interactive challenges.

I skip bot challenges for /api/ and /git/ paths:

http.request.uri.path startswith "/api/" or
http.request.uri.path startswith "/git/"

This allows automated tools and Git pushes to bypass managed challenges.

← All articles