web / bot prevention

updated October 29, 2026

Automated bots probe web servers for sensitive paths:

GET /.env
GET /docker-compose.yml
GET /proc/self/environ
GET /.git/config

Those probes return 404 responses that consume origin worker processes and connection slots.

Edge path filtering

Cloudflare Managed Rulesets target exploit payloads like SQL injection, but permit path probes to reach the origin.

I block path scans at the edge with custom WAF rules:

(http.request.uri.path contains "/." and not http.request.uri.path contains "/.well-known/") or
http.request.uri.path matches "(?i)\.(env|yml|yaml|tfstate|sql|pem|key|bak|conf)$" or
http.request.uri.path contains "proc/self"

I exclude /.well-known/ paths so OAuth discovery and RFC metadata resolve.

WAF attack score mitigation

Cloudflare assigns an attack score from 1 to 99, where lower scores indicate attacks. I block requests with an attack score of 20 or lower:

(cf.waf.score ge 1 and cf.waf.score le 20)

The rule leaves out my CI server's host. A failed check's output holds HTML, SQL, and request log lines, and the attack score reads that as an attack. The rule would refuse the runner's result with a 403 that the origin never sees.

Response-based rate limiting

Scanners send many invalid requests in seconds. I configure a rate limit rule that tracks origin HTTP 404 responses:

http.response.code eq 404

When an IP address triggers 15 404 responses in one minute, Cloudflare blocks that IP address for 10 minutes. Normal requests receive 200 responses and bypass the limit.

Non-browser traffic

CLI tools, Git over HTTPS, and webhooks cannot solve JavaScript bot challenges. I skip bot challenges for /api/ and /git/ paths:

http.request.uri.path startswith "/api/" or
http.request.uri.path startswith "/git/"

Origin limits

My CI server also limits the routes that anyone can call: a login challenge and an error post. One limiter holds a token bucket for each client address or key, built on golang.org/x/time/rate:

func (l *keyLimiter) take(key int64, n int, now time.Time) bool {
	l.mu.Lock()
	defer l.mu.Unlock()
	if l.m == nil {
		l.m = map[int64]*rate.Limiter{}
	}
	// A bucket that has refilled is full, so it holds no state. Sweep at
	// most once a second so the scan is not paid per request.
	if len(l.m) >= keyLimiterMaxKeys && now.Sub(l.swept) >= time.Second {
		l.swept = now
		for k, v := range l.m {
			if v.TokensAt(now) >= float64(l.burst) {
				delete(l.m, k)
			}
		}
	}
	if l.m[key] == nil && len(l.m) >= keyLimiterMaxKeys {
		key = keyLimiterOverflowKey
	}
	b := l.m[key]
	if b == nil {
		b = rate.NewLimiter(l.rate, l.burst)
		l.m[key] = b
	}
	return b.AllowN(now, n)
}

The map holds at most 4,096 keys. Past that bound, a key not yet seen shares one overflow bucket, so a flood of new addresses cannot grow the map. The limiter keeps its state in memory, not in Postgres, so it spends no database work to protect the database. The limit is per process and not exact: two processes allow twice the rate, and a restart starts each bucket full.

← All articles