The logs were a mess. While no single IP was doing anything egregious (most were just doing 1-3 requests per minute), the sheer volume was overwhelming. We were seeing hundreds of thousands of hits a day, all from the same broad Meta range. A quick look at the User-Agent showed meta-externalagent/1.1 everywhere.

The problem with traditional per-IP rate limiting is that these crawlers are distributed. If you set a limit of, say, 10 requests per second per IP, a botnet of 1,000 IPs can still hit you with 10,000 requests per second while each individual IP stays perfectly within the limits.

I needed a way to rate-limit the entire Meta block as a single entity.

The first step was figuring out exactly which IP ranges belonged to Meta. I took one of the offending IPs from the logs (e.g., 57.141.20.57) and ran it through a WHOIS lookup. It pointed me to the RIPE FB-BLOCK range, which covers a few large CIDR blocks: 57.141.0.0/16, 57.142.0.0/15, 57.144.0.0/14, and 57.148.0.0/15.

To implement this in Nginx, I used the geo module to tag these IPs and then the map module to create a shared key for them.

geo $git_is_meta {
    default 0;
    57.141.0.0/16 1;
    57.142.0.0/15 1;
    57.144.0.0/14 1;
    57.148.0.0/15 1;
}

# If the client is Meta, they all share this single string as their key
map $git_is_meta $git_meta_key {
    1 "meta-crawler";
    0 ""; # Empty key means the limit doesn't apply to them
}

# For everyone else, use their actual IP address as the key
map $git_is_meta $git_ip_key {
    0 $binary_remote_addr;
    1 ""; # Empty key means the limit doesn't apply to them
}

# Define the shared bucket for Meta (1 request per second, with a burst of 30)
limit_req_zone $git_meta_key zone=git_meta:10m rate=1r/s;

# Define the standard per-IP bucket for everyone else (10 req/s, burst 20)
limit_req_zone $git_ip_key zone=git_ip:10m rate=10r/s;

Then, in the git.sync.wtf server block, I enabled both limits. Nginx applies multiple limit_req directives, and the most restrictive one wins. For a Meta IP, the git_ip limit is disabled (empty key), so it only hits the git_meta limit. For a regular user, the git_meta limit is disabled, so they only hit the git_ip limit.

location / {
    limit_req zone=git_meta burst=30 delay=8;
    limit_req zone=git_ip burst=20 nodelay;
    limit_req_status 429;
}

The results were immediate. Before the change, we were seeing around 350,000 requests per day. Since implementing the shared bucket, that has dropped to roughly 86,000 requests per day—a reduction of about 75%—all while keeping the crawler’s access to the site functional (it just gets throttled).

So. No need to give up and/or set extremely short rate limits that also punish regular usage (like cloning a repo) when you’re dealing with distributed botnets or crawlers. Don’t just look at the individual IPs - look at the whole range, find the owner, and group them into a single bucket. Come to think of it, I’m sure there’s some list I can just download and automate a lot of this …

Why I’m not blocking those IP ranges outright? Hey - I want our excellent code to influence all the AI bots.