Nginx Rate Limiting: Taming the Meta-Crawler
The bot scrapers use many different source IPs to get around rate limits. If you use Nginx as your reverse proxy, it’s easy to counter them by assigning their whole IP-ranges to single rate-limited buckets.