How to Block AI Scrapers on Your VPS with Nginx

In mid-October 2026, communities were watching Cloudflare Radar’s public bot charts for Yandex’s network, AS13238. The metric Cloudflare publishes for that ASN classified 99.6% of the HTTP requests as bot traffic, and on Hacker News the community reported a sudden drop in that bot volume after a reported strike on a Yandex datacenter.[12] Whatever the exact cause, the signal is the same one you see in your own access logs: a large share of the “bot flood” hitting a public site comes from a handful of networks and crawlers, and it shows up as requests with recognizable user agents.

You cannot control those waves at the source. What you can control is your own edge. If you run nginx as the reverse proxy in front of your app, you have four real levers, and three of them are stock nginx:

1. robots.txt for the crawlers that cooperate.
2. map + return 444 to hard-block the known AI crawler user agents.
3. limit_req to rate-limit the flood that spoofs or skips a user agent.
4. TLS fingerprinting (JA3/JA4) for the crawlers that fake a browser user agent. This is the hard one, and stock nginx cannot do it (details below).

This guide assumes you already have a working nginx reverse proxy; if you do not, start with our hardened reverse-proxy config first.

Know the user agents you are blocking

Before you write a single directive, you need to know that “AI crawler” is not one thing. Each vendor runs several bots with different jobs, and blocking the wrong one costs you visibility in a search product you may actually want. These are the tokens you will see in 2026, from the vendors’ own documentation:

Two things in that table are easy to get wrong.

Google-Extended never appears in a user agent. No request ever carries “Google-Extended” in its User-Agent header. It is a standalone product token you put in robots.txt: it tells Google that content crawled from your site may not be used for training future Gemini models or for grounding in Gemini Apps and Vertex AI, and Google is explicit that it does not affect your site’s inclusion in Google Search or act as a ranking signal.[5] So you handle Google-Extended in robots.txt, not in your nginx map.

The user-triggered fetchers may not respect robots.txt. OpenAI documents that ChatGPT-User requests are initiated by a user and that robots.txt rules may not apply to them.[2] Amazon says the same for Amzn-User.[4] This matters for the enforcement layer: polite controls are not enough.

Layer 1: robots.txt, the cooperative layer

robots.txt is governed by RFC 9309 (the Robots Exclusion Protocol, an IETF standard since 2022). Two of its properties matter for this task. First, it is a set of requests, not access authorization: the standard is explicit that these rules are not a form of access control, and any crawler can ignore them.[1] Second, matching is by product token: a crawler finds the group whose user-agent line matches its token, and the longest matching path wins, with /robots.txt itself always allowed.[1]

So robots.txt is the polite negotiation layer for compliant crawlers. A 2026 config that keeps you visible in search while opting out of AI training looks like this:

# /var/www/example.com/robots.txt
User-agent: *
Allow: /

# AI training and model crawlers: block
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

# Google-Extended is a token, not a crawler: opt out of Gemini training/grounding
User-agent: Google-Extended
Disallow: /

# Search bots you may want to keep (edit to taste):
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

Note the choice to block OAI-SearchBot is a business decision: OpenAI says sites opted out of OAI-SearchBot will not appear in ChatGPT search answers, though they can still appear as navigational links.[2] If you want ChatGPT search to cite you, allow it.

One non-standard directive deserves a mention: Crawl-delay. Anthropic explicitly supports it for its bots and documents User-agent: ClaudeBot + Crawl-delay: 1 as a way to slow crawling without blocking it.[3] Amazonbot, by contrast, does not support the crawl-delay directive at all.[4] Treat Crawl-delay as vendor-specific, not universal.

The critical caveat is the one from RFC 9309 itself: these are requests, not enforcement.[1] The crawlers that matter most for your server load are precisely the ones that may not read or obey robots.txt. That is what the nginx layers are for.

Layer 2: map + return 444, the enforcement layer

The ngx_http_map_module creates a variable whose value depends on another variable, with the mapping defined once in the http context.[8] Its regex syntax is exactly what a user-agent block list needs: ~ for case-sensitive matching and ~* for case-insensitive matching, evaluated in order, with a default for everything else.[8]

Put this in a file included from the http block (typically /etc/nginx/conf.d/ai-scrapers.conf):

# /etc/nginx/conf.d/ai-scrapers.conf
# Match on the token, not the full UA string: version numbers change.
map $http_user_agent $is_ai_scraper {
    default 0;

    ~*GPTBot            1;
    ~*OAI-SearchBot     1;   # remove if you want ChatGPT search to cite you
    ~*ClaudeBot         1;
    ~*Claude-SearchBot  1;
    ~*Bytespider        1;
    ~*Amazonbot         1;
    ~*Meta-ExternalAgent 1;
    ~*meta-externalagent 1;
}

Then in the server block for your site, block the matches. The cleanest stock-nginx move is return 444, a non-standard nginx code that closes the connection without sending a response header at all.[9] It costs the crawler a full TCP round-trip and produces no body to parse:

# in server { }
if ($is_ai_scraper) {
    return 444;
}

Two practical notes on this layer.

Match the token, not the full string. OpenAI’s example GPTBot string ends compatible; GPTBot/1.4; +https://openai.com/gptbot, and the version number changes.[2] The substring regex ~*GPTBot catches it regardless of version. The same applies to Amazonbot/0.1 and meta-externalagent/1.1.[4][6] If a crawler rotates its UA casing, ~* (case-insensitive) handles it; if you want to be strict about a specific casing, use ~.

Prefer 444 over 403 for floods, but log the decision. return 403 returns a real HTTP response, which means the connection completes and the request is visible in your access log as a 4xx. return 444 drops the connection silently. For defense, 444 is cheaper; for debugging and for honest analytics, you may want the 403 so the block is countable. A common middle ground: keep 444 for the known training crawlers and count them in a dedicated access log, or just run grep on $http_user_agent.

Test it with curl before you reload into production:

curl -sS -o /dev/null -w '%{http_code}\n' -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot' https://example.com/
# 000  -> connection closed without response (444 worked)
curl -sS -o /dev/null -w '%{http_code}\n' https://example.com/
# 200  -> normal browsers are unaffected

Layer 3: limit_req, the flood damper

User-agent blocking handles the crawlers that identify themselves. The flood that remains is the one that spoofs a real browser UA or sends no UA at all. For that, the right tool is ngx_http_limit_req_module, which implements a leaky-bucket rate limit keyed on any variable, typically the client IP.[10]

A zone is declared in the http context and referenced in a server or location:

# /etc/nginx/conf.d/bot-flood.conf (http context)
limit_req_zone $binary_remote_addr zone=perip:10m rate=5r/s;

# server { ... location / { ... } }
server {
    listen 443 ssl;
    server_name example.com;

    location / {
        # proxy_pass / root config from your reverse-proxy setup...

        limit_req zone=perip burst=20 nodelay;
    }
}

A few facts from the module docs shape how you tune this:

• The zone uses the leaky bucket model: requests are processed at rate, excess is delayed up to burst, and beyond that the request is rejected.[10]
• nodelay drops the queueing: excessive requests are rejected immediately rather than delayed. Use it for bot traffic where a delayed response is meaningless; omit it where you would rather slow down than cut off.[10]
• The delay parameter (nginx 1.15.7+) lets you delay up to a limit and reject beyond it, which is a good fit for protecting an API from bursts while never queueing a flood.[10]
• limit_req_status defaults to 503; you can set 429 Too Many Requests if that fits your contract with clients better.[10]
• The $limit_req_status variable (1.17.6+) reports PASSED, DELAYED, or REJECTED per request, so you can log and count what the limiter actually did.[10]
• Zone size math: a one-megabyte zone holds roughly 16 thousand 64-byte states on 32-bit platforms, or 8 thousand 128-byte states on 64-bit platforms, and a state is created per distinct key (here, per client IP).[10] A 10m zone is comfortable for a small VPS; size it for the number of distinct IPs you expect, not the request rate.

There is no universal correct rate, and we are not going to invent one. The honest workflow is: look at your real access logs, find the peak rate from your normal traffic (humans, your own monitoring), and set the limit an order of magnitude above that so the flood is what gets cut, not your users. If you want to stage it safely, limit_req_dry_run on (1.17.1+) counts excessive requests in the zone without enforcing, which is the right way to validate a new limit against live traffic before it starts rejecting.[10]

Two caveats on IP-based limiting. First, IPv6 changes the picture: $binary_remote_addr is 16 bytes for IPv6 and a /64 behind a NAT or a large proxy pool can look like one address, so per-IP limits can over-reject legitimate users on shared addresses. Second, a distributed flood from many IPs defeats a per-IP limiter entirely; at that point the limit is a backstop for your origin, not a cure.

Layer 4: TLS fingerprints (JA3/JA4), and why it is the hard one

The crawler that defeats layers 2 and 3 is the one that sends a real Chrome user agent from many IPs. The common way to catch those is a TLS client fingerprint: JA3 (an MD5 hash of the TLS ClientHello fields) or its successor JA4, which normalizes the same signal into a greppable string. Different HTTP clients emit observably different ClientHello shapes, so a real Chrome on Windows and an AI scraper’s Python HTTP client produce different fingerprints even with identical user agents.

Here is the honest part: stock nginx has no JA3/JA4 support. It is not a directive in the nginx docs and you will not find it in a distro package. Implementing it means building nginx from source with a third-party module plus a patched OpenSSL that exposes the raw ClientHello. Projects like nginx-ssl-fingerprint do exactly that: they patch the nginx core and OpenSSL, and expose variables such as $http_ssl_ja3 and $http_ssl_ja4 that you can then use in a map exactly like a user agent.[13] It works, and it is a genuine capability, but it is a source-build commitment: you take on maintaining a patched nginx and OpenSSL yourself. For most small VPS setups, that maintenance cost is not worth it, and the UA map plus rate limiting already removes the large majority of AI-scraper traffic.

The realistic alternative when you need fingerprint-level defense without the build is to put a managed edge in front of nginx that does the fingerprinting for you. That is a different deployment, not something nginx on its own does.

Do not break the search engines

The whole point of matching the token rather than blocking everything is that real search crawlers identify themselves the same way, and you want them to keep coming. Google’s crawlers identify themselves three ways: the user agent, the source IP, and the reverse-DNS hostname, and Google publishes both the IP ranges and the verification procedure so you can tell a real Googlebot from a fake one.[14] If you are going to be aggressive with return 444, put Googlebot, Bingbot and YandexBot on an explicit allowlist and verify a sample of their requests by reverse DNS before you trust a log line that claims to be one of them.

Blocking Googlebot with a blanket rule is how a site quietly disappears from Google Search. That is exactly why Google split out Google-Extended: you can keep Googlebot crawling and indexing while opting out of Gemini training, entirely via robots.txt, with no nginx involvement.[5]

Verify, then reload

Apply the changes the usual way, and validate each layer independently:

sudo nginx -t          # config must parse cleanly first
sudo systemctl reload nginx

# sanity checks against the live server
curl -sS -o /dev/null -w '%{http_code}\n' -A 'GPTBot/1.4' https://example.com/
curl -sS -o /dev/null -w '%{http_code}\n' -A 'Bytespider' https://example.com/
curl -sS -o /dev/null -w '%{http_code}\n' https://example.com/     # humans still 200

# confirm what the limiter did
grep ' 503 ' /var/log/nginx/access.log | tail

If you automate reloads, the Control API guide shows how to turn the reload into a request that answers with a JSON verdict instead of a log line.

The short version

Three stock-nginx layers cover the practical 2026 case: robots.txt per RFC 9309 for the crawlers that cooperate (block GPTBot, ClaudeBot, Bytespider, Amazonbot, Meta-ExternalAgent and the Google-Extended token; keep OAI-SearchBot if you want ChatGPT to cite you); a map over $http_user_agent with ~* token regexes and return 444 for the crawlers that identify themselves; and a limit_req zone keyed on $binary_remote_addr with nodelay for the flood that spoofs or skips a user agent. Validate with limit_req_dry_run first, and never block Googlebot, Bingbot or YandexBot with the same brush. JA3/JA4 fingerprinting is real but requires a source-built nginx with a third-party module, so treat it as the escalation path, not the default. The defaults here are the ones that actually install on a stock VPS.

Sources

1. RFC 9309 - Robots Exclusion Protocol (groups, product tokens, longest-match; “not a form of access authorization”; /robots.txt implicitly allowed). https://www.rfc-editor.org/rfc/rfc9309.html - checked 2026-10-13
2. OpenAI - Overview of OpenAI Crawlers (GPTBot, OAI-SearchBot, ChatGPT-User; robots.txt tags; published IPs; ~24h update). https://developers.openai.com/api/docs/bots - checked 2026-10-13
3. Anthropic - Does Anthropic crawl data from the web… (ClaudeBot, Claude-SearchBot, Claude-User; Crawl-delay support; robots.txt honored). https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-how-can-site-owners-block-the-crawler - checked 2026-10-13
4. Amazon - About AmazonBot (Amazonbot/0.1; respects Robots Exclusion Protocol; does not support crawl-delay; robots meta tags; published IPs). https://developer.amazon.com/amazonbot - checked 2026-10-13
5. Google - Google’s common crawlers (Google-Extended is a standalone product token for Gemini training/grounding; no impact on Google Search). https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers - checked 2026-10-13
6. Meta - Meta Web Crawlers (Meta-ExternalAgent trains foundation AI models / indexes content). https://developers.facebook.com/docs/sharing/webmasters/crawler - checked 2026-10-13
7. monperrus/crawler-user-agents - maintained crawler user-agent list (Bytespider, GPTBot, Amazonbot, meta-externalagent, OAI-SearchBot, Claude-SearchBot). https://github.com/monperrus/crawler-user-agents - checked 2026-10-13
8. nginx - Module ngx_http_map_module (map variables, ~/~* regex, default). https://nginx.org/en/docs/http/ngx_http_map_module.html - checked 2026-10-13
9. nginx - Module ngx_http_rewrite_module (return 444 closes connection without a response). https://nginx.org/en/docs/http/ngx_http_rewrite_module.html - checked 2026-10-13
10. nginx - Module ngx_http_limit_req_module (leaky bucket, burst, nodelay, delay 1.15.7+, dry_run 1.17.1+, limit_req_status default 503, $limit_req_status 1.17.6+, zone size math). https://nginx.org/en/docs/http/ngx_http_limit_req_module.html - checked 2026-10-13
11. nginx - download (mainline 1.31.8, 2026-10-21; stable 1.30.6, 2026-10-14). https://nginx.org/en/download.html - checked 2026-11-07
12. Cloudflare Radar - Bot traffic from AS13238 (Yandex), last 7 days (99.6% bot classification); Hacker News thread “Sudden drop in Cloudflare bot traffic after strike on Yandex datacenter” (community report, 2026-10-11). https://radar.cloudflare.com/bots/as13238 ; https://news.ycombinator.com/item?id=50041619 - checked 2026-10-13
13. nginx-ssl-fingerprint - JA3/JA4 module for nginx (requires nginx core + OpenSSL patches, source build; exposes $http_ssl_ja3/$http_ssl_ja4). https://github.com/phuslu/nginx-ssl-fingerprint - checked 2026-10-13
14. Google - Overview of Google crawlers and fetchers (identification via user agent, source IP, reverse DNS). https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers - checked 2026-10-13