Block AI Crawlers and Keep Your Site Fast in Search

Host logs show a clear pattern. It has been there for a year. Sudden hits come from bots. A search bot should crawl bit by bit. These bots do not. AI bots hit live pages all at once. They use up all your DB links. They push central processing unit (CPU) use to 100%.

That leaves site owners in a bind. You can block all AI bots at your firewall. You can also block them in robots.txt. Either way your pages drop out of AI search. You lose them in ChatGPT Search. You lose them in Perplexity. You lose them in Google AI Overviews.

Leave the door open and you pay for it. Bots eat up your workers. They push up the Time to First Byte (TTFB). They hurt your Core Web Vitals. Your Google rank drops.

The fix is simple. Split search bots from bulk training bots. Set a clear block rule in your robots file. Add Nginx rate caps that guard your stack.

Search bots and training scrapers.

Sites often mix up two kinds of bot. One is for live search. One is for bulk training. That is the top reason they cut off good hits.

Modern AI bots fall into two groups.

1. Search and link bots. These bots index content for live search products. Or they fetch pages as a guest searches. They give sources and links.
2. Bulk model-training scrapers. These bots read huge parts of the public web. They build data sets for future model runs. They send no hits. They give no sources.

Bot name	Who runs it	Why it comes	Effect on traffic
OAI-SearchBot	OpenAI	ChatGPT Search index	High. It sends you clicks.
ChatGPT-User	OpenAI	Reads one link	Medium. It reads when a guest asks.
GPTBot	OpenAI	Feeds model training	None. It only trains.
PerplexityBot	Perplexity AI	Search index, source links	High. It sends you links.
ClaudeBot	Anthropic	Model training	None. It only trains.
Bytespider	ByteDance	Model training, web scrape	None. It sends big hit bursts.
Google-Extended	Google	Gemini and Vertex AI training	None. It trains, not Search.

Note the split between OAI-SearchBot and GPTBot. OpenAI uses OAI-SearchBot to fill ChatGPT Search results. If you block OAI-SearchBot, ChatGPT Search finds no pages of yours. If you block GPTBot, just your training data stops.

Google keeps a split too. Googlebot powers Search and Google AI Overviews. Google-Extended is its own opt-out for training. Say no to Google-Extended. Google stops using your text for Gemini. Your rank in Google Search does not change. Your place in AI Overviews does not change either.

Should you block all AI crawlers in robots.txt?

Short answer: no, not all of them.

One rule for all bots cuts off new ways. Guests then cannot find you through AI search. Millions of guests now search for products this way. They look for docs and tech help too. An AI search bot builds its reply from search bots. They find the good pages. It cites them and links them. If your robots file says no, it picks a site. It picks one that said yes.

So do not use one rule for all bots. Use a set of rules.

• Let in the search bots you trust. Let in OAI-SearchBot, PerplexityBot, and the big search bots.
• Block or cap the rough bots. Say no to Bytespider and CCBot. Do it when they hurt you. They add no search value.
• Cap the rate at your web host. This stops all bots from taking each worker.

How fast crawling hurts Core Web Vitals.

Bots crawl a site with no cap. The harm does not stop at the CPU. It shows up in your Google Search Console data.

A scraper asks for 40 live archive pages in five seconds. This can happen on a WordPress, Node.js, or Laravel site.

1. PHP-FPM or Node workers starve. Each call takes one worker thread. New calls wait in a line. The line grows with each new call. It stops at max_children. It stops at the thread cap too.
2. DB links fill up. Hard queries lock rows. Or they use up each one. Then queries slow down across the whole app.
3. Time to First Byte goes up. A guest who lands on your site then waits. The wait goes from 150 ms to 3,000 ms or more.
4. Largest Contentful Paint (LCP) gets worse. TTFB is the base of LCP. A high TTFB pushes LCP up. It can land in the "Needs Improvement" band. Or in the "Poor" band. Google shows this in the Chrome User Experience Report (CrUX).

Google ranks sites on real guest data. It looks at 28 days of it. If bot bursts hit during work hours, your scores fall.

Googlebot watches how fast your host replies. A 503 Service Unavailable reply slows it down. Slow replies have the same effect. It does this to save your host. So new posts take far more time to be found. They take far more time to show up in search too.

Setting up robots.txt for AI search.

Keep your search reach and still guard your host. Put your rules in order. First name the search bots to trust. Then add caps for the rest.

Here is a tested setup. It keeps your place in normal search and AI search. It blocks the big bots too.

# Allow traditional search engines
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# Allow AI search retrieval and citation bots
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

# Block bulk training scrapers that offer no search visibility
User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

# Optional: Opt out of foundation model training while keeping search
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ClaudeBot
Disallow: /

# Default policy for standard crawlers
User-agent: *
Disallow: /private/
Disallow: /api/
Disallow: /search

This setup keeps you in normal search and AI search. It blocks the bots that just waste your queries.

Rate limits in Nginx.

A robots.txt file is a request, not a rule. Good firm spiders obey it. Bad bots, fake user agents, and auto tools ignore it.

So set the cap at your web host or proxy. But a plain cap can backfire. It applies per Internet Protocol (IP) address. An office network can share one IP. So can a work VPN or a search bot. A tight cap then blocks real people. Or it makes Googlebot report crawl errors.

Nginx has a good way to do this. Sort each bot into its own rate zone. Real search bots get a high cap. Googlebot and Bingbot can skip the check. AI bots and other tools get a hard cap.

1. Map bots to rate limits.

In your nginx.conf file, add a map. Put it in the http block. It reads the $http_user_agent header.

http {
    # Classify traffic into rate limit zones based on User-Agent
    map $http_user_agent $bot_limit_key {
        default                     "";
        ~*(OAI-SearchBot|PerplexityBot) $binary_remote_addr;
        ~*(GPTBot|ClaudeBot|Bytespider) $binary_remote_addr;
    }

    # Zone for general visitors (generous threshold)
    limit_req_zone $binary_remote_addr zone=general_limit:10m rate=30r/s;

    # Zone specifically applied to identified AI bots (strict threshold)
    limit_req_zone $bot_limit_key zone=bot_limit:10m rate=2r/s;

    # Return standard HTTP 429 Too Many Requests on breach
    limit_req_status 429;

With this setup, plain guests pass. Their $bot_limit_key is empty. So the bot_limit zone never runs for them. A call that matches an AI bot name gets a key. The key is $binary_remote_addr. Then it caps that client at 2 calls per second.

2. Apply the rate limits to your server block.

Now attach the rate rules to your live app paths.

server {
    listen 443 ssl http2;
    server_name example.com;

    location / {
        # Apply the bot limit zone with a small burst pool
        limit_req zone=bot_limit burst=5 nodelay;

        # Standard proxy pass to PHP-FPM, Node.js, or application container
        proxy_pass http://127.0.0.1:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Say an AI bot asks for 20 pages at once. Nginx lets the first 5 through. It uses the burst buffer for them. Then it sends 429 Too Many Requests for the other 15. The bot waits and tries again. The standard reply headers tell it to. Your app workers never hit the CPU cap.

A small cache for bot replies.

A second guard is a small cache. Bots do not need a session. They do not need a cart. They do not need special headers. Give them saved HTML and your app never runs for them.

Your site may build pages on the fly. Set up a cache for them. Use the Nginx FastCGI cache or the proxy cache. Keep the time to live (TTL) short.

# Define proxy cache path
proxy_cache_path /var/cache/nginx/bot_cache levels=1:2 keys_zone=BOT_CACHE:10m max_size=500m inactive=60m;

server {
    location / {
        # Cache identical GET requests for bots for 10 minutes
        proxy_cache BOT_CACHE;
        proxy_cache_valid 200 301 302 10m;
        proxy_cache_use_stale error timeout updating http_500 http_502 http_503;
        proxy_cache_bypass $http_cookie; # bypass cache for logged-in users

        proxy_pass http://127.0.0.1:3000;
    }
}

With a small cache in place, bots read from your disk. One that checks 200 blog posts runs no queries. Reply times drop under 10 ms. You save RAM and CPU.

How to check if a bot is real.

A common mistake is to trust the user agent name. Scraper bots often set it to Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html). Or they set it to OAI-SearchBot. They do this to get past your robots rules.

So check who is really calling. Use a reverse DNS check. Then compare with the IP list each firm puts out.

Checking Googlebot and Bingbot.

Search bots let you check them from the shell. Run host on the odd looking IP.

host 66.249.66.1

The reply gives a host name. It ends in .googlebot.com or .google.com.

1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

Then run a forward DNS check on that host name. It must point back to the same IP.

host crawl-66-249-66-1.googlebot.com

If the host name does not point back, the bot is fake. You can block it at your firewall.

Checking OpenAI and Perplexity bots.

These two put out their own IP lists.

• OpenAI. It posts the crawler IP ranges on its own site. The docs have the exact links.
• Perplexity. It posts the ranges for PerplexityBot. It posts autonomous system numbers (ASNs) too.

Check the ASN for any IP you do not know. Look at the logs for hits from that IP. The request claims to be OAI-SearchBot. Use whois or your firewall logs. Then you know if the call came from their own gear.

Sizing your server for heavy crawl days.

Even with rate limits, a busy site needs room. It must take hits from many bots. Search bots and real guests can arrive at the same time.

When you plan your compute on a VPS or container stack.

• vCPU count. One vCPU thread can serve async Nginx proxy calls. But a live back end needs CPU time of its own. It needs that time while it renders. PHP-FPM and Python WSGI are such back ends. A site with steady search and AI bot hits needs 2 vCPUs. That gives you room. Calls do not wait in a line. See our app hosting plans for the CPU we give you.
• RAM and swap. A host with 4 GB of RAM needs a swap file. RAM means random access memory. Give it 2 GB on fast local disk. That catches a sudden spike. It beats the Linux out-of-memory (OOM) killer. Read our VPS hosting configuration guide for the steps.
• Fast disk reads and writes. Heavy access logs and disk cache write a lot. Slow disks choke under big crawl hits. The same goes for slow network disks. See why fast local storage for Indian hosting cuts the wait.
• One box for each app. Run web apps on one machine? Let bots hit their own box. Then one big bot run cannot starve the rest. It only hits one project. Our app hosting plans sets a cap for each container. See how it works.

Questions people ask.

Does blocking GPTBot remove your site from ChatGPT Search?

Short answer: no, not for search.

OpenAI uses OAI-SearchBot to index pages for ChatGPT Search. ChatGPT-User runs when a guest asks it to read a page. GPTBot just builds model training data. So you can say no to GPTBot in robots.txt. You keep your text safe. You keep your search links and clicks from ChatGPT Search.

Will sending back HTTP 429 hurt my search rank?

Short answer: not if you hit only the worst bots.

The 429 code means Too Many Requests. It is the standard way to say a bot went too far. Bots know this code. They wait, then try again at a slower pace. But keep Googlebot and real guests out of the strict zones. If Googlebot keeps getting 429 or 503 codes, it slows down. Then your new pages take far longer to show up. Search takes longer to list them.

How do AI search bots crawl?

Plain search bots send out web spiders. They crawl over weeks. They store raw pages in huge search lists. AI search bots often mix two jobs. They keep a list for plain lookups. They fetch pages on demand. That happens when a question needs fresh facts or new docs. Those on-demand reads run while a guest waits. So they often hit deep links. They skip the pages in your site map.

Does a fast TTFB help AI search cite you more?

Short answer: Yes.

AI search bots reply under a tight time limit. Say one tries to fetch your page. Your host takes more than 1.5 seconds to send the HTML. It gives up. Then it cites a rival page. That page replied in 200 ms, not 1.5 seconds. A fast TTFB lets search bots read your text. AI bots read it too. They do it before the time runs out.