AI Crawler Robots.txt Generator PHP Script

A lightweight PHP script that generates robots.txt rules for search crawlers, AI crawlers, SEO bots, sitemap URLs, and crawl-delay settings.

What this script does —

This lightweight PHP script generates a robots.txt file for regular search crawlers, AI crawlers, SEO crawlers, sitemap URLs, crawl-delay rules, and common private folders.

It does not need a database, API key, JavaScript framework, or admin panel. Drop it into a PHP file, open it in your browser, choose your options, and copy the generated robots.txt output.

Why it is useful —

More site owners are paying attention to AI crawler traffic. Some want to allow normal search engines while blocking bots used for AI training, SEO scraping, or heavy automated crawling.

This script gives you a simple starting point without needing to remember every user-agent name by hand.

The PHP script —

<?php
declare(strict_types=1);

function h(string $value): string {
  return htmlspecialchars($value, ENT_QUOTES, 'UTF-8');
}

$aiBots = [
  'GPTBot',
  'OAI-SearchBot',
  'Google-Extended',
  'ClaudeBot',
  'Claude-SearchBot',
  'PerplexityBot',
  'Applebot-Extended',
  'CCBot',
  'Bytespider',
  'Amazonbot',
];

$seoBots = [
  'AhrefsBot',
  'SemrushBot',
  'MJ12bot',
  'DotBot',
  'BLEXBot',
];

$sitemap = trim((string)($_POST['sitemap'] ?? ''));
$crawlDelay = trim((string)($_POST['crawl_delay'] ?? ''));
$blockAi = isset($_POST['block_ai']);
$blockSeo = isset($_POST['block_seo']);
$blockUploads = isset($_POST['block_uploads']);

$robots = "User-agent: *\n";
$robots .= "Allow: /\n";

if ($blockUploads) {
  $robots .= "Disallow: /uploads/\n";
  $robots .= "Disallow: /private/\n";
  $robots .= "Disallow: /admin/\n";
}

if ($crawlDelay !== '' && ctype_digit($crawlDelay)) {
  $robots .= "Crawl-delay: " . $crawlDelay . "\n";
}

if ($blockAi) {
  foreach ($aiBots as $bot) {
    $robots .= "\nUser-agent: " . $bot . "\n";
    $robots .= "Disallow: /\n";
  }
}

if ($blockSeo) {
  foreach ($seoBots as $bot) {
    $robots .= "\nUser-agent: " . $bot . "\n";
    $robots .= "Disallow: /\n";
  }
}

if ($sitemap !== '') {
  $robots .= "\nSitemap: " . $sitemap . "\n";
}
?>
<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>AI Crawler Robots.txt Generator</title>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <style>
    body {
      font-family: Arial, sans-serif;
      max-width: 760px;
      margin: 30px auto;
      padding: 0 16px;
      line-height: 1.5;
      background: #fff;
      color: #111;
    }

    input[type="url"],
    input[type="number"],
    textarea {
      width: 100%;
      box-sizing: border-box;
      padding: 10px;
      margin: 6px 0 14px;
      border: 1px solid #999;
      font: inherit;
    }

    textarea {
      min-height: 320px;
      font-family: Consolas, Monaco, monospace;
      font-size: 14px;
    }

    label {
      display: block;
      margin: 10px 0;
    }

    button {
      padding: 10px 14px;
      border: 0;
      background: #111;
      color: #fff;
      cursor: pointer;
      font: inherit;
    }

    .box {
      border: 1px solid #ccc;
      padding: 14px;
      margin: 18px 0;
      background: #f7f7f7;
    }
  </style>
</head>
<body>

<h1>AI Crawler Robots.txt Generator</h1>

<p>
  Generate a simple robots.txt file that can allow normal search crawling while blocking selected AI and SEO crawlers.
</p>

<form method="post">
  <div class="box">
    <label>
      <input type="checkbox" name="block_ai" value="1" <?= $blockAi ? 'checked' : ''; ?>>
      Block common AI crawlers
    </label>

    <label>
      <input type="checkbox" name="block_seo" value="1" <?= $blockSeo ? 'checked' : ''; ?>>
      Block common SEO crawlers
    </label>

    <label>
      <input type="checkbox" name="block_uploads" value="1" <?= $blockUploads ? 'checked' : ''; ?>>
      Block common private folders
    </label>
  </div>

  <label for="sitemap">Sitemap URL</label>
  <input
    id="sitemap"
    type="url"
    name="sitemap"
    value="<?= h($sitemap); ?>"
    placeholder="https://example.com/sitemap.xml"
  >

  <label for="crawl_delay">Crawl delay in seconds</label>
  <input
    id="crawl_delay"
    type="number"
    name="crawl_delay"
    value="<?= h($crawlDelay); ?>"
    placeholder="10"
    min="1"
    max="120"
  >

  <button type="submit">Generate robots.txt</button>
</form>

<h2>Your robots.txt</h2>

<textarea readonly><?= h($robots); ?></textarea>

<p>
  Upload this file as <strong>robots.txt</strong> in the root folder of your website.
</p>

</body>
</html>

How to use it —

Save the script as something like robots-generator.php, upload it to your site, and open it in your browser.

Choose the crawler groups you want to block, add your sitemap URL if you have one, then copy the generated output into a file named robots.txt.

Your robots.txt file should be placed at the root of your domain, like:

https://example.com/robots.txt

Important note —

Robots.txt is a request to crawlers, not a security system. Good crawlers usually respect it. Bad bots, scrapers, and abusive traffic may ignore it completely.

Do not use robots.txt to hide private files. Use passwords, server rules, noindex tags, or proper access control for anything sensitive.

Good next upgrade —

A stronger version could add checkboxes for each bot individually, a live preview, preset modes, and a download button for saving the generated robots.txt file.

Home Contact Privacy Policy Terms of Use Disclaimer Donate