BTC $64,505.00 −1.17% ETH $1,815.96 −0.77% USDT $0.9994 −0.02% BNB $590.51 −1.08% USDC $0.9997 −0.01% XRP $0.9557 −1.37% SOL $71.75 −0.99% TRX $0.3050 +0.39% FIGR_HELOC $1.04 — STETH $1,796.44 −1.86% HYPE $68.70 +2.84% ZEC $906.37 +0.33% XMR $493.53 −1.00% DOGE $0.0588 −1.13% BTC $64,505.00 −1.17% ETH $1,815.96 −0.77% USDT $0.9994 −0.02% BNB $590.51 −1.08% USDC $0.9997 −0.01% XRP $0.9557 −1.37% SOL $71.75 −0.99% TRX $0.3050 +0.39% FIGR_HELOC $1.04 — STETH $1,796.44 −1.86% HYPE $68.70 +2.84% ZEC $906.37 +0.33% XMR $493.53 −1.00% DOGE $0.0588 −1.13%

Robots.txt Generator

Choose what crawlers may visit, block folders you want kept out, add your sitemap and download a ready robots.txt.

  • 100% Free
  • No sign-up
  • Instant results

What is the Robots.txt Generator?

robots.txt is a small text file at the root of a website that tells crawlers which parts they may visit. Search engines respect it, and so do most AI and SEO bots.

This generator builds the file for you: choose a default rule, block folders such as /admin or /cart, decide per crawler (Google, Bing, AI bots and others) and add your sitemap address. You get a ready-to-upload file with correct syntax.

Remember that robots.txt is a polite request, not a lock. Never use it to hide private information.

Features

Per crawler rules

Treat Google, Bing, AI crawlers and other bots differently.

Block folders

Keep crawlers out of admin, search and cart pages.

Sitemap line

Point crawlers to your XML sitemap.

How to use the Robots.txt Generator

  1. 1

    Pick the default and the folders to block.

  2. 2

    Add your sitemap address.

  3. 3

    Download the file and put it at the root of your site (example.com/robots.txt).

Who it is for

New websites

Start with a clean, correct file instead of copying one.

AI crawler control

Decide whether AI bots may read your content.

Tips for better results

  • Do not block CSS and JavaScript files. Google needs them to see your page properly.
  • Blocking a page in robots.txt does not remove it from Google. Use a noindex tag for that.

Questions people ask

Where does robots.txt go?

In the main folder of your site, so it opens at https://yoursite.com/robots.txt.

Does robots.txt hide a page from Google?

No. It stops crawling, not showing. To keep a page out of results use a noindex meta tag.

Do all bots obey it?

Good bots do. Bad bots ignore it, so never rely on it to protect private pages.

Trustpilot

What our customers say

4.8 / 5Trustpilot rating
FREE BITCOIN TOOLS

Stuck Bitcoin transaction?

Paste your TXID, see where it stands, and give it a free push. No signup, no wallet access.

Try the free accelerator
  • Free
  • No signup
  • Live status