What is Robots.txt?
Robots.txt is a text file at the root of a domain that tells search-engine crawlers which URLs they may or may not request. Robots.txt controls crawling but does not reliably prevent indexing — use a noindex tag for that.
Why Robots.txt matters
Robots.txt is the first file a well-behaved crawler requests, and it is the cheapest way to keep bots out of search pages, admin areas, cart URLs and infinite calendar archives that would otherwise burn crawl budget. It is also where you point crawlers at the sitemap. In 2026 it carries a second job: declaring which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) may read the site, which is now a visibility decision, not only a privacy one.
Its limits are just as important. A disallowed URL can still be indexed if other pages link to it, just without a snippet; to keep a page out of search you need a noindex tag, which requires the page to be crawlable. And one typo, such as Disallow: / left over from staging, removes the whole site from search until someone notices.
Example
A typical file for a SaaS site that welcomes AI crawlers:
User-agent: *
Disallow: /admin/
Disallow: /api/
User-agent: GPTBot
Allow: /
Sitemap: https://example.com/sitemap.xmlPrivate areas are blocked, everything else is open, and the sitemap is declared.
Common mistakes
- •Shipping the staging file's
Disallow: /to production. - •Blocking CSS and JavaScript folders, which stops Google from rendering the page correctly.
- •Using robots.txt to hide a page that should carry
noindexinstead.
How Webmatik checks it
The Search engine instructions (robots.txt) check fetches the file, confirms it parses and that it does not block the whole site, and reads the declared sitemap. The Agent readiness checks read the AI crawler rules and Content Signals in the same file.
Check your own site
Free SEO Checker
Title tags, headings, structured data, crawlability — and the keywords you already rank for.
Related SEO terms
See how your website scores across SEO, GEO, performance, and more
Run a free audit