What is Robots.txt?

Robots.txt is a text file at the root of a domain that tells search-engine crawlers which URLs they may or may not request. Robots.txt controls crawling but does not reliably prevent indexing — use a noindex tag for that.