A robots.txt file is a plain text file placed in the root directory of a website that instructs search engine crawlers (like Googlebot, Bingbot) which pages or sections of the site they may or may not crawl. It's part of the Robots Exclusion Protocol (REP). While not a security measure, it's crucial for SEO — it helps prevent crawlers from wasting crawl budget on low-value pages (admin panels, staging sites, duplicate content) and ensures important pages are discovered efficiently.

The standard directives supported by most crawlers include:
User-agent: Specifies which crawler the rules apply to (e.g., Googlebot, * for all).
Disallow: Paths the crawler should NOT access (e.g., /admin/).
Allow: Paths the crawler MAY access, even within a disallowed directory.
Sitemap: URL pointing to an XML sitemap.
Crawl-delay: Seconds to wait between requests (supported by Bing, Yandex; ignored by Google).
Note: Noindex and Nofollow are not valid in robots.txt — use meta tags or HTTP headers instead.

Asterisk (*) matches any sequence of characters (including none). For example, /images/*.jpg matches all JPG files in the /images/ directory.
Dollar sign ($) marks the end of the URL path. For example, /admin$ only blocks /admin but allows /admin/ or /admin/login.
Without $, paths are treated as prefix matches — /admin blocks everything starting with /admin. The longest matching rule wins, and if an Allow and Disallow have equal length, Allow takes priority.

1. Using Noindex in robots.txt — This directive doesn't work here; use <meta name="robots" content="noindex"> or the X-Robots-Tag HTTP header.
2. Blocking all crawlers unnecessarily — Disallow: / with User-agent: * blocks everything, including your site from search results.
3. Missing trailing slash — /admin also blocks /administration; use /admin/ for precision.
4. Incorrect file location — robots.txt must be at the root (https://yoursite.com/robots.txt), not in a subdirectory.
5. Empty Disallow — Disallow: with no value means "allow everything," which can be confusing when placed unintentionally.

No. Robots.txt only prevents crawling — it does not prevent indexing. If a blocked page is linked from other sites, search engines may still index it (showing a "no description available" snippet). To truly prevent indexing, use a noindex meta tag or HTTP header. For best results, combine both: block crawling of low-value sections via robots.txt and use noindex for pages that must not appear in search results.

Use this validator to check syntax errors and test URL matching. Additionally, you can use Google Search Console's robots.txt Tester (under Settings > Crawl > robots.txt) to see how Googlebot interprets your file. Always test after making changes — a single typo can accidentally block your entire site from search engines.

Google enforces a 500 KB (kilobytes) maximum file size limit for robots.txt. If your file exceeds this, Google will only process the first 500 KB and ignore the rest. For most websites, this is more than sufficient — a well-structured robots.txt is usually under 5 KB. Keep it lean and avoid unnecessary directives.