Create crawler directives for your website
Robots.txt is a plain-text file at the root of a site that gives crawler-specific instructions about which URL paths may be crawled. It is useful for managing crawl behavior, but it is not a security feature and it should not be treated as a guaranteed de-indexing mechanism.
Common robots.txt directives
| Directive | Purpose |
|---|---|
| User-agent | Selects the crawler the following rules apply to |
| Disallow | Requests that matching paths not be crawled |
| Allow | Allows a more specific path within a broader blocked section for crawlers that support it |
| Sitemap | Provides the absolute URL of an XML sitemap |
Robots.txt is not a privacy control
Anyone can read a public robots.txt file. Do not list confidential locations there expecting them to become secret. Protect sensitive directories with authentication and server permissions.
Crawling and indexing are different
Blocking a crawler can prevent it from reading page content, but a search engine may still know that a URL exists from links or other signals. If a page must not appear in search, use an appropriate indexing control on a crawlable page or remove/protect the content as required.
How to use the generated file
Review every rule carefully, download the result as robots.txt, place it at the website root and test important URLs after deployment.
Frequently asked questions
Does robots.txt remove a page from Google?
Not reliably. Robots.txt controls crawling, not guaranteed indexing. A blocked URL can still be indexed from external signals without its content being crawled.
Where should robots.txt be located?
For a normal website it should be accessible at the root, for example https://example.com/robots.txt.
Can I block private content with robots.txt?
Do not use robots.txt as an access-control system. Sensitive content should be protected with authentication or other server security.
Should I add my sitemap to robots.txt?
Adding a Sitemap directive is a common way to tell crawlers where the XML sitemap is located.