How do I edit robots.txt in Webflow?

Add rules under SEO, then publish

A robots.txt file tells crawlers which paths they may request. On a Webflow site it lives at the top of the domain, for example yourdomain.com/robots.txt. Anyone can open that address, so the file is not a lock. Do not list private URLs there and expect them to stay secret.

To create or edit the file, go to Site settings, SEO, Indexing. Add the rules, click Save, and publish. You need a Site plan or a paid Workspace plan before you can add a robots.txt file.

Each rule has two parts. User-agent names the bot. A star means all bots. Googlebot means Google's crawler. Allow opens a path. Disallow blocks a path. A single slash means the whole site.

Webflow already adds your sitemap link. Leave that line alone unless you need a different sitemap address. In that case, turn on Remove sitemap.xml from robots.txt, then add your own Sitemap line, save, and publish.

Leave the defaults when the site should be crawled

Most marketing sites should stay open to search. If you want Google to read the public pages, do not paste a site-wide Disallow. Blocking every bot with Disallow: / asks crawlers to skip the whole site, including pages you hope to rank.

The same Indexing panel has traffic control toggles for groups of known search crawlers and AI bots. Those groups are allowed by default. Turning a group off asks those bots not to crawl or index the site. Publish after you change a toggle. Blocking search crawlers can cut bandwidth, and it can also cut discovery in search. Leave the toggles on unless you have a clear reason to refuse that traffic.

Not every bot obeys robots.txt. Malicious or poorly built bots can still request a folder you disallowed. Use a password, or keep the page as a draft, when the content itself must stay private.

Block one folder, not the whole site

Suppose you publish ad landing pages under /internal/ and you do not want those URLs crawled. You still want the homepage, the pricing page, and the blog indexed. Use a folder rule, save, and publish:

• User-agent: *
• Disallow: /internal/

After publish, open yourdomain.com/robots.txt and confirm the new lines are there. The Designer does not serve this file. Only the published domain does.

A page rule looks the same with a slug, such as Disallow: /secret-page, without a trailing slash on the path. A folder rule keeps the trailing slash so it covers the pages inside that folder.

To block one named bot and leave everyone else alone, put that bot first:

• User-agent: BadBot
• Disallow: /
• User-agent: *
• Allow: /

Replace BadBot with the user-agent string you actually see in your logs. A made-up name does nothing.

Use noindex when a URL is already in search

robots.txt asks a bot not to crawl. It does not reliably pull a URL out of Google after Google already indexed it. A page blocked only in robots.txt can still sit in the auto-generated sitemap. For a page that was indexed before, do not rely on Disallow. Turn Sitemap indexing off on that page, publish, and use Google's removals tool if you need the old result taken down.

That toggle is the noindex control for a static page. Canonical host settings are a separate decision about which domain should be the preferred copy. This file only decides which paths you are willing to let crawlers request.

If you clear the rules box, the published file can remain. A robots.txt file cannot be fully removed once it exists. Replace it with an allow-all pair, save, and publish:

• User-agent: *
• Disallow:

The Disallow line is empty on purpose. That pattern allows crawling again. If the old rules still appear on the live file, publish once more. If the published file stays stale after that, contact Webflow support.

Check robots.txt on the custom domain and, if you use it, on the webflow.io subdomain. Staging indexing is a separate switch in the same Indexing section. Turning staging indexing off publishes a robots.txt on the subdomain that tells search engines to ignore that host. That switch needs a Site plan or a paid Workspace. It does not rewrite the rules you saved for the custom domain.