How to fix a robots.txt that blocks the whole site

Replace the robots.txt rule that disallows everything with one that lets search engines crawl the site, on every hostname it is served from.

Last reviewed

Why it matters

With the whole host disallowed, Google stops crawling the site, so new and changed pages are not picked up. robots.txt blocks crawling rather than indexing, so blocked pages can still be listed without their content, and a blocked page's noindex can never be read.

How to fix it

Replace the blocking group with an allow-all file at https://www.example.com/robots.txt. Keep any narrower Disallow lines you really need, such as an admin path; remove only Disallow: /. The sitemap address must be a full URL:

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

robots.txt is per hostname: www and the apex each serve their own.

Replace the highlighted values: www.example.com with the site's hostname, https://www.example.com/sitemap.xml with your sitemap's address, example.com with your domain.

Any server

Edit or replace robots.txt in the document root or publish directory (nginx, Apache, Caddy, Vercel, Netlify, any static site), redeploy if the site is built, and check every hostname serves the file you meant:

curl -sS https://example.com/robots.txt
curl -sS https://www.example.com/robots.txt

Sources: Google: create a robots.txt, robots.txt introduction.

WordPress

WordPress serves a virtual robots.txt only when there is no physical robots.txt file and it is installed in the site root.

  1. If a physical robots.txt exists in the document root, it wins: edit it or delete it.
  2. If not, core's own file never says Disallow: / (not even with Discourage search engines ticked, since 5.3). A Disallow: / comes from a plugin or theme using the robots_txt filter, often an SEO plugin's robots.txt editor. Find it there and remove the block.

Sources: do_robots, Reading settings.

Shopify

Shopify's default robots.txt never blocks the whole store, so a Disallow: / under User-agent: * comes from a customised robots.txt.liquid.

  1. Online Store > the theme's ⋯ menu > Edit code, and open templates/robots.txt.liquid.
  2. Remove the added Disallow: / line under User-agent: *. Keep the {% for group in robots.default_groups %} loop so Shopify's own rules still come through.
  3. Or return to Shopify's default: in Edit code, right-click robots.txt.liquid, choose Delete permanently, then Delete.

Sources: Shopify: editing robots.txt, robots.txt.liquid.

Cloudflare

Cloudflare's managed robots.txt (part of its AI crawler settings) never adds Disallow: / for User-agent: *: it prepends a block that allows everyone and disallows only named AI crawlers, followed by the origin's own file.

  1. Fetch the full file.
  2. Look below Cloudflare's managed block for the origin's User-agent: * / Disallow: /.
  3. Fix it at the origin, as for your platform above.

Source: Cloudflare: managed robots.txt.

Check that it worked

  1. curl -sS https://www.example.com/robots.txt returns the new file on every hostname you serve.
  2. In Google Search Console, open the robots.txt report, and from the file's menu choose Request a recrawl; otherwise Google refreshes its copy about once a day.
  3. Run the Owl Keeper Indexing check again.

Sources: robots.txt report, submit an updated robots.txt.

What can go wrong

  • Never lift the block on a staging site that should stay private. robots.txt does not keep a site private, and disallowed addresses can still be indexed from links. Protect staging with authentication instead.
  • The deploy may put it back. A staging robots.txt copied into production is the usual cause; check the deploy does not overwrite the file again.
  • On Shopify, a mistake in robots.txt.liquid can cost all search traffic, and the file is not imported when a theme is uploaded from the Themes page.