How to remove noindex from a live site

Find where the homepage's noindex comes from, a robots meta tag or an X-Robots-Tag header, and remove it from the live site.

Last reviewed

Why it matters

Google drops a page from its results when it finds noindex in a robots meta tag or an X-Robots-Tag header. On the homepage this takes the site's main way in out of search, and it is usually a staging setting carried over at launch.

How to fix it

First find where it comes from. A header means the server, CDN or host adds it; a meta tag means the CMS, theme or app does:

curl -sS -L -D - -o /dev/null https://www.example.com/ | grep -i '^x-robots-tag'
curl -sS -L https://www.example.com/ | grep -ioE "<meta[^>]+name=.?(robots|googlebot)[^>]*>"

Is this site meant to be public? If it is a staging, preview or client-review site, keep the noindex and protect it with a password instead. noindex is only a request: it does not stop anyone opening the address.

Replace the highlighted values: www.example.com with the site's hostname.

WordPress

  1. Core setting: Settings > Reading > Search engine visibility. Untick Discourage search engines from indexing this site and click Save Changes. Since WordPress 5.3 this box adds a robots meta tag with noindex; it no longer changes robots.txt.
  2. Yoast SEO, the page: if the homepage is a static page, edit that page, open the Yoast SEO panel's Advanced tab, and under Allow search engines to show this content in search results choose Yes.
  3. Yoast SEO, the content type: Yoast SEO > Settings > Content types > Pages, and make sure the search-results toggle is on.
  4. Rank Math, the homepage: Rank Math SEO > Titles & Meta > Homepage > Homepage Robots Meta, and make sure No Index is not selected.
  5. Rank Math, the page: in the page's Rank Math panel, Advanced tab, Robots Meta: choose Index instead of No Index, then Update. If the Advanced tab is missing, switch Rank Math to Advanced Mode from Rank Math SEO > Dashboard.
  6. If none of these is set, search the active theme and must-use plugins for a hard-coded noindex.

Sources: WordPress Reading settings, wp_robots_no_robots, Yoast: noindex, Rank Math: noindex, Rank Math: Advanced tab, Rank Math: Titles & Meta.

Shopify

  1. Online Store > the theme's actions menu > Edit code, and open layout/theme.liquid.
  2. In <head>, look for <meta name="robots" content="noindex">, usually inside a Liquid condition such as {% if template contains … %}, and remove the block that matches the homepage.
  3. If a page has the SEO Hidden (seo.hidden) metafield set, clear its value and save.
  4. If the whole store is in Private (password) mode and is meant to be live, turn it off from the Private label in Online Store; if it is not meant to be live yet, this finding is expected.

Sources: Shopify: hide a page from search engines, password page.

Cloudflare

A Response Header Transform Rule can add X-Robots-Tag on a proxied site.

  1. Open the Rules Overview page and review the Response Header Transform Rules.
  2. Delete or narrow any rule that sets or adds X-Robots-Tag on the production hostname.
  3. To strip a header the origin sends and you cannot fix there: Create rule > Response Header Transform Rule, match the hostname, operation Remove, header name X-Robots-Tag. Fixing it at the origin is better.

Cloudflare Pages adds X-Robots-Tag: noindex to preview deployments; make sure any rule doing the same is limited to pages.dev hostnames, not the custom domain.

Sources: Response header transform rules, create in the dashboard, Pages preview deployments.

nginx

  1. Find the directive:
grep -rniE 'x-robots-tag' /etc/nginx/
  1. Remove the add_header X-Robots-Tag …; line from the production server or location block. add_header is inherited from the level above only when a block has none of its own, so a line in http {} or an included snippet can apply to some locations and not others.

  2. If an app behind proxy_pass sends it, fix the app, or hide the header:

proxy_hide_header X-Robots-Tag;
  1. Test and reload:
sudo nginx -t && sudo systemctl reload nginx

Sources: ngx_http_headers_module, ngx_http_proxy_module.

Apache

  1. Find the directive in the server config and the site's .htaccess:
grep -rniE 'x-robots-tag' /etc/apache2/ /etc/httpd/ 2>/dev/null
  1. Remove the Header set X-Robots-Tag … line.

  2. To strip it when an app or proxy sends it, unset it in both of Apache's header tables, since a header set in one is not removed by an unset aimed at the other:

Header unset X-Robots-Tag
Header always unset X-Robots-Tag
  1. Run apachectl configtest, then reload.

Source: mod_headers.

Caddy

Remove any header X-Robots-Tag … line from the site block. To strip one sent by a reverse_proxy backend:

www.example.com {
	header -X-Robots-Tag
	reverse_proxy <your backend, for example localhost:8080>
}

Source: the header directive.

Vercel

Vercel adds x-robots-tag: noindex itself only to preview deployments and to deployment URLs, never to a production domain.

  1. Make sure the address being checked is the production domain, not a *.vercel.app deployment URL.
  2. If the production domain sends it, it comes from the project: look in headers in vercel.json, in headers() in next.config.js, and in middleware or the framework's metadata.
  3. To send noindex only on a staging domain, limit the rule to that host:
{
  "headers": [
    {
      "source": "/(.*)",
      "has": [{ "type": "host", "value": "staging.example.com" }],
      "headers": [{ "key": "X-Robots-Tag", "value": "noindex" }]
    }
  ]
}

A custom domain assigned to a Git branch does not get Vercel's noindex, so protect such staging domains with Deployment Protection.

Source: Vercel: X-Robots-Tag.

Netlify

  1. Netlify adds X-Robots-Tag: noindex to Deploy Previews and older branch deploys. Make sure the address being checked is the production domain.
  2. On production it comes from your own config: a _headers file in the publish directory, a [[headers]] block in netlify.toml, or a function, which must stop sending it itself.
  3. Headers in _headers and netlify.toml apply to every deploy context, production included. To send noindex only on previews, copy a preview-only headers file into place in the preview build command:
[context.deploy-preview]
  command = "npm run build && cp ./custom-headers/_previewHeaders ./dist/_headers"

Replace npm run build and dist with your build command and publish directory.

Sources: Netlify: deploy overview, custom headers.

Check that it worked

  1. Run both commands from the top again: neither should show noindex (or none) for the production hostname.
  2. Check view-source in a private window, so you are not shown a logged-in view, and purge any CDN or page cache that stores HTML.
  3. In Google Search Console, open the URL Inspection tool on the homepage, click Test live URL, then Request indexing.
  4. Make sure robots.txt does not block the page, or Google never sees that the tag is gone.
  5. Run the Owl Keeper Indexing check again.

Sources: Google: block indexing, URL Inspection.

What can go wrong

  • Removing noindex from a staging or client-review site puts it in search. Keep it there and add authentication: Vercel Deployment Protection, Netlify password protection, Cloudflare Access, or HTTP auth.
  • WordPress's Discourage search engines box also stops update-service pings; unticking it turns them back on.
  • It is not instant: the page returns to results only after Google crawls it again.