How to remove noindex from a live site
Find where the homepage's noindex comes from, a robots meta tag or an X-Robots-Tag header, and remove it from the live site.
Last reviewed
Why it matters
Google drops a page from its results when it finds noindex in a robots meta tag or an X-Robots-Tag header. On the homepage this takes the site's main way in out of search, and it is usually a staging setting carried over at launch.
How to fix it
First find where it comes from. A header means the server, CDN or host adds it; a meta tag means the CMS, theme or app does:
curl -sS -L -D - -o /dev/null https://www.example.com/ | grep -i '^x-robots-tag'
curl -sS -L https://www.example.com/ | grep -ioE "<meta[^>]+name=.?(robots|googlebot)[^>]*>"
Is this site meant to be public? If it is a staging, preview or client-review site, keep the noindex and protect it with a password instead. noindex is only a request: it does not stop anyone opening the address.
Replace the highlighted values: www.example.com with the site's hostname.
WordPress
- Core setting: Settings > Reading > Search engine visibility. Untick Discourage search engines from indexing this site and click Save Changes. Since WordPress 5.3 this box adds a robots meta tag with noindex; it no longer changes robots.txt.
- Yoast SEO, the page: if the homepage is a static page, edit that page, open the Yoast SEO panel's Advanced tab, and under Allow search engines to show this content in search results choose Yes.
- Yoast SEO, the content type: Yoast SEO > Settings > Content types > Pages, and make sure the search-results toggle is on.
- Rank Math, the homepage: Rank Math SEO > Titles & Meta > Homepage > Homepage Robots Meta, and make sure No Index is not selected.
- Rank Math, the page: in the page's Rank Math panel, Advanced tab, Robots Meta: choose Index instead of No Index, then Update. If the Advanced tab is missing, switch Rank Math to Advanced Mode from Rank Math SEO > Dashboard.
- If none of these is set, search the active theme and must-use plugins for a hard-coded noindex.
Sources: WordPress Reading settings, wp_robots_no_robots, Yoast: noindex, Rank Math: noindex, Rank Math: Advanced tab, Rank Math: Titles & Meta.
Shopify
- Online Store > the theme's actions menu > Edit code, and open
layout/theme.liquid. - In
<head>, look for<meta name="robots" content="noindex">, usually inside a Liquid condition such as{% if template contains … %}, and remove the block that matches the homepage. - If a page has the SEO Hidden (
seo.hidden) metafield set, clear its value and save. - If the whole store is in Private (password) mode and is meant to be live, turn it off from the Private label in Online Store; if it is not meant to be live yet, this finding is expected.
Sources: Shopify: hide a page from search engines, password page.
Cloudflare
A Response Header Transform Rule can add X-Robots-Tag on a proxied site.
- Open the Rules Overview page and review the Response Header Transform Rules.
- Delete or narrow any rule that sets or adds
X-Robots-Tagon the production hostname. - To strip a header the origin sends and you cannot fix there: Create rule > Response Header Transform Rule, match the hostname, operation Remove, header name
X-Robots-Tag. Fixing it at the origin is better.
Cloudflare Pages adds X-Robots-Tag: noindex to preview deployments; make sure any rule doing the same is limited to pages.dev hostnames, not the custom domain.
Sources: Response header transform rules, create in the dashboard, Pages preview deployments.
nginx
- Find the directive:
grep -rniE 'x-robots-tag' /etc/nginx/
-
Remove the
add_header X-Robots-Tag …;line from the production server or location block.add_headeris inherited from the level above only when a block has none of its own, so a line inhttp {}or an included snippet can apply to some locations and not others. -
If an app behind
proxy_passsends it, fix the app, or hide the header:
proxy_hide_header X-Robots-Tag;
- Test and reload:
sudo nginx -t && sudo systemctl reload nginx
Sources: ngx_http_headers_module, ngx_http_proxy_module.
Apache
- Find the directive in the server config and the site's
.htaccess:
grep -rniE 'x-robots-tag' /etc/apache2/ /etc/httpd/ 2>/dev/null
-
Remove the
Header set X-Robots-Tag …line. -
To strip it when an app or proxy sends it, unset it in both of Apache's header tables, since a header set in one is not removed by an unset aimed at the other:
Header unset X-Robots-Tag
Header always unset X-Robots-Tag
- Run
apachectl configtest, then reload.
Source: mod_headers.
Caddy
Remove any header X-Robots-Tag … line from the site block. To strip one sent by a reverse_proxy backend:
www.example.com {
header -X-Robots-Tag
reverse_proxy <your backend, for example localhost:8080>
}
Source: the header directive.
Vercel
Vercel adds x-robots-tag: noindex itself only to preview deployments and to deployment URLs, never to a production domain.
- Make sure the address being checked is the production domain, not a
*.vercel.appdeployment URL. - If the production domain sends it, it comes from the project: look in
headersinvercel.json, inheaders()innext.config.js, and in middleware or the framework's metadata. - To send noindex only on a staging domain, limit the rule to that host:
{
"headers": [
{
"source": "/(.*)",
"has": [{ "type": "host", "value": "staging.example.com" }],
"headers": [{ "key": "X-Robots-Tag", "value": "noindex" }]
}
]
}
A custom domain assigned to a Git branch does not get Vercel's noindex, so protect such staging domains with Deployment Protection.
Source: Vercel: X-Robots-Tag.
Netlify
- Netlify adds
X-Robots-Tag: noindexto Deploy Previews and older branch deploys. Make sure the address being checked is the production domain. - On production it comes from your own config: a
_headersfile in the publish directory, a[[headers]]block innetlify.toml, or a function, which must stop sending it itself. - Headers in
_headersandnetlify.tomlapply to every deploy context, production included. To send noindex only on previews, copy a preview-only headers file into place in the preview build command:
[context.deploy-preview]
command = "npm run build && cp ./custom-headers/_previewHeaders ./dist/_headers"
Replace npm run build and dist with your build command and publish directory.
Sources: Netlify: deploy overview, custom headers.
Check that it worked
- Run both commands from the top again: neither should show noindex (or none) for the production hostname.
- Check view-source in a private window, so you are not shown a logged-in view, and purge any CDN or page cache that stores HTML.
- In Google Search Console, open the URL Inspection tool on the homepage, click Test live URL, then Request indexing.
- Make sure robots.txt does not block the page, or Google never sees that the tag is gone.
- Run the Owl Keeper Indexing check again.
Sources: Google: block indexing, URL Inspection.
What can go wrong
- Removing noindex from a staging or client-review site puts it in search. Keep it there and add authentication: Vercel Deployment Protection, Netlify password protection, Cloudflare Access, or HTTP auth.
- WordPress's Discourage search engines box also stops update-service pings; unticking it turns them back on.
- It is not instant: the page returns to results only after Google crawls it again.