Skip to content
Get a free mockup
Menu

"Discourage search engines from indexing this site": what the WordPress setting does, and how to check yours

Haider Ali· · 4 min read

One checkbox in WordPress, under Settings, Reading, adds a noindex tag to every page of your site. It is meant for sites still being built, and easy to forget at launch. What it does since WordPress 5.3, how to check it in a minute, and the other ways the same tag gets switched on.

In WordPress, under Settings, Reading, there is a checkbox labelled "Discourage search engines from indexing this site". Tick it and every page of the site asks search engines to leave it out. It is meant for sites still being built, and because nothing on the site itself looks different, it is easy to forget at launch.

5 of the 315 small-business homepages in our check still told search engines not to index them. Here is what the setting does, how to check it in a minute, and the other ways the same tag ends up on a site.

What the setting does

Since WordPress 5.3, ticking the box adds this tag to the head of every page:

<meta name='robots' content='noindex, nofollow' />

Older versions added Disallow: / to robots.txt instead. WordPress changed it because, as the announcement explains (quoting Yoast's Joost de Valk), a site can be listed without ever being crawled: a page blocked in robots.txt can still appear in Google if other sites link to it. Since WordPress 5.7 the tag comes from the built-in robots API.

Two things the setting is not:

  • It is not a lock. WordPress's own documentation is clear that nothing is blocked: "it is up to search engines to honor your request" (Reading settings). Visitors and crawlers can still load every page.
  • It is not partial. It applies to the whole site at once.

Why it hides you from more than Google

Google honors the tag fully: a page with noindex is dropped from Google Search altogether, even when other sites link to it (Google's noindex documentation). Bing reads the same tag, and its webmaster guidelines say noindex keeps a page out of Copilot's answers as well as Bing search.

AI assistants that search the web build their answers from search results and indexes, so a site that asks to be left out of indexes is unlikely to be cited by them either. That is our reading of how the pieces fit, not a rule any of them has published, but we have not seen a noindexed small-business site turn up in an AI answer.

How to check yours in a minute

  1. In WordPress: Settings, Reading, "Search engine visibility". The box should be unticked on a live site.
  2. On the page itself: open your homepage, press Ctrl+U (Cmd+Option+U on a Mac) to view the source, then Ctrl+F and search for noindex. No match is what you want.
  3. In Google Search Console: URL Inspection on your homepage. "Excluded by 'noindex' tag" confirms it.
  4. All at once: our free AI visibility checker reports noindex, nosnippet and the other rules that keep a page out of search and AI answers.

How to switch it off

  1. Untick the box and click Save Changes.
  2. Clear any page cache (your caching plugin, your host's cache or Cloudflare), or visitors and crawlers keep getting the old pages with the tag.
  3. Check the source again: the noindex should be gone.
  4. In Search Console, request indexing for your homepage and resubmit your sitemap. Google recrawls in its own time, usually days, sometimes weeks for a young site.

Other ways noindex gets switched on

If the checkbox is clear and the tag is still there, one of these is adding it:

  • An SEO plugin. Yoast and Rank Math can set noindex per page, per post type or per taxonomy. Check the plugin's search appearance or titles and meta settings, and the plugin box on the page itself.
  • A coming-soon or maintenance mode in a theme, page builder or plugin.
  • A copied staging site. A staging copy is often set to noindex, and when it is pushed to the live domain the setting comes with it.
  • An HTTP header. The same instruction can be sent as X-Robots-Tag: noindex by the server, a security plugin or the CDN. It does not show in the page source; URL Inspection and our checker both read headers.

Keeping a staging site out of Google properly

The checkbox is a request. For a staging or development copy, put it behind a password instead: Google's advice for keeping a page out of search is noindex or a password, and a password also keeps out everyone else. Do not add a robots.txt block on top of a noindex: if Google cannot crawl the page it never sees the noindex, and the page can still show up in results.

Questions

Does "Discourage search engines" block visitors?

No. People can still visit every page. It only asks search engines not to list them.

How long until my site shows up after I untick it?

Google has to recrawl the pages first. For an established site that can be a few days; a new site can take a few weeks. Requesting indexing for your key pages and resubmitting the sitemap is the fastest legitimate route.

I unticked it but Search Console still says noindex.

Search Console reports what Google saw on its last visit. Use URL Inspection's "Test live URL" to see the page as it is now; if the live test is clean, request indexing and wait for the next crawl.

Day 7 of 20. The rest of the findings: what we found checking 400 small-business websites, and nine reasons a site does not show up on Google.

Want to know what AI crawlers see on your site? Our free AI visibility checker runs these checks against any page in about 10 seconds, no signup. If it finds something you'd rather hand over, our AI SEO service fixes it.

More from the blog

Get a free homepage mockup

Tell us about your business and we'll design a custom homepage concept within 48 hours — free, no strings attached.