Meta directives provide guidance to search engines about how to crawl and index information on a particular website that they discover. When search engine bots find these instructions, meta directives act as solid suggestions on how to index or crawl those pages.
Noindex and nofollow tags are two directives that many websites employ. Whether you want specific pages or your website to not be crawled by Google, or you’re looking to preserve crawl budget to improve your site’s keyword rankings, read our latest guide that walks through common meta directives and best practices for using them.
Noindex tags are directives that tell search engines to not index a web page and not include it on the search engine's result pages.
Nofollow tags tell search engine spiders to not follow the links on a web page that are marked with nofollow tags. It advises search engines how to crawl a site and what pages should be indexed to help preserve what is known as crawl budget.
Looking to learn more about search engine optimization? Read our SEO beginner’s guide for everything you need to know about SEO and how to drive business results through search engines like Google and Bing.
Here’s a list of common robots meta directives you can use for your site pages:
Below are a few different page types that can benefit from the use of noindex or nofollow meta tags.
With many ecommerce sites, you may have hundreds of thousands, or even millions, of product pages listing your specific inventory. However, these products may provide little value to a user on the search results, and can cause bloat as crawl bots struggle to discover and index these millions of pages on your site.
This can cause these bots to not find or index your priority pages that you want indexed. A common solution to help with preserving crawl budget is to noindex, nofollow individual product listing pages.
By default, WordPress will create category pages listing all articles that use the tags that you set within the CMS. However, category tag pages provide little value to users or search engines, especially if you have actual categories in place for your website. It’s best to noindex, nofollow these types of pages.
The bulk of admin or login pages are not meant to be on Google. Sometimes though, they are indexed. To fix this, you can apply a noindex to it to keep it from being displayed in the search results.
Exceptions are the login pages that, like Dropbox or related sites, support a community. Just ask yourself if, if you were not with your business, would you Google one of your login sites. If not, it is generally fair to assume that these login pages don't need to be indexed by Google. Fortunately, if you are running WordPress, you're secure as the CMS immediately noindexes the login page of your blog.
One of the last places Google wants to drive visitors are internal search results. Internal search results also create unique pages with every query a user enters into your internal search engine, which harms your website’s crawl budget and can prevent important pages from being discovered.
To prevent this, you’ll want to noindex your internal search results pages.
A thank you page serves no other purpose than to thank your first-time commenter/customer/newsletter subscriber. Most thank you pages are typically thin in content, with little value for users looking for specific information on Google. Because of the thin content and low SEO value, these types of pages should be noindexed.
Below are a few best practices to follow when implementing noindex or nofollow meta tags on your website:
This is a given, but if you want a page to rank or be shown on Google, don’t include a noindex tag on them, as this will prevent those pages from being indexed and displayed to users on search engines.
Adding noindex tags in the robots.txt file of your website once was a supported practice. However, in September 2019 Google announced that this isn’t a standard practice, and its search engine may choose to ignore your robots.txt directives that include noindex or nofollow tags.
If you want pages to not be indexed, or followed, instead of creating a directive in your robots.txt file, you want to include it in the <head> section of your page, or just before the <body> section. This will ensure that the directive is read and followed by search crawlers.
After a period of time has passed for a web page that has a noindex tag directive implemented, Google will eventually stop crawling that page, treating it as a nofollow in its archive to preserve crawl budget.
If you have sponsored or paid links on your website, you may want to add a directive informing Google not to follow, or pass equity, to those links. You can use HTML rel attributes to mark up these individual links with the nofollow tag directive, instead of adding noindex or nofollow tags for the entire web page.