How Find And Edit The Robots.txt File On Your Website

Brandon LazovicMay 8, 2021

Looking to find the robots.txt file on your website? You’re in luck - this article will cover where to find your robots.txt file, both on the frontend and backend. 

What Is A Robots.txt File? 

The robots.txt file is a text-based document that tells search engines what they can and cannot crawl on your site. For example, you may want to block the search engine from indexing certain pages or folders of your website. You can also use this file to tell the robot: "Please crawl all my pages." 

Where To Find The Robots.txt File On Your Website’s Frontend

Finding the robots.txt file on the frontend of your website is simple. 

All you need to do is type “ /robots.txt “ at the end of your root domain to pull up the file. An example would be https://brandonlazovic.com/robots.txt.

Where To Find The Robots.txt File On Your Website’s Backend

Finding the robots.txt file on the backend of your website to directly change rules is a little trickier compared to discovering it on the frontend. Luckily we’ll walk through where to find it on the backend for three of the most popular CMSs - WordPress, Squarespace, and Wix. 

WordPress

In most cases, you’ll need a WordPress SEO plugin in order to change the robots.txt file on the backend of your site. 

  1. One plugin is All in One SEO; simply go to All In One SEO >> Tools to directly edit your robots.txt file. 
  2. You can also use Rank Math; simply go into the sidebar, click on Rank Math >> General Settings, and then edit the file. 
  3. Lastly, you can use Yoast SEO; simply go into Yoast SEO plugin > Tools > File editor to edit your robots.txt file. 

Squarespace

While you don’t have direct access to make updates to your robots.txt on squarespace, they do a relatively decent job creating one on behalf of a webmaster. However, if you need to make custom edits, I would reach out to Squarespace’s customer support for more assistance in this regard. 

Wix

Wix is relatively painless to update for robots.txt files. Simply go into your site’s dashboard >> Marketing & SEO > SEO Tools > Robots.txt File Editor >  View File. 

Other Website CMS

In other cases, you may have your robots.txt file set up via your website hosting as a FTP client. Simply navigate into the FTP client, find the robots.txt file, and make edits as needed inside your website’s root folder.

Be sure to check out our latest search engine optimization guide to learn about the best practices of SEO for your website.

Categories:
Digital Marketing
Social Media
Digital PR
Photography
SEO
Analytics
Tool Reviews
Local SEO                                                          
Contact: 
brandon.lazovic@gmail.com                           

Sitemap           Privacy            Editorial Guidelines
Copyright © 2021 Brandon Lazovic LLC. All rights reserved.
linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram