How to add custom robots.txt file in BlogSpot blog?

Do you know why we use the robots.txt file on our blog? What is the significance of robots.txt file?

In this tutorial, we will cover all the concepts about the robots.txt file.

The robots.txt file contains some lines of code which will help the search engine to crawl, index the web pages present on your website. We can define in a robots.txt file which pages to crawl or not to crawl. Along with it, you can easily block or restrict the page from web crawlers to index so that it will not be seen in the search result.

Search engines check your robot.txt (it should be present in your blog’s root folder or you can set it in the Google Search Console). According to the robots.txt file search engine start to index the web pages.

The format of Robot.txt file

User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: https://www.scrollbucks.com/root/sitemap/sitemap.xml

Here I will tell you the meaning of each term.

User-agent: Mediapartners-Google
This code is just for Google Adsense helpful for serving the related ads on your page. If you are an Adsense user then its useful for you. Keep it as it is.

User-agent:*
In the default setting our blogs links are restricted to indexed by search crawlers that mean the web crawlers will not index our label pages link because of this code

Disallow:/search

that means the links having the keyword search just after the domain name will be ignored

https://www.scrollbucks.com/search/label/affiliate.

This above is an example of the link of label affiliate if we remove Disallow:/seach from the code then crawlers will access your entire blog pages. You can disallow the particular page to be indexed by the crawlers by simply putting the URL of that page like this

Disallow:/www.scrollbucks.com/About me

In this case, the search engine will ignore the about me page of my website.

Custom Robot.txt in BlogSpot blog

1. Login to BlogSpot blog, open the dashboard.
2. Go to Setting>Search Preferences>Crawlers & Indexing.
3. Click the edit tab in front of robot.txt
4. Paste your robot.txt code here and click on the save.

Here is the sample robots.txt file you can refer

Sitemap: https://www.scrollbucks.com/page-sitemap.xml
Sitemap: https://www.scrollbucks.com/post-sitemap.xml
User-agent: *
Disallow: /?s=*
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Allow: /wp-admin/images/

Once you prepared the robots.txt file, you should have updated it in your Google Search Console. You can update it by going here-

Google Search Console>Crawl>robots.txt tester

Its very important for you to define the right rule in the robots.txt file otherwise search engine will not able to find your blog and ultimately it will affect your blog traffic.

If you have any concerns about the robots.txt file then let us know, we try to listen all the queries came into our bucket.

• Hidden features of Google search engine
• How to configure Google Search Console for the website?
• How to add Google Analytics filters to sort the traffic?
• How to add Google Analytics to BlogSpot blog?
• How to check the page rank with Google Analytics?