How To Add Custom robots.txt File in BlogSpot Blog?

by

In my recent article, I have written about the Google Search Console setup. It helps us to manage our website coverage, URL indexing, etc.

The robots.txt file contains some lines of code which will help the search engine to crawl, index the web pages present on your website. Search engines first read the robot.txt file of your website and accordingly start indexing URLs.

We can define in a robots.txt file which pages to crawl or not to crawl. Along with it, you can easily block or restrict the page from web crawlers to index so that it will not be seen in the search result.

Here is a sample of a robots.txt file. You can change the robot.txt file as per your requirement.

User-agent:  *
Disallow: /wp-admin/
Disallow: /recommended/
Disallow: /tag/

Sitemap: https://scrollbucks.com/post-sitemap.xml
Sitemap: https://scrollbucks.com/page-sitemap.xml

The asterisk after user-agent means that the robots.txt file applies to all the web robots.

Disallow means tells the web robots not to visit and crawl the pages.

The sitemap tells the web robot that where is your all web pages present and crawl these web pages.

Add Robots.txt in BlogSpot blog

Login to your BlogSpot blog and navigate to Setting -> Search Preferences

You will see the Crawlers and Indexing section enable the custom robots.txt and paste the above code. Make sure that you have updated all the website details in the robots.txt file.

You’ll love to read

The following two tabs change content below.
My Twitter profileMy Facebook profileMy LinkedIn profileMy Instagram profile

Amit Kharbade

Editor & Writer at ScrollBucks
Software engineer by profession and Blogger by passion. Amit has been contributing to ScrollBucks since 2014. He is writing about Blogging and his practical blogging experiences. You can connect with him at [email protected]

Leave a Comment