Recently I have written about Google Search Console setup which is required for every website. We can manage our website coverage, URL’s indexing, etc from search console.
Before proceeding I would recommend you to read about Google Search Console configuration.
The robots.txt file contains some lines of code which will help the search engine to crawl, index the web pages present on your website. Search engines first read the robot.txt file of your website and accordingly start indexing URLs.
We can define in a robots.txt file which pages to crawl or not to crawl. Along with it, you can easily block or restrict the page from web crawlers to index so that it will not be seen in the search result.
Here is the sample of a robots.txt file. You can change the robot.txt file as per your requirement.
User-agent: *
Disallow: /wp-admin/
Disallow: /recommended/
Disallow: /tag/
Sitemap: https://scrollbucks.com/post-sitemap.xml
Sitemap: https://scrollbucks.com/page-sitemap.xmlThe asterisk after user-agent means that robots.txt file applies to all the web robots.
Disallow means tells the web robots that not to visit and crawl the pages.
The sitemap tells the web robot that where is your all web pages present and crawl these web pages.
Add Robots.txt in BlogSpot blog
Login to your BlogSpot blog and navigate to Setting -> Search Preferences
You will see the Crawlers and Indexing section enable the custom robots.txt and paste the above code. Make sure that you have updated all the website details in the robots.txt file.
Here are the few tutorials available for newbie bloggers.
- How to configure Google Search Console for the website?
- How to add Google Analytics filters to sort the traffic?
- How to add Google Analytics to BlogSpot blog?
- How to check the page rank with Google Analytics?
Amit Kharbade
Latest posts by Amit Kharbade (see all)
- How to Reduce Form Abandonment in WordPress? - July 28, 2019
- Scroll Depth Tracking Using Google Analytics by MonsterInsights - July 20, 2019
- How To Fix Email Issues in WordPress With WP Mail SMTP Pro Plugin? - July 20, 2019