Do you know why we use the robots.txt file on our blog? What is the significance of robots.txt file?
In this tutorial, we will cover all the concepts about the robots.txt file.
The robots.txt file contains some lines of code which will help the search engine to crawl, index the web pages present on your website. We can define in a robots.txt file which pages to crawl or not to crawl. Along with it, you can easily block or restrict the page from web crawlers to index so that it will not be seen in the search result.
Search engines check your robot.txt (it should be present in your blog’s root folder or you can set it in the Google Search Console). According to the robots.txt file search engine start to index the web pages.
The format of Robot.txt file
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: https://www.scrollbucks.com/root/sitemap/sitemap.xml
Here I will tell you the meaning of each term.
User-agent: Mediapartners-Google
This code is just for Google Adsense helpful for serving the related ads on your page. If you are an Adsense user then its useful for you. Keep it as it is.
User-agent:*
In the default setting our blogs links are restricted to indexed by search crawlers that mean the web crawlers will not index our label pages link because of this code
Disallow:/search
that means the links having the keyword search just after the domain name will be ignored
https://www.scrollbucks.com/search/label/affiliate.
This above is an example of the link of label affiliate if we remove Disallow:/seach from the code then crawlers will access your entire blog pages. You can disallow the particular page to be indexed by the crawlers by simply putting the URL of that page like this
Disallow:/www.scrollbucks.com/About me
In this case, the search engine will ignore the about me page of my website.
Custom Robot.txt in BlogSpot blog
- Login to BlogSpot blog, open the dashboard.
- Go to Setting>Search Preferences>Crawlers & Indexing.
- Click the edit tab in front of robot.txt
- Paste your robot.txt code here and click on the save.
Here is the sample robots.txt file you can refer
Sitemap: https://www.scrollbucks.com/page-sitemap.xml Sitemap: https://www.scrollbucks.com/post-sitemap.xml User-agent: * Disallow: /?s=* Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Allow: /wp-admin/images/
Once you prepared the robots.txt file, you should have updated it in your Google Search Console. You can update it by going here-
Google Search Console>Crawl>robots.txt tester
Its very important for you to define the right rule in the robots.txt file otherwise search engine will not able to find your blog and ultimately it will affect your blog traffic.
If you have any concerns about the robots.txt file then let us know, we try to listen all the queries came into our bucket.
Amit
Latest posts by Amit (see all)
- How to change WordPress login URL using WPS hide login? - June 30, 2018
- How to configure web push notifications for WordPress blog using OneSignal? - June 29, 2018
- How to transfer domain from GoDaddy to Google Domains [no downtime]? - March 25, 2018
Leave a Reply