How to use Robots.txt Effectively

Using Robots.txt in a very effective manner can procure a lot of SEO and organic traffic benefits. Playing fouls in the use of robots text will result in your site/blog getting out of search engine's index(database). Successful on site optimization lies in the efficient use of robots and sitemap file to index a blog or website on search engine.

How to Use Robots.txt

What is Robots.txt?

About 90% of SEO (experts) use robots and word with it to rank their website more than it really deserve. It is the first file in any hosting folder that a search engine's bot look for. It directs the bot which page is to be crawled and which has to be skipped while crawling. In lack of efficient use of robots text, search engine crawler with index every web page in the hosting folder and show every page in search results. It could harm the SEO score of your website if multiple pages have same content.

How to Use it?

It is very easy if you know a little fact that this file consists directions for every search engine bot. For example, if you want to give directions to google's bot,  you can do it separately and can use separate index criteria for every search engine in this way.

How to direct a specific Search Engine Crawler (bot)?

To add any rule for a specific bot, you can put the name of the bot in the place of botname (in below snippet).

User-agent: botname
Disallow: /ht-doc

Create a criteria for all the search engine 

You can direct every search engine with a single line as well. just replace botname with "*" in the above snippet and put any rule there.

How to prohibit every crawler to visit a specific page?

Below is the robots text to ask a crawler not to index and crawl a page.
For example, You domain is www.exampledomain.com and the page you want to disallow bots to crawl is www.exampledomain.com/page.php.

User-agent: *
Disallow: /page.php

Using Robots in Blogger 

Your blog at blogger gives you the leverage to use custom robots.txt but using it badly can harm the whole on site optimization you have done before and hence I explained the whole concept of robots above. To use robots in blogger follow steps below:

Goto:  Settings >> Search preferences

Click use custom robots.txt and type the rule you want and click save.

You are all done.

Best of Luck and Happy blogging friends!