what is robots.txt file

 Robot.txt is a text file. In this file, we can give the instruction google bots how to crawl your website. It's very important to website SEO. Different search engines have different bots. We can allow specific files and webpage through the robots.txt file. We can also spread malware help of this.

Syntax of robots.txt file

First, look at this code. Then we can discuss what is meant by each code.

User-agent: *
Disallow: /search
Allow: /
Disallow: /
Sitemap: https://learnwithcertifiedteacher.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500

Let's discuss this code. 

user-agent:  -----> we gave the instructions to the search engine " * " mean all search engines access my sites.

Disallow: ------> If you went to disable any part of sites and any links.

Allow: ------> Like as Disallow this code mean we can allow some files.

At Last Sitemap: ----> mean which website page link which page.

Where robots.txt file placed?

On the Blogger website robots.txt file placed in the setting area. On the setting area scroll down you can see the custom robots.txt file you can write your code here.

And the custom websites like WordPress and HTML  sites we can paste you Control Panel public Directory where your website pasted.

How could you make your own robots.txt file?

If your programming is strong and you are familiar with the syntax of the robots.txt file. you can write your own robots.txt file. If you don't know what is the syntax of the robots.txt file you can read my second paragraph. If you don't write code don't worry I will give you a trick where you can generate a robots.txt file.

https://www.labnol.org/blogger/sitemap/ here you can generate your robots.txt file free.

Different bots of search engines:

There are many search engines on the Internet. They have their own bots who can crawl websites. In this section we can discuss some search engines and their bots.

1)    Google

Bots Name: -     Googlebot's

2)    Bing

Bot Name: -     Bing bots

3)    Yahoo

          Bot Name: -     Slurp Bot

3)    DuckDuckGo

          Bot Name: -     DuckDuckbot

4)    Baiduspider

          Bot Name: -     Baiduspiderbot

5)    Yandex

          Bot Name: -     Yandexbot

6)    Exalead

           Bot Name: -     Exabot

7)    Facebook

            Bot Name: -     Facebook external hit

8)    Alexa

            Bot Name: -     
Alexa Crawler