what is robots.txt file ?

It is necessary especially when you want to deindex some pages o

Robots.txt is a text (not html) file you put on your root directory to tell search robots which files to ignore (or alternatively) which files to crawl. It also helps Search Engines to locate the Sitemap of the website and hence crawl the entire website in depth... helping in your rankings and traffic.

Creating a robots.txt file is a matter of just a few seconds.
- Right click on your desktop
- Choose new - text document (you can use notepad too)
- Insert instructions to crawling bots.
- Save the document as robots.txt

You may now upload this file to your top/ root directory
e.g. http://yourdomainname/robots.txt

Robots.txt file helps you to control search engine bots in indexing the pages of your site. You can prevent some of the pages being crawled by search engine bots.

If you want all spiders to crawl your site. Add this coding in a notepad and name it as "robots.txt" and upload it in your root folder.

User-agent: *
Allow:/

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

Originally Posted by C.Rebecca

Robots.txt is a text (not html) file you put on your root directory to tell search robots which files to ignore (or alternatively) which files to crawl. It also helps Search Engines to locate the Sitemap of the website and hence crawl the entire website in depth... helping in your rankings and traffic.

Creating a robots.txt file is a matter of just a few seconds.
- Right click on your desktop
- Choose new - text document (you can use notepad too)
- Insert instructions to crawling bots.
- Save the document as robots.txt

You may now upload this file to your top/ root directory
e.g. http://yourdomainname/robots.txt

Thanks Rebecca for the great help through these easy words.

Normally Robots.txt file is to inform the search engine robots which pages of your site to be crawled and indexed. There are many files or folders i.e. admin folders, cgi-bin, image folder, which are not relevant to the search engines. Robots.txt helps spiders what is useful and public for sharing in the search engine indexes and what is not. It saves the time of Spiders.

robots.txt is a file in which we place the code of that url's that we dont want to crawl.

Robots.txt is the must for any website, where the search engine can crawl your website easily.

It is used to crawl the website in search engine.