Can I use robots.txt to optimize Googlebot's crawl?

Thaks Matt for sharing the video. Its awesome.

Thaks Matt for sharing the video. Its awesome.

useful video,thank you so much,mmmm

If a site owner wishes to give instructions to web robots he must place a text file called robots.txt in the root of the web site hierarchy (e.g. www.example.com/robots.txt). This text file should contain the instructions in a specific format (see examples below). Robots that choose to follow the instructions try to fetch this file and read the instructions before fetching any other file from the web site. If this file doesn't exist web robots assume that the web owner wishes to provide no specific instructions.

A robots.txt file on a website will function as a request that specified robots ignore specified files or directories in their search. This might be, for example, out of a preference for privacy from search engine results, or the belief that the content of the selected directories might be misleading or irrelevant to the categorization of the site as a whole, or out of a desire that an application only operate on certain data.

For websites with multiple subdomains, each subdomain must have its own robots.txt file. If example.com had a robots.txt file but a.example.com did not, the rules that would apply for example.com would not apply to a.example.co

talking about search engine crawlers and robots here. Every day hundreds of them go out and scour the web, whether it's Google trying to index the entire web, or a spam bot collecting any email address it could find for less than honorable intentions. As site owners, what little control we have over what robots are allowed to do when they visit our sites exist in a magical little file called "robots.txt."

"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.

A robots .txt is a text file that has to be placed in the root folder of your web server (where you place index page of your website). You can simple create this file in a notepad. It tells various search engine bots that which part of website should not be crawled or should not be indexed. By using this we can instruct bots to prevent our website from being crawled or we can instruct them that they should not crawl or index certain areas of the website. Even we can use same robots .txt to give different instructions to different bots.


Even if you don't want to protect any area of your website from indexing or crawling, still you should use robots .txt as it can act as a open invitation for search engines to crawl your complete website.


There can be several scenarios in which you might be interested in blocking Search Engine Bots from crawling certain parts of your website.


For example


1) Protecting your administration panel of your website.
2) Protecting your under construction pages from getting indexed in search engines.
3) Protecting directory that you don't want to be indexed like cgi-bin
4) Protecting pages that have email addresses as they can be used by spammers if got indexed in search engines.
Reasons can be various but the solution is same, that is using robots.txt now let's start writing in robots.txt


Basic syntax of robots.txt is
User-Agent: [Spider or Bot name]
Disallow: [Directory or File Name]
You can repeat these lines for blocking different directories or giving different instructions to different spiders. Let's get in few examples that will make it more clear.

yes you can use "robot.txt" in meta tags.

Hi

I have a blog on blogger.com, and i want to use robots.txt on my blog, can i do this, if yes then how, please show me the way.

Thanks.