This is a discussion on what is robots.txt file ? within the Search Engine Optimization forums, part of the Internet Marketing category; Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The ...
Re: what is robots.txt file ?
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.
It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:
User-agent: *
Disallow: /
The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
There are two important considerations when using /robots.txt:
*robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
*the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.
Re: what is robots.txt file ?
"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.
Re: what is robots.txt file ?
Robots.txt file used to allow or disallow robots or crawlers.to get indexing or no follow your page or website in search engine.
Re: what is robots.txt file ?
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.
Re: what is robots.txt file ?
When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.
Re: what is robots.txt file ?
The file is specially for crawler to tell it which file you would index and which would not index.
Posting Permissions
LinkBack URL
About LinkBacks
Bookmarks