Results 1 to 12 of 12

What is robots.txt?

This is a discussion on What is robots.txt? within the Search Engine Optimization forums, part of the Internet Marketing category; Read More: Best IAS Academy in Chennai|IAS Academy in Chennai officersiasacademy...

  1. #1
    Newbie
    Join Date
    Mar 2015
    Posts
    1

    Default What is robots.txt?

    Read More:
    Best IAS Academy in Chennai|IAS Academy in Chennai
    officersiasacademy

  2. #2
    Member
    Join Date
    Mar 2015
    Posts
    81

    Default Re: What is robots.txt?

    Robots.txt file plays a major role in search engine ranking.Robots.txt file helps
    search engine robots to direct which part to crawl and which part to avoid.
    When bots or spider of Search Engine comes to your site and wants to index
    your site, they follow Robots.txt file first,And it follows this files direction
    for index or no index any page of your website.

  3. #3
    Member cbdmoversperth's Avatar
    Join Date
    Feb 2015
    Location
    Perth (Australia)
    Posts
    53

    Default Re: What is robots.txt?

    Robot.txt is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want.

  4. #4
    Junior Member
    Join Date
    Feb 2015
    Location
    California
    Posts
    7

    Default Re: What is robots.txt?

    The robots.txt file is used to provide instructions about the Web site to Web robots and spiders. Web authors can use robots.txt to keep cooperating Web robots from accessing all or parts of a Web site that you want to keep private.

  5. #5
    Senior Member
    Join Date
    Dec 2013
    Posts
    1,185

    Default Re: What is robots.txt?

    It is an HTML tag placed on the source of a web page which redirects search engine spiders which files to crawl on or not.

    Robots.txt is a text file present in the root directory of a website.

  6. #6
    Junior Member mandymovingandpackin's Avatar
    Join Date
    Feb 2015
    Location
    India
    Posts
    13

    Default Re: What is robots.txt?

    Robots.txt is a text file we use to put on our website to search engine which pages we would like user not to visit. It is also used for security porpose and we can costumise it according to our need.

  7. #7
    Senior Member
    Join Date
    Nov 2014
    Posts
    325

    Default Re: What is robots.txt?

    Robots.txt is used to write the code for search engines to secure or crawling purpose .

  8. #8
    Senior Member
    Join Date
    Feb 2015
    Posts
    217

    Default Re: What is robots.txt?

    The robots.txt file is a set of instructions for visiting robots from search engines that index the content of your web site pages.

  9. #9
    Member
    Join Date
    Feb 2015
    Posts
    85

    Default Re: What is robots.txt?

    Therobotsexclusion standard, also known as the robotsexclusion protocol or robots.txt
    protocol, is a standard used by websites to communicate with web crawlers and other web robots

  10. #10
    Senior Member
    Join Date
    Jan 2012
    Posts
    1,804

    Default Re: What is robots.txt?

    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter.

  11. #11
    Senior Member
    Join Date
    Feb 2017
    Posts
    211

    Default Re: What is robots.txt?

    Robots.txt file is at the root of the website that involves sectors of your website you don’t want to be attained by search engine crawlers. Webmasters use a robot.txt file to instruct the search engine robots on how to crawl & index the web pages.
    HostechSupport
    Best Linux/ Windows Server Management
    24x7 Skype: hostechsupportindia

  12. #12
    Senior Member RH-Calvin's Avatar
    Join Date
    Jun 2013
    Posts
    3,122

    Default Re: What is robots.txt?

    Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
    Cheap VPS | $1 VPS Hosting | Windows with Remote Desktop
    Cheap Dedicated Server | $29 Dedicated Server with Free IPMI Setup