Results 1 to 14 of 14

What is robots.txt?

This is a discussion on What is robots.txt? within the Advertising & Marketing forums, part of the Internet Marketing category; Hello, What is robots.txt?...

  1. #1
    Member
    Join Date
    Dec 2017
    Posts
    97

    Default What is robots.txt?

    Hello,

    What is robots.txt?

  2. #2
    OPH
    OPH is offline
    Senior Member
    Join Date
    Sep 2017
    Location
    DELHI, INDIA
    Posts
    391

    Default Re: What is robots.txt?

    robots.txt is a text file containing two lines of text to restrict the Search Engine Crawling bots for crawling the website or the Webpages. This file placed in the root folder of the website.

  3. #3
    Senior Member
    Join Date
    Mar 2018
    Location
    Gurdaspur
    Posts
    233

    Default Re: What is robots.txt?

    thanks for sharing with us

  4. #4
    Senior Member RH-Calvin's Avatar
    Join Date
    Jun 2013
    Posts
    2,893

    Default Re: What is robots.txt?

    Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
    Cheap VPS | $1 VPS Hosting | Windows with Remote Desktop
    Cheap Dedicated Server | $29 Dedicated Server with Free IPMI Setup

  5. #5
    Senior Member
    Join Date
    Nov 2016
    Posts
    150

    Default Re: What is robots.txt?

    Robots.txt is a text file to restrict search engine bots to crawl/not to crawl particular web pages like admin...

  6. #6
    Senior Member
    Join Date
    Nov 2017
    Posts
    275

    Default Re: What is robots.txt?

    thanks for sharing with us

  7. #7
    Senior Member
    Join Date
    Jan 2012
    Posts
    1,664

    Default Re: What is robots.txt?

    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too na´ve to rely on robots.txt to protect it from being indexed and displayed in search results.

  8. #8
    Senior Member
    Join Date
    Apr 2017
    Location
    india
    Posts
    452

    Default Re: What is robots.txt?

    thanks for sharing with us

  9. #9
    Senior Member
    Join Date
    Mar 2018
    Posts
    300

    Default Re: What is robots.txt?

    thanks for sharing with us

  10. #10
    Senior Member
    Join Date
    Feb 2018
    Location
    Haryana
    Posts
    224

    Default Re: What is robots.txt?

    thanks for sharing with us

  11. #11
    Senior Member
    Join Date
    May 2017
    Location
    jalandhar
    Posts
    401

    Default Re: What is robots.txt?

    thanks for sharing with us

  12. #12
    Senior Member
    Join Date
    Feb 2016
    Location
    USA
    Posts
    113

    Default Re: What is robots.txt?

    This is a great post.

  13. #13
    Member
    Join Date
    Oct 2017
    Location
    Delhi, India
    Posts
    39

    Default Re: What is robots.txt?

    In Simple Words Robots.txt is a file which controls the accessibility of crawlers.

  14. #14
    Senior Member Amit Tiwari's Avatar
    Join Date
    Feb 2017
    Posts
    224

    Default Re: What is robots.txt?

    robots.txt is a text file containing two lines of text to restrict the Search Engine Crawling bots for crawling the website or the Webpages.