Page 2 of 2 FirstFirst 12
Results 21 to 26 of 26

What is robots.txt?

This is a discussion on What is robots.txt? within the Search Engine Optimization forums, part of the Internet Marketing category; Robots.txt is a file which directs search engine spiders not to crawl certain webpages....

  1. #21
    Senior Member
    Join Date
    Aug 2015
    Location
    india
    Posts
    168

    Default Re: What is robots.txt?

    Robots.txt is a file which directs search engine spiders not to crawl certain webpages.

  2. #22
    Junior Member
    Join Date
    Nov 2015
    Posts
    11

    Default Re: What is robots.txt?

    It's the Search Engine Spider to index your website.

  3. #23
    Senior Member
    Join Date
    Jul 2012
    Location
    india
    Posts
    269

    Default Re: What is robots.txt?

    A robots.txt file is a file at the root of your site that indicates those parts of your site you don't want accessed by search engine crawlers.

  4. #24
    Member
    Join Date
    Nov 2015
    Location
    India
    Posts
    61

    Default Re: What is robots.txt?

    hi,

    Robots.txt is a text file that provide information to search engine for crawling or not.

  5. #25
    Senior Member
    Join Date
    Jan 2012
    Posts
    2,044

    Default Re: What is robots.txt?

    Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website.

  6. #26
    Senior Member
    Join Date
    Jul 2017
    Posts
    121

    Default Re: What is robots.txt?

    Robots.txt file is a text file for restricting bots (robots, search engine crawlers ) from a website or certain pages on the website. Using a robots.txt file and with a disallow direction, we can restrict bots or search engine crawling program from websites and or from certain folders and files.

Page 2 of 2 FirstFirst 12