Page 1 of 2 12 LastLast
Results 1 to 20 of 26

What is robots.txt?

This is a discussion on What is robots.txt? within the Search Engine Optimization forums, part of the Internet Marketing category; Please tall me answers friend What is robots.txt ?...

  1. #1
    Newbie
    Join Date
    Sep 2015
    Posts
    1

    Default What is robots.txt?

    Please tall me answers friend

    What is robots.txt ?

  2. #2
    Senior Member
    Join Date
    Sep 2015
    Posts
    142

    Default Re: What is robots.txt?

    Robots.txt is a kind of coding which is upload in root file which is telling search engine that not to crawl or read specific part of website.
    Top Android apps development services | Dotstudioapps.com

  3. #3
    Senior Member RH-Calvin's Avatar
    Join Date
    Jun 2013
    Posts
    3,427

    Default Re: What is robots.txt?

    Robots.txt is a text file that is inserted into your website instructing search engine crawlers to crawl and disallow certain webpages. They are helpful to control the robots on your website pages.
    Cheap VPS | $1 VPS Hosting | Windows with Remote Desktop
    Cheap Dedicated Server | $29 Dedicated Server with Free IPMI Setup

  4. #4
    Member
    Join Date
    Aug 2015
    Posts
    43

    Default Re: What is robots.txt?

    Robots.txt is a text document file which is mainly used for the purpose of preventing some secured web pages from search engine bots/robots/spiders as well as from users.

  5. #5
    Senior Member
    Join Date
    Apr 2015
    Posts
    236

    Default Re: What is robots.txt?

    Robots.txt is a simple text file which is use to tell a message to search engine that specific parts of web page are private and not to crawl.

  6. #6
    Senior Member
    Join Date
    Dec 2013
    Posts
    1,185

    Default Re: What is robots.txt?

    It is an HTML tag placed on the source of a web page which redirects search engine spiders which files to crawl on or not.

  7. #7
    Member
    Join Date
    Oct 2015
    Location
    New York City
    Posts
    61

    Default Re: What is robots.txt?

    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots.

  8. #8
    Senior Member dennis123's Avatar
    Join Date
    Apr 2013
    Location
    Bangalore
    Posts
    2,568

    Default Re: What is robots.txt?

    The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

  9. #9
    Senior Member
    Join Date
    Aug 2014
    Posts
    195

    Default Re: What is robots.txt?

    Robot.txt is an option which we can use in webmaster tool it is useful for to say the search engine to index any link or not to index a link.

  10. #10
    Senior Member
    Join Date
    Aug 2014
    Location
    Singapore
    Posts
    116

    Default Re: What is robots.txt?

    Basically robots.txt file tell the Google spider how to crawl the webpage on their websites by allowing and disallow.

  11. #11
    Senior Member Nexevo Technologies's Avatar
    Join Date
    Aug 2015
    Location
    Bangalore
    Posts
    852

    Default Re: What is robots.txt?

    robots txt file is the used to intimate the website information in search engine crawlers..and also instruct the how to crawl my website...

  12. #12
    Junior Member
    Join Date
    Aug 2015
    Location
    United States
    Posts
    17

    Default Re: What is robots.txt?

    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots.

  13. #13
    Member
    Join Date
    Jan 2015
    Location
    USA
    Posts
    35

    Default Re: What is robots.txt?

    Robots.txt is used to tell the Google spider to which page has to crawl or not by allowing and disallowing.

  14. #14
    Senior Member
    Join Date
    Mar 2014
    Location
    Canada
    Posts
    415

    Default Re: What is robots.txt?

    Robots.txt is a text file located in the siteís root directory.there you put on your site to tell search robots which pages you would like them not to visit.

  15. #15
    Senior Member Brucey's Avatar
    Join Date
    Sep 2015
    Location
    Dallas, TX
    Posts
    128

    Default Re: What is robots.txt?

    robots.txt is the file that help your website disallow some kinds of bot.

  16. #16
    Member
    Join Date
    Oct 2015
    Posts
    39

    Default Re: What is robots.txt?

    It's a file with which you can tell Google & other SE's not to visit some pages of your website! ;)

  17. #17
    Member
    Join Date
    Oct 2015
    Location
    Banagladesh
    Posts
    75

    Default Re: What is robots.txt?

    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naÔve to rely on robots.txt to protect it from being indexed and displayed in search results.

  18. #18
    Member
    Join Date
    Apr 2015
    Posts
    32

    Default Re: What is robots.txt?

    robots.txt is the file which is allow to index a file of no index.

  19. #19
    Senior Member cybnetics's Avatar
    Join Date
    Nov 2015
    Location
    Delhi
    Posts
    105

    Default Re: What is robots.txt?

    Robots.txt files inform search engine spiders how to interact with indexing your content.

  20. #20
    Senior Member
    Join Date
    Aug 2015
    Posts
    113

    Default Re: What is robots.txt?

    What are the benefits of the robots.txt from the view of SEO.

Page 1 of 2 12 LastLast