Page 1 of 2 12 LastLast
Results 1 to 20 of 32

What is the use of robots.txt file?

This is a discussion on What is the use of robots.txt file? within the Web Design forums, part of the Web Designing & Development category; Can you tell me the answer...

  1. #1
    Member
    Join Date
    Oct 2014
    Location
    US
    Posts
    55

    Default What is the use of robots.txt file?

    Can you tell me the answer

  2. #2
    Senior Member 3mdigital's Avatar
    Join Date
    Jul 2014
    Location
    Bangalore
    Posts
    141

    Default Re: What is the use of robots.txt file?

    robots.txt file is file to indicate web page or a web site index/crowel or not to search engines

  3. #3
    Senior Member
    Join Date
    Feb 2014
    Location
    India
    Posts
    126

    Default Re: What is the use of robots.txt file?

    Have you search on Google? one time you should do it. Check out the proper definition of Robots.txt.

    https://www.google.com/#q=define+robots.txt

  4. #4
    Senior Member
    Join Date
    Oct 2014
    Location
    USA, New York
    Posts
    105

    Default Re: What is the use of robots.txt file?

    Robot.txt file is implemented and is used so that a web page which is used in a website design service couldn't be cached in search engines so that your existing website couldn't be indexed.

  5. #5
    Member
    Join Date
    Oct 2014
    Posts
    73

    Default Re: What is the use of robots.txt file?

    At the OP, do you plan to use it or what?

  6. #6
    Senior Member
    Join Date
    Feb 2014
    Location
    India
    Posts
    126

    Default Re: What is the use of robots.txt file?

    Quote Originally Posted by apdesignsonline View Post
    Can you tell me the answer
    Proper definition of Robots.txt

    http://www.google.co.in/?gfe_rd=cr&e...fine+Robot.txt

  7. #7
    Member
    Join Date
    Oct 2014
    Location
    US
    Posts
    55

    Default Re: What is the use of robots.txt file?

    Thank you friends for the information

    Website Design Louisville KY | Website Design Cincinnati OH

  8. #8
    Senior Member
    Join Date
    Oct 2014
    Posts
    187

    Default Re: What is the use of robots.txt file?

    The content of this file tells search engine crawlers how they should visit your site. If there are files and directories you do not want indexed by search engines, you can use a robots.txt file to define where the robots should not go. These files are very simple text files that are placed on your web server.

  9. #9
    Junior Member
    Join Date
    Nov 2014
    Posts
    16

    Default Re: What is the use of robots.txt file?

    what is the use of robot. text files.

  10. #10
    Junior Member
    Join Date
    Nov 2013
    Location
    Australia
    Posts
    29

    Default Re: What is the use of robots.txt file?

    Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

  11. #11
    Newbie
    Join Date
    Nov 2014
    Posts
    1

    Default Re: What is the use of robots.txt file?

    It provides Google bots what are the pages which they should not crawl also now it used for SEO by adding the sitemap in the robots.txt

  12. #12
    Member
    Join Date
    Aug 2014
    Location
    Singapore
    Posts
    93

    Default Re: What is the use of robots.txt file?

    The content of this file tells search engine crawlers how they should visit your site. If there are files and directories you do not want indexed by search engines, you can use a robots.txt file to define where the robots should not go. These files are very simple text files that are placed on your web server.

  13. #13
    Member
    Join Date
    Dec 2014
    Location
    Delhi
    Posts
    44

    Default Re: What is the use of robots.txt file?

    If you don't want to index your site but not crawl by search engine then we use robot.txt file.

  14. #14
    Senior Member
    Join Date
    Sep 2014
    Posts
    202

    Default Re: What is the use of robots.txt file?

    The content of this file tells search engine crawlers how they should visit your site. If there are files and directories you do not want indexed by search engines, you can use a robots.txt file to define where the robots should not go. These files are very simple text files that are placed on your web server.

  15. #15
    Member
    Join Date
    Feb 2015
    Location
    India
    Posts
    38

    Default Re: What is the use of robots.txt file?

    Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders. Web authors can use robots.txt to keep cooperating Web robots from accessing all or parts of a Web site that you want to keep private.

  16. #16
    Member
    Join Date
    Mar 2015
    Posts
    62

    Default Re: What is the use of robots.txt file?

    The preferred way of preventing parts of your site from being indexed is to use the Control Center page exclusion mechanism. This is covered in How to Exclude Pages from Search. You should read that "how to" first. The only reason you might need to use a robots.txt file is if you want to prevent someone else from using this search engine to index your site.

    A "robots.txt" file is a text file placed on your server which contains a list of robots and "disallows" for those robots. Each disallow will prevent any address that starts with the disallowed string from being accessed.

  17. #17
    Senior Member
    Join Date
    Mar 2015
    Posts
    144

    Default Re: What is the use of robots.txt file?

    Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

  18. #18
    Junior Member
    Join Date
    Apr 2015
    Posts
    29

    Default Re: What is the use of robots.txt file?

    The robots.txt files help ensure Google and other search engines are crawling and indexing your site properly.

  19. #19
    Junior Member
    Join Date
    May 2015
    Posts
    11

    Default Re: What is the use of robots.txt file?

    Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

  20. #20
    Member
    Join Date
    May 2015
    Location
    12991 Keele Street King City, Ontario L7B 1G2 Canada
    Posts
    61

    Default Re: What is the use of robots.txt file?

    It is very important text file for your website. he robots.txt file is used to provide instructions about the Web site to Web robots and spiders.

Page 1 of 2 12 LastLast