Results 1 to 16 of 16

Robot.txt

This is a discussion on Robot.txt within the Blogging forums, part of the Internet Marketing category; What is Robot.txt file in seo ?...

  1. #1
    Junior Member
    Join Date
    Jul 2017
    Location
    mumbai
    Posts
    15

    Default Robot.txt

    What is Robot.txt file in seo?

  2. #2
    Member
    Join Date
    Jul 2017
    Posts
    46

    Default Re: Robot.txt

    Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

    The location of robots.txt is very important. It must be in the main directory because otherwise user agents will not be able to find it they do not search the whole site for a file named robots.txt.

  3. #3
    Senior Member RH-Calvin's Avatar
    Join Date
    Jun 2013
    Posts
    2,158

    Default Re: Robot.txt

    Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
    Cheap VPS | $1 VPS Hosting | Windows with Remote Desktop
    Cheap Dedicated Server | $29 Dedicated Server with Free IPMI Setup

  4. #4
    Junior Member
    Join Date
    Apr 2017
    Posts
    21

    Default Re: Robot.txt

    Robots.txt is a text file which contains list of all webpages that can be easily accessible by search engine.

  5. #5
    Senior Member
    Join Date
    Jul 2017
    Posts
    111

    Default Re: Robot.txt

    Robots.txt is a text file that details websites which contain instructions for google robots. The file details websites that are allowed and disallowed from online search engine crawling.

  6. #6
    Member
    Join Date
    Jan 2017
    Posts
    61

    Default Re: Robot.txt

    The file lists webpages that are allowed and disallowed from search engine crawling.

  7. #7
    Junior Member
    Join Date
    Jul 2017
    Location
    london,united kingdom
    Posts
    19

    Default Re: Robot.txt

    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naļve to rely on robots.txt to protect it from being indexed and displayed in search results.

  8. #8
    Senior Member
    Join Date
    Aug 2017
    Location
    Chandigarh
    Posts
    154

    Default Re: Robot.txt

    Thanks for the sharing and updating

  9. #9
    Senior Member chinomoreno's Avatar
    Join Date
    Apr 2014
    Posts
    888

    Default Re: Robot.txt

    Robots.txt file is a text file where a list of commands is listed; it tells whether a specific site url is to be index/follow or not.

  10. #10
    Senior Member
    Join Date
    Jul 2017
    Posts
    111

    Default Re: Robot.txt

    Robots.txt is a text file that details websites which contain instructions for google robots. The file details websites that are allowed and disallowed from online search engine crawling.

  11. #11
    Member
    Join Date
    Sep 2017
    Posts
    50

    Default Re: Robot.txt

    Robots.txt file is a text file for limiting bots (robots, search engine crawlers ) since a website or certain pages on the website. By a robots.txt file and with a disallow direction, we can restrict bots or search engine crawling program from websites and or since certain folders and files.

  12. #12
    Member
    Join Date
    Jul 2016
    Location
    Delhi, India
    Posts
    69

    Default Re: Robot.txt

    A robots.txt file is a file at the root of your site that indicates those parts of your site you don't want accessed by search engine crawlers.

  13. #13
    Senior Member
    Join Date
    Aug 2017
    Location
    Amritsar
    Posts
    102

    Default Re: Robot.txt

    Quote Originally Posted by doorhardwareUK View Post
    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naļve to rely on robots.txt to protect it from being indexed and displayed in search results.
    Thanks for posting, this is very useful. i like your views

  14. #14
    Member
    Join Date
    Oct 2017
    Location
    Patiala
    Posts
    36

    Default Re: Robot.txt

    Quote Originally Posted by priya456 View Post
    Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

    The location of robots.txt is very important. It must be in the main directory because otherwise user agents will not be able to find it they do not search the whole site for a file named robots.txt.
    Thanks for the useful information

  15. #15
    Member
    Join Date
    Jul 2017
    Posts
    90

    Default Re: Robot.txt

    Web site owners use the /robots.txt file to give instructions about their site to google that not crawl these URL.

  16. #16
    Member
    Join Date
    Nov 2017
    Posts
    51

    Default Re: Robot.txt

    Quote Originally Posted by priya456 View Post
    Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

    The location of robots.txt is very important. It must be in the main directory because otherwise user agents will not be able to find it they do not search the whole site for a file named robots.txt.
    With a followup to this post, i can say we should not miss use the robots file. If you wrongly enters the web pages under do not index title then they won't appear in search engine. And try to hide the pages with sensitive information like payment pages, logins, user profiles (which you wish to hide from search engines).