Results 1 to 19 of 19

What is robots.txt?

This is a discussion on What is robots.txt? within the Advertising & Marketing forums, part of the Internet Marketing category; Hello, What is robots.txt?...

  1. #1
    Member
    Join Date
    Dec 2017
    Posts
    97

    Default What is robots.txt?

    Hello,

    What is robots.txt?

  2. #2
    OPH
    OPH is offline
    Senior Member
    Join Date
    Sep 2017
    Location
    DELHI, INDIA
    Posts
    391

    Default Re: What is robots.txt?

    robots.txt is a text file containing two lines of text to restrict the Search Engine Crawling bots for crawling the website or the Webpages. This file placed in the root folder of the website.

  3. #3
    Senior Member
    Join Date
    Mar 2018
    Location
    Gurdaspur
    Posts
    233

    Default Re: What is robots.txt?

    thanks for sharing with us

  4. #4
    Senior Member RH-Calvin's Avatar
    Join Date
    Jun 2013
    Posts
    3,290

    Default Re: What is robots.txt?

    Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
    Cheap VPS | $1 VPS Hosting | Windows with Remote Desktop
    Cheap Dedicated Server | $29 Dedicated Server with Free IPMI Setup

  5. #5
    Senior Member
    Join Date
    Nov 2016
    Posts
    150

    Default Re: What is robots.txt?

    Robots.txt is a text file to restrict search engine bots to crawl/not to crawl particular web pages like admin...

  6. #6
    Senior Member
    Join Date
    Nov 2017
    Posts
    275

    Default Re: What is robots.txt?

    thanks for sharing with us

  7. #7
    Senior Member
    Join Date
    Jan 2012
    Posts
    1,934

    Default Re: What is robots.txt?

    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too na´ve to rely on robots.txt to protect it from being indexed and displayed in search results.

  8. #8
    Senior Member
    Join Date
    Apr 2017
    Location
    india
    Posts
    471

    Default Re: What is robots.txt?

    thanks for sharing with us

  9. #9
    Senior Member
    Join Date
    Mar 2018
    Posts
    309

    Default Re: What is robots.txt?

    thanks for sharing with us

  10. #10
    Senior Member
    Join Date
    Feb 2018
    Location
    Haryana
    Posts
    227

    Default Re: What is robots.txt?

    thanks for sharing with us

  11. #11
    Senior Member
    Join Date
    May 2017
    Location
    jalandhar
    Posts
    407

    Default Re: What is robots.txt?

    thanks for sharing with us

  12. #12
    Senior Member
    Join Date
    Feb 2016
    Location
    USA
    Posts
    113

    Default Re: What is robots.txt?

    This is a great post.

  13. #13
    Member
    Join Date
    Oct 2017
    Location
    Delhi, India
    Posts
    39

    Default Re: What is robots.txt?

    In Simple Words Robots.txt is a file which controls the accessibility of crawlers.

  14. #14
    Senior Member Amit Tiwari's Avatar
    Join Date
    Feb 2017
    Posts
    227

    Default Re: What is robots.txt?

    robots.txt is a text file containing two lines of text to restrict the Search Engine Crawling bots for crawling the website or the Webpages.

  15. #15
    Senior Member
    Join Date
    Mar 2018
    Location
    Australia
    Posts
    475

    Default Re: What is robots.txt?

    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

  16. #16
    Senior Member hostechsupport's Avatar
    Join Date
    Feb 2017
    Posts
    404

    Default Re: What is robots.txt?

    Robots.txt file is at the root of the website that involves sectors of your website you don’t want to be attained by search engine crawlers. Webmasters use a robot.txt file to instruct the search engine robots on how to crawl & index the web pages.
    HostechSupport
    Best Linux/ Windows Server Management
    24x7 Skype: hostechsupportindia

  17. #17
    Senior Member buyinstagramfollower's Avatar
    Join Date
    Dec 2017
    Posts
    210

    Default Re: What is robots.txt?

    The robots.txt file is primarily used to specify which parts of your website should be crawled by spiders or web crawlers. It can specify different rules for different spiders.
    Googlebot is an example of a spider. It’s deployed by Google to crawl the Internet and record information about websites so it knows how high to rank different websites in search results.
    Using a robots.txt file with your website is a web standard. Spiders look for the robots.txt file in the host directory (or main folder) of your website. This text file is always named “robots.txt”. You can find your robots.txt file by going to:
    yourwebsite.com/robots.txt
    Most mainstream spiders comply with directions specified in robots.txt files but nefarious spiders may not. The content within robot.txt files are publicly available. You can attempt to ban unwanted spiders by editing the .htaccess file associated with your website.
    It’s important that marketers check their robots.txt file to make sure search engines are invited to crawl important pages. If you ask search engines to not crawl your website, then your website won’t appear in search results.
    You can also use the robots.txt file to show spiders where to find a sitemap of your website, which can make your content more discoverable.
    You can also specify a crawl-delay, or how many seconds robots should wait before collecting more information. Some websites may need to use this setting if bots are eating up bandwidth and causing your website to load slower for human visitors.

  18. #18
    Member
    Join Date
    May 2016
    Location
    Florida, United States
    Posts
    72

    Default Re: What is robots.txt?

    The file robots.txt is used to give instructions to web robots, such as search engine crawlers, about locations within the web site that robots are allowed, or not allowed, to crawl and index.

  19. #19
    Senior Member
    Join Date
    Jul 2017
    Posts
    259

    Default Re: What is robots.txt?

    Robots.txt may be a text (not html) file you place on your website to inform search robots that pages you'd like them to not visit. Robots.txt is by no suggests that necessary for search engines however usually search engines adjust what they're asked to not do. it's vital to clarify that robots.txt isn't the way from preventing search engines from creep your website (i.e. it's not a firewall, or a form of arcanum protection) and also the proven fact that you place a robots.txt file are a few things like golf shot a note “Please, don't enter” on associate degree unbarred door – e.g. you can't forestall thieves from coming back in however the great guys won't hospitable door and enter. that's why we are saying that if you have got very subunit sitive information, it's too na´ve to have confidence robots.txt to shield it from being indexed and displayed in search results.


    AUDITING | SEAT CUSHION | PAYROLL SERVICES