Page 1 of 2 12 LastLast
Results 1 to 20 of 31

What is a robots.txt file?

This is a discussion on What is a robots.txt file? within the Search Engine Optimization forums, part of the Internet Marketing category; hi friends, What is a robots.txt file?...

  1. #1
    Senior Member orientation's Avatar
    Join Date
    Feb 2016
    Location
    Bangalore
    Posts
    108

    Default What is a robots.txt file?

    hi friends,
    What is a robots.txt file?

  2. #2
    Member
    Join Date
    Mar 2016
    Posts
    84

    Default Re: What is a robots.txt file?

    Robots.txt is a file which contains instructions for web crawlers,

  3. #3
    Senior Member Nexevo Technologies's Avatar
    Join Date
    Aug 2015
    Location
    Bangalore
    Posts
    852

    Default Re: What is a robots.txt file?

    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

  4. #4
    Senior Member chinomoreno's Avatar
    Join Date
    Apr 2014
    Posts
    898

    Default Re: What is a robots.txt file?

    Quote Originally Posted by orientation View Post
    hi friends,
    What is a robots.txt file?
    It is a file that command or tell bots whether a certain page will be indexed or not.

  5. #5
    Senior Member
    Join Date
    Jun 2015
    Location
    India
    Posts
    409

    Default Re: What is a robots.txt file?

    Robots is text file that indicate to search engine spiders, which url's crawl and which url's are not crawl in the website/blog. By using robots you can block the security webpages.

  6. #6
    Senior Member
    Join Date
    Apr 2016
    Posts
    144

    Default Re: What is a robots.txt file?

    The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website

  7. #7
    Senior Member BrilltechEngineers's Avatar
    Join Date
    May 2016
    Location
    Delhi
    Posts
    139

    Default Re: What is a robots.txt file?

    The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

  8. #8
    Junior Member
    Join Date
    May 2016
    Location
    Koramangala,KR Puram,Bangalore,Karnataka,India
    Posts
    27

    Default Re: What is a robots.txt file?

    The robots.txt file is a simple text file placed on your web server which tells webcrawlers like Googlebot if they should access a file or not.

  9. #9
    Senior Member RH-Calvin's Avatar
    Join Date
    Jun 2013
    Posts
    3,255

    Default Re: What is a robots.txt file?

    Robots.txt is a text file that is inserted into your website to contain instructions for search engine spiders. The file lists webpages that are allowed and disallowed from search engine crawling.
    Cheap VPS | $1 VPS Hosting | Windows with Remote Desktop
    Cheap Dedicated Server | $29 Dedicated Server with Free IPMI Setup

  10. #10
    Member
    Join Date
    Apr 2015
    Posts
    56

    Default Re: What is a robots.txt file?

    Robots.txt is a text file webmasters create to instruct robots how to crawl and index some external files like images,css,on their website.
    Last edited by Leasedlayer; 05-30-2016 at 06:39 AM.

  11. #11
    Junior Member
    Join Date
    Jun 2016
    Posts
    12

    Default Re: What is a robots.txt file?

    Robots.txt is used in websites to communicate with web crawlers. Any part of your sites not accessed by search engine crawlers it included in robots.txt file.

  12. #12
    Member
    Join Date
    Apr 2016
    Location
    Bangalore
    Posts
    95

    Default Re: What is a robots.txt file?

    It’s a text file which allow search engines to which page, directory, domain have to index or not.

  13. #13
    Senior Member
    Join Date
    Nov 2012
    Posts
    493

    Default Re: What is a robots.txt file?

    robots.txt files that help ensure Google and other search engines are crawling and indexing your site properly.

  14. #14
    Senior Member
    Join Date
    Apr 2016
    Location
    Bangalore
    Posts
    128

    Default Re: What is a robots.txt file?

    Robot.txt is a text file which is an important factor of a website as it tells the search engine to crawl the pages of a website completely.

  15. #15
    Senior Member telemart's Avatar
    Join Date
    Jun 2016
    Posts
    111

    Default Re: What is a robots.txt file?

    robot.txt is a file and which is used to disallow the crawler,allow the crawler of our page.

  16. #16
    Member
    Join Date
    Apr 2015
    Location
    28th Floor -The Gherkin,London EC3A 8BF
    Posts
    50

    Default Re: What is a robots.txt file?

    Robots.txt file is used to tell the Google Crawler to which page has to be crawled or not by allowing or disallowing.

  17. #17
    smt
    smt is offline
    Senior Member
    Join Date
    Jun 2016
    Posts
    105

    Default Re: What is a robots.txt file?

    Robots.txt file describe the instructions how the Google crawl of our page..............

  18. #18
    Senior Member
    Join Date
    Feb 2016
    Location
    USA
    Posts
    113

    Default Re: What is a robots.txt file?

    Robots.txt file is simple not pad file with extension .txt and it's help to allow or block Google crawler to crawl a particular page of website. It's upload in the root directory of a website with the help of admin login detail and FTP detail.

  19. #19
    Member Disdwarka23's Avatar
    Join Date
    Mar 2016
    Location
    New Delhi
    Posts
    36

    Default Re: What is a robots.txt file?

    In my knowledge the robots.txt file is a very important file if you want to have a good ranking on search engines, many Web sites don't offer this file.
    When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

  20. #20
    Junior Member
    Join Date
    Jun 2016
    Posts
    5

    Default Re: What is a robots.txt file?

    it has instruction to the crowler to index the webpage, firstly search engine search the robots.txt file instruction to index or ignore the webpage.

Page 1 of 2 12 LastLast