Page 2 of 2 FirstFirst 12
Results 21 to 32 of 32

What is the use of robots.txt file?

This is a discussion on What is the use of robots.txt file? within the Web Design forums, part of the Web Designing & Development category; you can hide whatever hide from google crawler your content category administrator like those thing...

  1. #21
    Member
    Join Date
    May 2015
    Posts
    39

    Default Re: What is the use of robots.txt file?

    you can hide whatever hide from google crawler your content category administrator like those thing

  2. #22
    Member
    Join Date
    May 2015
    Location
    12991 Keele Street King City, Ontario L7B 1G2 Canada
    Posts
    61

    Default Re: What is the use of robots.txt file?

    Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

  3. #23
    Member MrJaswal's Avatar
    Join Date
    Jul 2015
    Location
    Ludhiana,Punjab,India
    Posts
    30

    Default Re: What is the use of robots.txt file?

    Quote Originally Posted by apdesignsonline View Post
    Can you tell me the answer
    A robots.txt file gives instructions to web robots about the pages the website owner doesn’t wish to be ‘crawled’. For instance, if you didn’t want your images to be listed by Google and other search engines, you’d block them using your robots.txt file. One wrong step in robot.txt file and your website going downfall in Google ranking list.

  4. #24
    Senior Member
    Join Date
    Jun 2015
    Posts
    940

    Default Re: What is the use of robots.txt file?

    Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

  5. #25
    Senior Member Thoughtgrid's Avatar
    Join Date
    Jul 2016
    Location
    Bangalore HSR layout 1 sector
    Posts
    137

    Default Re: What is the use of robots.txt file?

    The robot.txt file used to block the specific folder and content,allow access to specific crawlers these are the main uses of the robot.txt file

  6. #26
    Member
    Join Date
    May 2016
    Posts
    64

    Default Re: What is the use of robots.txt file?

    Robot.txt file is one of primary way of telling a search engine where it can and can’t go on your website.

  7. #27
    Senior Member pxljobs's Avatar
    Join Date
    Jul 2016
    Location
    Bangalore
    Posts
    128

    Default Re: What is the use of robots.txt file?

    Main uses of the robots.txt files are these specify the search engine crawler and spider which web page and files want or not visit that page.

  8. #28
    Senior Member
    Join Date
    Jul 2017
    Posts
    210

    Default Re: What is the use of robots.txt file?

    A robots.txt is a small file that tells a search engine which pages to index and which pages to ignore.

  9. #29
    Member
    Join Date
    Apr 2018
    Posts
    52

    Default Re: What is the use of robots.txt file?

    RObot.txt is used when you dont want search engines to crawl that particular page of your website.

  10. #30
    Senior Member
    Join Date
    Apr 2018
    Posts
    107

    Default Re: What is the use of robots.txt file?

    Thanks for sharing views about robots.txt file. I just want not to index my website few pages. So robots.txt file will work for me.

  11. #31
    Junior Member
    Join Date
    Jan 2018
    Location
    Nagpur
    Posts
    22

    Default Re: What is the use of robots.txt file?

    Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl & index pages on their website.

  12. #32
    Member
    Join Date
    Mar 2015
    Location
    Ahmedabad, India
    Posts
    59

    Default Re: What is the use of robots.txt file?

    Actually robots.txt file is used to block or allow search engine. It gives access to search engine to allow crawl or visit the particular page of your website.
    Great Market Place to Buy Premium Plugins & Premium Themes

Page 2 of 2 FirstFirst 12