Results 1 to 9 of 9

Is robots.txt file necessary

This is a discussion on Is robots.txt file necessary within the Search Engine Optimization forums, part of the Internet Marketing category; Hi Is it necessary to put robots.txt file in the root directory of website, i know it is helpful to ...

  1. #1
    Junior Member Array
    Join Date
    Oct 2008
    Posts
    19

    Default Is robots.txt file necessary

    Hi

    Is it necessary to put robots.txt file in the root directory of website, i know it is helpful to control crawlers, but if i not put this file in the website, does this cause some problem, please advice
    Thanks for your support

  2. #2
    Senior Member Array ddwebguru's Avatar
    Join Date
    Oct 2008
    Posts
    299

    Default Re: Is robots.txt file necessary

    [FONT=Georgia]It is not absolutely necessary. But if you want to hide some files or directories from Google robot, you can give instruction in this section. You can create a robot.txt file and then instruct the robot exude them while crawling.

    [/FONT]

  3. #3
    Moderator Array mackien's Avatar
    Join Date
    Jun 2008
    Location
    frozen throne
    Posts
    245

    Default Re: Is robots.txt file necessary

    Robot.txt file is like a gateway of robots once you prevent them from indexing your site then they won't crawl it. But some SE bots don't follow this robot.txt file.
    SEO Uragon Website Optimization Guides | Google Search Engine Optimization

  4. #4

  5. #5
    Senior Member Array Hafsoh's Avatar
    Join Date
    Apr 2008
    Posts
    433

    Default Re: Is robots.txt file necessary

    Website will operate without robots.txt file, though there are so many benefits that are attached toit, that you should strongly consider including one.

  6. #6
    Member Array
    Join Date
    Sep 2008
    Location
    New York,US
    Posts
    40

    Default Re: Is robots.txt file necessary

    Robot.txt is not necessary but it has lots of benefit. It includes disallow facility also.
    So its better to have robot.txt file on your root directory.

  7. #7
    Junior Member Array
    Join Date
    Apr 2009
    Posts
    4

    Default Re: Is robots.txt file necessary

    It is not necessary, but you can get benefits from robots.txt so, don't worry about it.

  8. #8
    Junior Member Array
    Join Date
    Apr 2009
    Posts
    22

    Default Re: Is robots.txt file necessary

    You should use robots for telling search engines stuff.

  9. #9
    Member Array
    Join Date
    Apr 2009
    Posts
    40

    Default Re: Is robots.txt file necessary

    Ofcourse its not very necessary. Its just the helpful for crawling websites.



Similar Threads

  1. Robots.txt Help?
    By Cal S in forum Web & Server Administration
    Replies: 6
    Last Post: 02-27-2012, 06:30 AM
  2. Sitemaps supported via robots.txt
    By SticKer in forum Web & Server Administration
    Replies: 3
    Last Post: 09-19-2011, 09:36 PM
  3. Replies: 1
    Last Post: 01-20-2008, 01:19 AM
  4. Robots.txt question...?
    By onyxenchanted in forum Web & Server Administration
    Replies: 1
    Last Post: 01-10-2008, 10:46 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
SEO Forum | Web Hosting Forum | Websites For Sale |