How to stop Google from crawling secure content/directory?

**Andrew** · 01-07-2008 10:48 PM

My post title is clearly stating about my question

. But lets repeat it. How can we stop google bots to crawl through a secure (not-willing-to-share) information from one of the website's directories.

Can we use robots.txt for this or you may suggest any other better treatment for this?

If Google crawls any secure information from a website content and shows it in seacrh engines then what can we do to let this information disappeared from Google search results?

Your co-operation is appreciated in adavance.

**slnet** · 01-07-2008 11:19 PM

Two ways to do this:

1)

The following example "/robots.txt" file specifies that no robots should visit any URL starting with "/webicy/secret/" or "/tmp/", or /personal.html:

Code:

# robots.txt for http://www.example.com/

User-agent: *
Disallow: /webicy/secret/ # This is an infinite virtual URL space
Disallow: /tmp/ # these will soon disappear
Disallow: /personal.html

This example "/robots.txt" file specifies
that no robots should visit any URL starting with
"/webicy/secret/", except the robot called
"techmaster":

Code:

 
# robots.txt for http://www.example.com/

User-agent: *
Disallow: /webicy/secret/ # This is an infinite virtual URL space

# techmaster knows where to go.
User-agent: techmaster
Disallow:

This example indicates that no robots should visit this site further:

Code:

  
# go away
User-agent: *
Disallow: /

2)

To prevent all robots from indexing a page on your site, you'd place the following meta tag into the <HEAD> section of your page:

Code:

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

To allow other robots to index the page on your site, preventing only Google's robots from indexing the page, you'd use the following tag:

Code:

<META NAME="GOOGLEBOT" CONTENT="NOINDEX, NOFOLLOW">

To allow robots to index the page on your site but instruct them not to follow outgoing links, you'd use the following tag:

Code:

<META NAME="ROBOTS" CONTENT="NOFOLLOW">

To allow robots to index the page on your site but instruct them not to index images on that page, you'd use the following tag:

Code:

<META NAME="ROBOTS" CONTENT="NOIMAGEINDEX">

Once you have done one of the above, your content will be no more indexed.. as far as how to remove contents.. the above will work if you are willing to wait a week... but if you want it done instantly... then visit ... http://www.google.com/support/webmas...n&answer=61062