Robots.txt Question

[TheJediSlayer]

Hello,

I was wondering if it were possible to deny all bots, but bots from search engines like Google. I realize that certain bots will ignore the robots.txt file, but I still would like to create a robots.txt that would deny from all except certain search engines that I put into the robots.txt file. Thanks for any help,

Tyler.

 

[kimsland]

You know there's a lot in this
http://www.robotstxt.org/

But because of new bots everyday (no doubt!) I don't think you can stop all.

It'll be interesting to hear from others replies, especially the Spyware experts here.

This may help you too:
http://www.robotstxt.org/faq/blockjustbad.html

 

[TheJediSlayer]

Yeah, I checked that site out. I'm just unsure how to go about making a robots.txt with just a few bots allowed. Anyways, I'll keep looking around, but any help is appreciated, again.

 

[DelJo63]

from the robottxt.org site

User-agent: Google
Disallow:

User-agent: *
Disallow: /​

notice the EMPTY disallow ... implies Allow Google

 

[TheJediSlayer]

Hey,

Thank you very much. I appreciate it.

Oh, one last question. For the robots.txt file. Would I just enter User-agent: Yahoo under Google, in order to make it so that Yahoo too can scan my site?

 

[DelJo63]

read http://www.robotstxt.org/

 

[TheJediSlayer]

Thank you,

Tyler