TechSpot

Robots.txt Question

By TheJediSlayer
Feb 9, 2008
  1. Hello,

    I was wondering if it were possible to deny all bots, but bots from search engines like Google. I realize that certain bots will ignore the robots.txt file, but I still would like to create a robots.txt that would deny from all except certain search engines that I put into the robots.txt file. Thanks for any help,

    Tyler.
     
  2. kimsland

    kimsland Ex-TechSpotter Posts: 14,524

  3. TheJediSlayer

    TheJediSlayer TS Rookie Topic Starter Posts: 164

    Yeah, I checked that site out. I'm just unsure how to go about making a robots.txt with just a few bots allowed. Anyways, I'll keep looking around, but any help is appreciated, again.
     
  4. jobeard

    jobeard TS Ambassador Posts: 9,342   +622

    from the robottxt.org site
    User-agent: Google
    Disallow:

    User-agent: *
    Disallow: /​
    notice the EMPTY disallow ... implies Allow Google :)
     
  5. TheJediSlayer

    TheJediSlayer TS Rookie Topic Starter Posts: 164

    Hey,

    Thank you very much. I appreciate it.

    Oh, one last question. For the robots.txt file. Would I just enter User-agent: Yahoo under Google, in order to make it so that Yahoo too can scan my site?
     
  6. jobeard

    jobeard TS Ambassador Posts: 9,342   +622

  7. TheJediSlayer

    TheJediSlayer TS Rookie Topic Starter Posts: 164

    Thank you,

    Tyler
     
Topic Status:
Not open for further replies.

Similar Topics

Add New Comment

You need to be a member to leave a comment. Join thousands of tech enthusiasts and participate.
TechSpot Account You may also...