bots still coming thru

Client Joined: Jan 03, 2007 Posts: 94

I have read in here how I can stop bots from crawling my site. I inserted the new txt into the robot txt file and they still are showing up. I have even added the additional wording to keep googlbot out but it too keeps crawling.

Here is a copy of my robot file. What am I doing wrong?

Code:

User-agent: *


Disallow: /


User-agent: Googlebot 


Disallow: /


Crawl-delay: 5


Disallow: /abuse/


Disallow: /admin/


Disallow: /blocks/


Disallow: /cgi-bin/


Disallow: /db/


Disallow: /images/


Disallow: /includes/


Disallow: /language/


Disallow: /modules/


Disallow: /ShortLinks/


Disallow: /themes/


Disallow: /admin.php


Disallow: /config.php


Disallow: /main.php


Disallow: /index.php


Disallow: /embed.php


Disallow: /upgrade/


Disallow: /install/


Disallow: /images/

Posted: Tue Dec 11, 2007 2:57 pm

You should be able to block them using your .htaccess. If you search the forums you should find a number of posts discussing how to do this.

Posted: Tue Dec 11, 2007 5:23 pm

Googlebot does follow the robots.txt rules. You need to disallow the loading scripts - mostly modules.php

Many other bots will ignore robots.txt. You can ban them by .htaccess using their User-Agent strings, but remember that these are easily changed and falsified. There isn't an easy way to stop your site from being searched. If that's the way to run your site, don't put it on the Internet! Smile

Members-only sites typically have some kind of password system to allow only authorized users entry into a site.