Ravens PHP Scripts: Forums
 

 

View next topic
View previous topic
Post new topic   Reply to topic    Ravens PHP Scripts And Web Hosting Forum Index -> Security - PHP Nuke
Author Message
express
Client


Joined: Jan 03, 2007
Posts: 94

PostPosted: Tue Dec 11, 2007 1:30 pm Reply with quote

I have read in here how I can stop bots from crawling my site. I inserted the new txt into the robot txt file and they still are showing up. I have even added the additional wording to keep googlbot out but it too keeps crawling.

Here is a copy of my robot file. What am I doing wrong?

Code:
User-agent: *

Disallow: /
User-agent: Googlebot
Disallow: /
Crawl-delay: 5
Disallow: /abuse/
Disallow: /admin/
Disallow: /blocks/
Disallow: /cgi-bin/
Disallow: /db/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /modules/
Disallow: /ShortLinks/
Disallow: /themes/
Disallow: /admin.php
Disallow: /config.php
Disallow: /main.php
Disallow: /index.php
Disallow: /embed.php
Disallow: /upgrade/
Disallow: /install/
Disallow: /images/
 
View user's profile Send private message
jakec
Site Admin


Joined: Feb 06, 2006
Posts: 3048
Location: United Kingdom

PostPosted: Tue Dec 11, 2007 2:57 pm Reply with quote

You should be able to block them using your .htaccess. If you search the forums you should find a number of posts discussing how to do this.
 
View user's profile Send private message
evaders99
Former Moderator in Good Standing


Joined: Apr 30, 2004
Posts: 3221

PostPosted: Tue Dec 11, 2007 5:23 pm Reply with quote

Googlebot does follow the robots.txt rules. You need to disallow the loading scripts - mostly modules.php

Many other bots will ignore robots.txt. You can ban them by .htaccess using their User-Agent strings, but remember that these are easily changed and falsified. There isn't an easy way to stop your site from being searched. If that's the way to run your site, don't put it on the Internet! Smile

Members-only sites typically have some kind of password system to allow only authorized users entry into a site.

_________________
- Only registered users can see links on this board! Get registered or login! -

Need help? Only registered users can see links on this board! Get registered or login! 
View user's profile Send private message Visit poster's website
Display posts from previous:       
Post new topic   Reply to topic    Ravens PHP Scripts And Web Hosting Forum Index -> Security - PHP Nuke

View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum


Powered by phpBB © 2001-2007 phpBB Group
All times are GMT - 6 Hours
 
Forums ©