Ravens PHP Scripts: Forums
 

 

View next topic
View previous topic
Post new topic   Reply to topic    Ravens PHP Scripts And Web Hosting Forum Index -> RN v2.20.00 - All Issues
Author Message
mrix
Client


Joined: Dec 04, 2004
Posts: 757

PostPosted: Tue Mar 18, 2008 9:50 am Reply with quote

Hello all, just recently I have had loads of msn bots on my site Question
Basically I can have about 20 members on and around 400 to 600+ visitors Crying or Very sad when checking the ip`s most of the visitors are MSN bots.
Has anyone had this issue and if so what can I do to help it Question the last thing i would want to do is block the MSN Bots all together as you can imagine.

thanks fo any idea`s all
mrix
 
View user's profile Send private message Visit poster's website
Guardian2003
Site Admin


Joined: Aug 28, 2003
Posts: 6793
Location: Ha Noi, Viet Nam

PostPosted: Tue Mar 18, 2008 10:20 am Reply with quote

The Inktomi/Slurp bot is reknowned for this behaviour - just don't try banning them as they will come back with the second wave but I have not see this with MSN bots.

To save your resources and bandwidth, in your robots.txt file you can add the following directive for Yahoo/Slurp bot
Code:


User-agent: Slurp
Crawl-delay: 10

where 10 is the number of seconds between requests.
For the MSN bot you can use;
Code:
User-agent: msnbot

Crawl-delay: 10

However, to cover for all bots that adhere to the crawl delay directive you might want to use a blanket directive like;
Code:
User-agent: *

Crawl-delay: 10
 
View user's profile Send private message Send e-mail
mrix
PostPosted: Tue Mar 18, 2008 10:29 am Reply with quote

Thats great I`ll give it a go
cheers
mrix
 
Display posts from previous:       
Post new topic   Reply to topic    Ravens PHP Scripts And Web Hosting Forum Index -> RN v2.20.00 - All Issues

View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum


Powered by phpBB © 2001-2007 phpBB Group
All times are GMT - 6 Hours
 
Forums ©