Ravens PHP Scripts: Forums
 

 

View next topic
View previous topic
Post new topic   Reply to topic    Ravens PHP Scripts And Web Hosting Forum Index -> RN v2.20.00 - All Issues
Author Message
mrix
Client


Joined: Dec 04, 2004
Posts: 757

PostPosted: Wed Mar 12, 2008 4:49 am Reply with quote

Hello all, at the moment my ravennule sites are getting hammered by Bots Sad mainly MSN and a fair yahoo and most of the time its shows I have around 300 to 600 users online at one time which is not true as My uniques are around 1400 pepr day etc.
When checking I find most of these users are bots msn and yahoo.
I have read that I can put something in my robots.txt but I am not sure if I have done it correctly Question this is what I have...

User-agent: *
Crawl-delay: 5
User-agent: msnbotCrawl-delay: 120
User-agent: Slurp
Crawl-delay: 10
Disallow: /abuse/
Disallow: /admin/
Disallow: /blocks/
Disallow: /cgi-bin/
Disallow: /db/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /modules/
Disallow: /ShortLinks/
Disallow: /themes/
Disallow: /admin.php
Disallow: /config.php

Is this correct ? and is there anything else not just to slow down msn and yahoo but to stop other types of bots etc ?

thanks for any help all
mrix
 
View user's profile Send private message Visit poster's website
Trubador
Regular
Regular


Joined: Dec 28, 2004
Posts: 94

PostPosted: Wed Mar 12, 2008 9:25 am Reply with quote

LMFAO Most people try for years to get search engines all over thier site. Shocked

I think would be right m8 yes. At least for any search engine that now uses robot.txt

(Eeeeeek I feel an elevated discussion on robots text coming up.)

This I found some time ago and added to the end of my robots.txt

Code:
User-agent: *

Crawl-delay: 10
Disallow: /abuse/
Disallow: /admin/
Disallow: /blocks/
Disallow: /cgi-bin/
Disallow: /db/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /modules/
Disallow: /ShortLinks/
Disallow: /themes/
Disallow: /admin.php
Disallow: /config.php

Request-rate: 1/10
Visit-time: 0100-0500


You see the added Request-rate and Visit-time.

You could also try editing your meta data. includes/meta.php

I've got this in mine to try and restrict the masses of bots.

After

Code:
$metastring .= "<meta name=\"ROBOTS\" content=\"INDEX, FOLLOW\" />\n";


Add

Code:
$metastring .= "<meta name=\"REVISIT-AFTER\" content=\"5 DAYS\" />\n";


Now I've tried myself to search the net and find code that would tell the crawlers to either come back another day or at a certain time to NO avail. They just dont seem listen and are still always over my site, only maxed out to about 150 though. I dont think the above work at all for Google and Yahoo bots. There is also many a discussion on whether robots.txt or the meta tag actually works.

Here's a few things I found the other day.
Only registered users can see links on this board! Get registered or login!
Only registered users can see links on this board! Get registered or login!

SEO........ it SUCKS

TTFN

Trub
 
View user's profile Send private message
Susann
Moderator


Joined: Dec 19, 2004
Posts: 3191
Location: Germany:Moderator German NukeSentinel Support

PostPosted: Thu Mar 13, 2008 7:12 am Reply with quote

Code:
$metastring .= "<meta name=\"REVISIT-AFTER\" content=\"5 DAYS\" />\n";
That never worked in the past and it will not work in the future I think.
 
View user's profile Send private message
Trubador
PostPosted: Thu Mar 13, 2008 8:05 am Reply with quote

Susann wrote:
Code:
$metastring .= "<meta name=\"REVISIT-AFTER\" content=\"5 DAYS\" />\n";
That never worked in the past and it will not work in the future I think.


I know, that was me clutching at straws last year. Very Happy
 
Display posts from previous:       
Post new topic   Reply to topic    Ravens PHP Scripts And Web Hosting Forum Index -> RN v2.20.00 - All Issues

View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum


Powered by phpBB © 2001-2007 phpBB Group
All times are GMT - 6 Hours
 
Forums ©