Slowing Down Crawl of MSN and Yahoo Bots ?

Client Joined: Dec 04, 2004 Posts: 757

Hello all, at the moment my ravennule sites are getting hammered by Bots Sad

mainly MSN and a fair yahoo and most of the time its shows I have around 300 to 600 users online at one time which is not true as My uniques are around 1400 pepr day etc.
When checking I find most of these users are bots msn and yahoo.
I have read that I can put something in my robots.txt but I am not sure if I have done it correctly Question

this is what I have...

User-agent: *
Crawl-delay: 5
User-agent: msnbotCrawl-delay: 120
User-agent: Slurp
Crawl-delay: 10
Disallow: /abuse/
Disallow: /admin/
Disallow: /blocks/
Disallow: /cgi-bin/
Disallow: /db/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /modules/
Disallow: /ShortLinks/
Disallow: /themes/
Disallow: /admin.php
Disallow: /config.php

Is this correct ? and is there anything else not just to slow down msn and yahoo but to stop other types of bots etc ?

thanks for any help all
mrix

Regular Joined: Dec 28, 2004 Posts: 94

LMFAO Most people try for years to get search engines all over thier site. Shocked

I think would be right m8 yes. At least for any search engine that now uses robot.txt

(Eeeeeek I feel an elevated discussion on robots text coming up.)

This I found some time ago and added to the end of my robots.txt

Code:

User-agent: *


Crawl-delay: 10


Disallow: /abuse/


Disallow: /admin/


Disallow: /blocks/


Disallow: /cgi-bin/


Disallow: /db/


Disallow: /images/


Disallow: /includes/


Disallow: /language/


Disallow: /modules/


Disallow: /ShortLinks/


Disallow: /themes/


Disallow: /admin.php


Disallow: /config.php





Request-rate: 1/10


Visit-time: 0100-0500

You see the added Request-rate and Visit-time.

You could also try editing your meta data. includes/meta.php

I've got this in mine to try and restrict the masses of bots.

After

Code:

$metastring .= "<meta name=\"ROBOTS\" content=\"INDEX, FOLLOW\" />\n";

Add

Code:

$metastring .= "<meta name=\"REVISIT-AFTER\" content=\"5 DAYS\" />\n";

Now I've tried myself to search the net and find code that would tell the crawlers to either come back another day or at a certain time to NO avail. They just dont seem listen and are still always over my site, only maxed out to about 150 though. I dont think the above work at all for Google and Yahoo bots. There is also many a discussion on whether robots.txt or the meta tag actually works.

Here's a few things I found the other day.
[ Only registered users can see links on this board! Get registered or login! ]
[ Only registered users can see links on this board! Get registered or login! ]

SEO........ it SUCKS

TTFN

Trub

Posted: Thu Mar 13, 2008 7:12 am

Code:

$metastring .= "<meta name=\"REVISIT-AFTER\" content=\"5 DAYS\" />\n";

That never worked in the past and it will not work in the future I think.

Posted: Thu Mar 13, 2008 8:05 am

Susann wrote:

Code:

$metastring .= "<meta name=\"REVISIT-AFTER\" content=\"5 DAYS\" />\n";

That never worked in the past and it will not work in the future I think.

I know, that was me clutching at straws last year. Very Happy