sentinel blocking Mediapartners-Google/2.1

Sells PC To Pay For Divorce Joined: Posts: 5661

if noticed that sentinel 2.1.0 just blocked the visits from Mediapartners-Google/2.1

all visits blocked as :

Abuse-Admin
User Agent: Mediapartners-Google/2.1
Query String: admin.php?op=ABConfigFilter
Query String: admin.php?op=ABBlockedIP&min=0&column=date&direction=desc&sip=
Query String: admin.php?op=ABConfigAuthor

any way to prevent this ?

John

Site Admin/Owner Joined: Aug 27, 2002 Posts: 17088

John,

If you're using v2.1.0 you can protect IP ranges and that one is a protected range by default, I believe. If not, you can add the IP range in the Protected Range Panel. However, if you are using v2.0.2 then see this post to protect the range [ Only registered users can see links on this board! Get registered or login! ]

Posted: Mon Oct 18, 2004 6:42 am

Try adding this to your robots.txt

Code:

User-agent: Mediapartners-Google*


Disallow: admin.php

Posted: Mon Oct 18, 2004 6:51 am

hi thanks for the reply,

the site in question is a subdomain,so the allowed is infact in the root so there shouldnt be that problem.
and for raven....i installed the 2.1.0 version.
there wasnt any version installed yet so ....
but i dont know exactly what you mean that google should already be excluded ?
But should the protection from google show as default?...cause it doesnt show as there isnt any IP's that are excluded....
And as i was typing it get banned another 2 times.... Sad

Posted: Mon Oct 18, 2004 6:55 am

In the Protected Range menu you should see a couple of entries like:

66.249.64.0 66.249.79.255

That should encompass that bot as I use it too. If that range is not in there, or if the range you have is different, just add it.

Posted: Mon Oct 18, 2004 7:12 am

well i looked and when you use the function to show the excluded range its empty,so yes i entered the full range now.
so will see what happens now...
thanks.

Posted: Mon Oct 18, 2004 7:19 am

Not Excluded Range, Protected Range.

Posted: Mon Oct 18, 2004 7:26 am

yes oops sorry...
i changed it...
see if it will work...
google just got banned again....

Posted: Mon Oct 18, 2004 8:07 am

Try this in your robots.txt

User-agent: *
Disallow: admin.php
Disallow: /admin/
Disallow: /images/
Disallow: /includes/
Disallow: /themes/
Disallow: /blocks/
Disallow: /modules/
Disallow: /language/
Disallow: /abuse/
Disallow: /cgi-bin/
Disallow: /curl/
Disallow: /db/
Disallow: /docs/
Disallow: /gcf/
Disallow: /public/

Posted: Mon Oct 18, 2004 8:22 am

ok thanks raven will add that also...
but i dont understand why it bans google now.
is that related to the new version of sentinel ?
On my nuke site i also use the 2.1.0 and isnt banning google.
so this is kinda weird behaviour...

Posted: Mon Oct 18, 2004 8:33 am

Under 2.0.2 on my site, google and media partners would get banned when they tried to go down the admin path. Once I modified my robots.txt and put that range in, I have not had any problems. It is weird, I agree.

Posted: Mon Oct 18, 2004 9:02 am

Well the difference between the 2 sites is nothing...both are now 2.1.0
Website 1 isnt banning google ,btw it never did...(even before the forbidden directories were added to the robot.txt

Website 2 is or was banning google and did that 30 minutes after install...
So for me trying to understand this it seems weird cause both are in the same situation and both respond and act differently.

As for the crawler story...like many others i have been crawled a million times by the msn bot and this would probably help someone cause this is what msn maild me some time ago....

Code:




User-agent: Mediapartners-Google*


Disallow: 


User-agent: *


Disallow: /cgi-bin/


Disallow: /images/


User-Agent: msnbot


Crawl-Delay: 200


Disallow: /cgi-bin/


Disallow: /images/


 


This will tell MSNBot to wait 200 seconds between each request it makes to your server.

With this they offer a solution to slow down the msn request.
I didnt used it and completely banned msn from even entering the site.
But perhaps someone want to use this,but i havent seen it before so thats why i post it...[/code]

Client Joined: Jul 18, 2003 Posts: 977

Raven, what do you have under disallow for User-agent: Mediapartners-Google*

Posted: Mon Oct 18, 2004 11:24 am

blith wrote:

Raven, what do you have under disallow for User-agent: Mediapartners-Google*

That is my entire robots.txt file. I don't have any problems.

Posted: Mon Oct 18, 2004 11:27 am

I ask becasue I went 2.1 IP tracking and found the googlebots in all the stuff I just uploaded!! They were all over the admin folder! And private messages and everything else I thought I had disallowed