Ravens PHP Scripts: Forums
 

 

View next topic
View previous topic
Post new topic   Reply to topic    Ravens PHP Scripts And Web Hosting Forum Index -> NukeSentinel™
Author Message
boar
Hangin' Around


Joined: Jul 18, 2005
Posts: 33

PostPosted: Mon Jul 18, 2005 2:37 pm Reply with quote

hello, i have been having this problem for about a week now, each day i get an email from sentinel saying googlebot was blocked because it tries to access admin.php files.... this is my robots.txt file

User-agent: Mediapartners-Google*
Disallow:

User-agent: *
Disallow: /admin.php
Disallow: /admin/
Disallow: /images/
Disallow: /includes/
Disallow: /themes/
Disallow: /blocks/
Disallow: /modules/
Disallow: /language/

i dont want to add googlebot to the protected ip range because i dont think it would a good idea.

now..i dont think it's someone tryin to impersonate google's bot because the ip address is from google's corporation.

i have sentinel 2.3.2 on platinum 7.6
how can i fix this?
 
View user's profile Send private message
boar
PostPosted: Mon Jul 18, 2005 2:56 pm Reply with quote

btw, is it bad if googlebot gets banned?
does it mean that probably it wont crawl my site in the future?
 
sonicman
Regular
Regular


Joined: Apr 17, 2005
Posts: 64

PostPosted: Mon Jul 18, 2005 4:27 pm Reply with quote

Boar thats The same problem I am having, google trying to access admin.php files ADMIN is the reason sentinel gets banned, I have the same robots file as you have, I have nuke 7.5 and sentinel 2.3.1, Maybe we could just rename the admin file, but lets see what the experts have to say... Smile
 
View user's profile Send private message
Dauthus
Worker
Worker


Joined: Oct 07, 2003
Posts: 211

PostPosted: Mon Jul 18, 2005 5:39 pm Reply with quote

I ran into this a little bit ago. Try this:
Only registered users can see links on this board! Get registered or login!

Hope that helps.

_________________
Only registered users can see links on this board! Get registered or login!
Vivere disce, cogita mori 
View user's profile Send private message Visit poster's website
sonicman
PostPosted: Mon Jul 18, 2005 6:32 pm Reply with quote

I will try that
 
VinDSL
Life Cycles Becoming CPU Cycles


Joined: Jul 11, 2004
Posts: 614
Location: Arizona (USA) Admin: NukeCops.com Admin: Disipal Designs Admin: Lenon.com

PostPosted: Mon Jul 18, 2005 6:59 pm Reply with quote

Google bots aren't going to hurt anything. They're stupid! All they are doing is using the same link as some script kiddie hacker wannabe, that tried to hack your site, and it got picked up through the PR system. Punish a Google bot, and you punish yourself!

Taken to its logical conclusion, let's say a Google bot set itself up as an admin on your site, and/or got into your admin panel. What's a Google bot going to do, you know? It's not like it can think for itself!

I exclude all Google bots from being banned. I recommend you do the same... Wink

_________________
.:: "The further in you go, the bigger it gets!" ::.
.:: Only registered users can see links on this board! Get registered or login! | Only registered users can see links on this board! Get registered or login! ::. 
View user's profile Send private message Visit poster's website ICQ Number
boar
PostPosted: Mon Jul 18, 2005 7:45 pm Reply with quote

Well, i dropped all the sentinel's tables and started with fresh ones, i hope that's the problem because i dont understand why we are few having this problem while others probably using the same version of nuke and same version of sentinel aren't having this problem... now.. i have to wait and see if google's bot gets banned.

VinDSL, perhaps.. but, like i said, i dont understand why others aren't having this problem :/

Google's bot is a Spider On The Nuke? hehe
 
sonicman
PostPosted: Mon Jul 18, 2005 8:09 pm Reply with quote

VinDSL So you think I should just keep the google bot banned
 
boar
PostPosted: Mon Jul 18, 2005 9:14 pm Reply with quote

sonicman. dont do that unless you DONT want google's bot to crawl your site.

ok, google's bot got banned again even with new tables.. so the problem isn't there :/

Raven wrote:
The parentheses are causing it. Nuke should also forbid that string even if Sentinel was not installed. Comment out the Sentinel include statement in mainfile.php and try it.


i should do this?

/*****************************************************/
/* Security - Sentinel v.2.3.1 START */
/*****************************************************/
/*if (defined('FORUM_ADMIN')) {
@include_once("../../../includes/nukesentinel.php");
} elseif (defined('INSIDE_MOD')) {
@include_once("../../includes/nukesentinel.php");
} else {
@include_once("includes/nukesentinel.php");
}*/
/*****************************************************/
/* Security - Sentinel v.2.3.1 END */
/*****************************************************/

if i do it, it wont do anything bad? (something else not working..)


Last edited by boar on Mon Jul 18, 2005 10:49 pm; edited 1 time in total 
Dauthus
PostPosted: Mon Jul 18, 2005 9:45 pm Reply with quote

Vin is saying protect the ips of the googlebots. (I think) LOL.

Don't Comment out sentinel in the mainfile and leave it that way. What Raven was showing is that string was being banned, and it couldn't be looked up even without sentinel because Nuke doesn't allow that string either.

Just protect the IPs of Google if you have to. Googlebots are a GOOD thing. You don't want them banned.
 
boar
PostPosted: Mon Jul 18, 2005 10:52 pm Reply with quote

well, i will eventually expose my administration area to the public if i let Googlebots to crawl through my admin.php file.... I am not saying googlebots are bad, i never thought that way either, what i am trying to accomplish here is to deny access to googlesbots or any other bot to my admin.php file like other websites do using sentinel.
 
VinDSL
PostPosted: Tue Jul 19, 2005 1:03 am Reply with quote

Dauthus wrote:
Vin is saying protect the ips of the googlebots. (I think) LOL... Just protect the IPs of Google if you have to. Googlebots are a GOOD thing. You don't want them banned.

Exactly!
 
boar
PostPosted: Tue Jul 19, 2005 1:16 am Reply with quote

Okay!

i wont protect googlebots IP's.....
 
money
New Member
New Member


Joined: Aug 24, 2003
Posts: 11

PostPosted: Tue Jul 19, 2005 8:10 am Reply with quote

This entry here grants this particular googlebot access to all areas on your website.

Quote:
User-agent: Mediapartners-Google*
Disallow:


I don't know if this one needs to be broken out separately, but if it does you have to list which files and directories you don't want accessed because this bot won't read the second entry.

The second entry you posted looks okay and is what the other googlebots should obey. If it's one of them triggering sentinel and you trace the IP back to their company, I recommend contacting Google and tell them you think their bot is misbehaving. They may need to adjust their software or there might be something in your robots.txt which wasn't posted that's throwing it off.
 
View user's profile Send private message
boar
PostPosted: Tue Jul 19, 2005 8:51 am Reply with quote

ok..i guess i should search some more.
 
grantb
Regular
Regular


Joined: Feb 16, 2005
Posts: 67
Location: Canada

PostPosted: Tue Jul 19, 2005 11:35 am Reply with quote

I had the same problem as well. The adsense I was using on the site was on everypage and even tho the robots.txt file told all bots to not access admin.php and folder, googlebot still tried to crawl the url's that I had visited while logged in as admin. I mean everytime, login as admin and goto say modules admin, then log out and bamn, googlebot banned for trying to access the exact page. So eventually after trying many things to prevent googlebot (any any other crawler) from accessing any admin url's I went extreme. I placed a allow deny rule in .htaccess to block any ip's except for ones that I list. Yes people might say this is a bit radical, but for me, it is worth it, even tho I have to change the ip's once and a while.
Code:
<Files "admin.php">

  Order Deny,Allow
  Deny from all
  Allow from xx.xx.xx.xx
  Allow from xx.xx.xxx.xxx
</Files>

_________________
Only registered users can see links on this board! Get registered or login! 
View user's profile Send private message
sonicman
PostPosted: Tue Jul 19, 2005 7:04 pm Reply with quote

Dauthus
we discussed this on your site, all I would have to do is add the google bots ips in the protected ip range right
 
Dauthus
PostPosted: Tue Jul 19, 2005 7:41 pm Reply with quote

sonicman wrote:
Dauthus
we discussed this on your site, all I would have to do is add the google bots ips in the protected ip range right


Yep.
 
VinDSL
PostPosted: Tue Jul 19, 2005 8:10 pm Reply with quote

This is my theory...

When Google bots spider your site, under normal conditions, I believe they follow the rules. However, many ppl, including admins, hackers, et cetera, are running Google Toolbar in their browser, with PR (PageRank) enabled. When you go to your admin page, a hacker tries to run an exploit against your site, or whatever, and PR is enabled, Google sends a bot to check out the URI. It doesn't matter if it's you accessing your admin page, or a hacker running a UNION exploit, Google (PR) bots will blindly pick up the link and run with it.

The point I was trying to make, above - and I guess I didn't state it clearly enough - even if a Google bot accesses your admin panel, what's it going to do? Answer: sit there... Laughing

Personally, I think it's a bad form to ban Google bots, even if they're acting silly - accessing your admin panel, running exploits against you, or whatever - but that's just me. I've never had a spider crack my site yet, knock on wood...
 
sonicman
PostPosted: Tue Jul 19, 2005 8:57 pm Reply with quote

vinDSL
Cool I will just add the google bots IP address range in sentinel

Thanks
 
boar
PostPosted: Wed Jul 20, 2005 2:49 am Reply with quote

VinDSL, i am not using any type of third-parties toolbars or anything other than a clean browser. Now, my site is still under consctruction, no one knows about it besides few ppl.

I have emailed google, let's see what they have to say about it.

Everyone with Platinum 7.6 (if that matters) and Sentinel having problems with googlebot please raise your hand! only few ppl? mm how come others dont have this problem? :/

for now, i have set the Admin blocker to Email and Redirect everyone to the main site...i havent had any new emails YET!. i hope they didnt removed google's bot from crawling my site, i hope it's temporary if they did.
 
boar
PostPosted: Thu Jul 21, 2005 10:59 am Reply with quote

well, googlebot isnt crawling my site anymore thanks to NukeSentinel!.

i will stop complaining now.
 
Manuel
Regular
Regular


Joined: May 28, 2005
Posts: 90

PostPosted: Thu Jul 21, 2005 11:17 am Reply with quote

i have another question: why rename admin.php while in robots.txt the new name is visible to the world? :/

_________________
Only registered users can see links on this board! Get registered or login! 
View user's profile Send private message Send e-mail Visit poster's website MSN Messenger
boar
PostPosted: Thu Jul 21, 2005 11:48 am Reply with quote

i was thinking the same exact thing for a couple of months now.. i havent asked that the question... i guess few people know about robots.txt file? i know..
 
Display posts from previous:       
Post new topic   Reply to topic    Ravens PHP Scripts And Web Hosting Forum Index -> NukeSentinel™

View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum


Powered by phpBB © 2001-2007 phpBB Group
All times are GMT - 6 Hours
 
Forums ©