Ravens PHP Scripts: Forums
 

 

View next topic
View previous topic
Post new topic   Reply to topic    Ravens PHP Scripts And Web Hosting Forum Index -> NukeSentinel(tm)
Author Message
ring_c
Involved
Involved



Joined: Dec 28, 2003
Posts: 276
Location: Israel

PostPosted: Mon Nov 08, 2004 6:29 am Reply with quote

Any idea why this happened? all the user did (according to what I can see), was running index.php under the main site. Or am I missing something?

Date & Time: 2004-11-08 06:41:57
Blocked IP: 80.230.116.*
User ID: not registered (1)
Reason: Abuse-Harvest
String Match: microsoft url control
--------------------
User Agent: Microsoft URL Control - 6.00.8862
Query String: hagigim.com/index.php
Forwarded For: none
Client IP: none
Remote Address: 80.230.116.151
Remote Port: 21059
Request Method: GET
 
View user's profile Send private message Visit poster's website
ring_c







PostPosted: Mon Nov 08, 2004 6:32 am Reply with quote

This happened 10 minutes ago. I've unblocked the ip, thinking it was safe, and now I get this:

Date & Time: 2004-11-08 07:23:35
Blocked IP: 80.230.116.*
User ID: not registered (1)
Reason: Abuse-Harvest
String Match: microsoft url control
--------------------
User Agent: Microsoft URL Control - 6.00.8862
Query String: [ Only registered users can see links on this board! Get registered or login! ]
Forwarded For: 80.230.116.151
Client IP: none
Remote Address: 80.230.116.151
Remote Port: 47880
Request Method: GET

Am I being hacked or what?!

PS: am using Sentinel 2.1.0, for about a month in which it blocked a Chinese IP, and nothing since till today. Could anyone explain?
 
VinDSL
Life Cycles Becoming CPU Cycles



Joined: Jul 11, 2004
Posts: 614
Location: Arizona (USA) Admin: NukeCops.com Admin: Disipal Designs Admin: Lenon.com

PostPosted: Mon Nov 08, 2004 6:45 am Reply with quote

ring_c wrote:
Am I being hacked or what?!

It's hard to tell without looking through your logs, but most likely it's a spam bot looking for a formmail to send out spam. Either that or it's a snoop bot trying to collect email addies from your site. It just depends on how it's setup...

_________________
.:: "The further in you go, the bigger it gets!" ::.
.:: VinDSL's Lenon.com | The Disipal Site ::. 
View user's profile Send private message Visit poster's website ICQ Number
Nukeum66
Life Cycles Becoming CPU Cycles



Joined: Jul 30, 2003
Posts: 551
Location: Neurotic, State, USA

PostPosted: Mon Nov 08, 2004 8:15 am Reply with quote

microsoft url control is not listed in the Sentinel Harvesters list for nothing. In my opinion I would add the IP back in to your blocked list.

_________________
Scott Johnson MIS Ubuntu/Linux 11.10 
View user's profile Send private message Visit poster's website
sixonetonoffun
Spouse Contemplates Divorce



Joined: Jan 02, 2003
Posts: 2496

PostPosted: Mon Nov 08, 2004 8:57 am Reply with quote

Anytime you see Reason: Abuse-Harvest visit Ravens Agent Inspector page and paste the User-Agent in and click submit to see if it matches a default trapped user-agent.

In this case you would have seen:
Agent: microsoft url control is trapped by this Harvester entry: microsoft url control

_________________
[b][size=5]openSUSE 11.4-x86 | Linux 2.6.37.1-1.2desktop i686 | KDE: 4.6.41>=4.7 | XFCE 4.8 | AMD Athlon(tm) XP 3000+ | MSI K7N2 Delta-L | 3GB Black Diamond DDR
| GeForce 6200@433Mhz 512MB | Xorg 1.9.3 | NVIDIA 270.30[/size:2b8 
View user's profile Send private message
ring_c







PostPosted: Mon Nov 08, 2004 9:02 am Reply with quote

sixonetonoffun wrote:
Anytime you see Reason: Abuse-Harvest visit Ravens Agent Inspector page and paste the User-Agent in and click submit to see if it matches a default trapped user-agent.

In this case you would have seen:
Agent: microsoft url control is trapped by this Harvester entry: microsoft url control

Now, this was Chinese to me. sorry.
Anyway, should I realy re-enter the IP to the list of blocked addresses?
 
64bitguy
The Mouse Is Extension Of Arm



Joined: Mar 06, 2004
Posts: 1164

PostPosted: Mon Nov 08, 2004 9:13 am Reply with quote

He was saying that Raven has an "agent inspector" here on this site (See the menu block!) where you can copy in the "Agent Reason" information that was emailed to you.

It will then tell you if this was a valid agent blocking function.

Quite frankly though, this looks like a harvest agent slurping your website to steal email addresses and images, so yes, this was a valid action by NukeSentinal that protected your site from abusive "Harvest" functions. NukeSentinel did not block a user, but rather an automated process being used by someone to steal information.

Just as a heads up, your site is not compatible with Firefox or Mozilla browsers which I would consider to be a serious issue.

Hope this helps.

_________________
Steph Benoit
100% Section 508 and W3C HTML5 and CSS Compliant (Truly) Code, because I love compliance. 
View user's profile Send private message
ring_c







PostPosted: Mon Nov 08, 2004 9:20 am Reply with quote

64bitguy wrote:
Just as a heads up, your site is not compatible with Firefox or Mozilla browsers which I would consider to be a serious issue.

Hope this helps.

Thanks. I'm using Php-Nuke v6.7, how can I fix it then?
 
Raven
Site Admin/Owner



Joined: Aug 27, 2002
Posts: 17088

PostPosted: Mon Nov 08, 2004 9:39 am Reply with quote

Running that agent string in the Agent Inspector you get

Agent: microsoft url control is trapped by this Harvester entry: microsoft url control

If you want to allow it then delete that entry from your Harvester settings in NukeSentinel.
 
View user's profile Send private message
ring_c







PostPosted: Mon Nov 08, 2004 9:58 am Reply with quote

Raven wrote:
Agent: microsoft url control is trapped by this Harvester entry: microsoft url control

But what does it mean? Is it harmfull/harmless/good/bad?
I'm cluless...
 
Raven







PostPosted: Mon Nov 08, 2004 10:10 am Reply with quote

It is a known harvester that is not trusted/wanted. Do a google search for more information.
 
Nukeum66







PostPosted: Mon Nov 08, 2004 11:04 pm Reply with quote

Just add the IP back to the ban list . You don't need that User-Agent on your site... Wink
 
ring_c







PostPosted: Tue Nov 09, 2004 12:02 am Reply with quote

Raven wrote:
It is a known harvester that is not trusted/wanted. Do a google search for more information.

Thanks, Raven. I did.
Is there a way to deny this agent to run? Or maybe Sentinel is the best solution, and I should let it to the job like it did this time?
 
Raven







PostPosted: Tue Nov 09, 2004 12:09 am Reply with quote

Just let Sentinel stop it. You could use .htaccess to deny it also. That way it never makes it to your site.
 
ring_c







PostPosted: Tue Nov 09, 2004 12:31 am Reply with quote

Raven wrote:
Just let Sentinel stop it. You could use .htaccess to deny it also. That way it never makes it to your site.

Do you think I should? If so, could you please guide me how to do it?
 
ring_c







PostPosted: Tue Nov 09, 2004 2:14 am Reply with quote

I've just got this one. I guess I should check your agent inspector... Anything you might add?

Date & Time: 2004-11-09 02:08:05
Blocked IP: 209.167.50.22
User ID: not registered (1)
Reason: Abuse-Harvest
String Match: linkwalker
--------------------
User Agent: LinkWalker
Query String: [ Only registered users can see links on this board! Get registered or login! ]
Forwarded For: none
Client IP: none
Remote Address: 209.167.50.22
Remote Port: 45972
Request Method: GET
 
ring_c







PostPosted: Tue Nov 09, 2004 2:24 am Reply with quote

ring_c wrote:
I've just got this one. I guess I should check your agent inspector...

I got this:
Code:
Agent: LinkWalker is trapped by this Harvester entry: linkwalker


Now, what does it tell me? How can I tell if it's good or bad?
 
ring_c







PostPosted: Tue Nov 09, 2004 2:27 am Reply with quote

Ok, being googling a little, and found a good (hopefully) database of web robots, which might be helpfull to some of us, here: [ Only registered users can see links on this board! Get registered or login! ]

There, I could find a link to LinkWalker's tech page, here: [ Only registered users can see links on this board! Get registered or login! ]

Yet again, I can't realy tell if this kind of harvest is good or bad. how do you do that?
 
blith
Client



Joined: Jul 18, 2003
Posts: 977

PostPosted: Tue Nov 09, 2004 9:15 am Reply with quote

Harvest is generally bad. There is a reason they are in the Sentinel harvest blocker list. If the makers of Sentinel deem they are bad then I will listen. Anything in that list... leave it. Block 'em all. Crawlers are a different story, sites want some crawlers to increase traffic. The good crawlers are not in the list...
 
View user's profile Send private message Visit poster's website
Raven







PostPosted: Tue Nov 09, 2004 12:09 pm Reply with quote

There are harvester lists if you google for them. We include the 'master' list, if you will. That's why YOU make the final decision Wink
 
64bitguy







PostPosted: Tue Nov 09, 2004 1:41 pm Reply with quote

I spent an entire day reviewing the Harvester List provided by NukeSentinel when I first loaded it. I did this because I've been maintaining my own harvester list in my existing .htaccess file as well as some of the other Programs that I used in conjunction with my site.

I was rather surprised that NukeSentinel's list not only contained all of the standard Nuke recommended to block harvesters, but also a few others that I hadn't heard of yet. On the other side of the coin, I had a few on my list that were not yet in NukeSentinel, which I promptly added.

My feeling (after exhaustively researching the list of included harvesters for blocking) is that the list is pretty extensive and thorough. It blocks the majority of Harvesters that are designed to extract email addresses, images and other proprietary information from websites. For those not familiar, this information would most likely extracted with abusive intentions to steal or hotlink your resources and to SPAM you and your users. The Federal Trade Commission estimates that well over 80% of all SPAM is directly attributable to website harvesting. This fact alone should be enough to convince people to use this valuable NukeSentinel feature to block all harvesters.

See: [ Only registered users can see links on this board! Get registered or login! ]

The FTC replicated this test in 2004 with the same results and still attribute 85+% of all SPAM to originate from automated harvesters employed by Spammers.

See: [ Only registered users can see links on this board! Get registered or login! ]

While Raven points out that "YOU make the final decision" on how to deal with harvesters (as well as the list you keep inside NukeSentinel) I would recommend from personal experience that you maintain the existing list of abusive harvesters and even add to that list as you discover additional abuses.

As pointed out above, harvesting is different than "spidering" or "crawling" your site. My experience has been that harvesting always results in abuse, whereas "spidering" or "crawling" is simply a method used by search engines to index data for productive purposes. Of course this also is not always the case, as there are also abusive robots and spiders. Again, it is up to YOU to decide who you want spidering or crawling your site.

With a few tools, you'll often find that SPAM is a direct result of an unknown harvester "having its way" with your site and through your data.

One of the tools I employ is Visual Route ( See [ Only registered users can see links on this board! Get registered or login! ] ) to determine where SPAM is really coming from. With a little dilligence you can figure out what harvester they are using and block it. You can also employ some creative mail management techniques to prevent further abuse. I (For example) "blacklist" the abuser in my server hosted Spam Assassin and "bounce" the emails back. I also ban their IPs and report the abuse to the Host and ISP as well as Operation Web Snare. (See [ Only registered users can see links on this board! Get registered or login! ] ) If you attempt abuse on my site, chances are pretty good that the FBI will be monitoring your activities in short order. A few of the recent SPAM busts are a good example of successes. I believe we will be seeing many more arrests in the near future as CAN-SPAM ( See [ Only registered users can see links on this board! Get registered or login! ] ) starts to be enforced.

The long and short of it? Use NukeSentinel to protect your resources from Harvesters that would abuse you and your users and Add to the list, don't reduce it. Finally, take actions to protect email addresses on your site.

1) Never publish an email address ANYWHERE on your site.
2) Never include non-spam proof email addresses in any program you create. (use "joe at joes dot com" and not [ Only registered users can see links on this board! Get registered or login! ]")
3) Enable phpbb's function to "User email via board" (Admin/Forums/Configuration) which will hide user email addresses normally visible in "Userinfo" and force mail sent to your users to go via your site and "Private Messenging"
4) Maintain a good Harvester List in NukeSentinel and .htaccess to keep them out of your data.

Hope this helps.
 
Raven







PostPosted: Tue Nov 09, 2004 1:52 pm Reply with quote

Excellent! And I would Enable phpbb's function to "User email via board" and have tried umpteen times. But, alas, to no avail. It does not work and I have had others look at it.
 
64bitguy







PostPosted: Tue Nov 09, 2004 2:09 pm Reply with quote

hmmmm... It used to work here... In fact this is the site where I discovered I didn't have mine enabled. I dunno what happened in that regard. I'm using 2.0.10, but yours for some reason is reporting the old 2.0.6, though I thought it had been patched to a later version.

One thing I have noticed is that your GT link to profiles is different than mine. My link to profile data is a little shorter (SITENAME-forum-userprofile-USERID#.html) and I have disabled the next layer functions in GT (Removed the in/out definitions) and it's working fine. Basically, I didn't want that data GT'd anyway, that's why I did that.

I'm more than happy to share my .htaccess data and in/outs if you think it might help.
 
Raven







PostPosted: Tue Nov 09, 2004 4:02 pm Reply with quote

I just reactivated the email via board. Try to send me an email.
 
64bitguy







PostPosted: Tue Nov 09, 2004 4:29 pm Reply with quote

Sent, but I should note that a copy of the sent message does not appear in either my sentbox or outbox, though that option was checked
 
Display posts from previous:       
Post new topic   Reply to topic    Ravens PHP Scripts And Web Hosting Forum Index -> NukeSentinel(tm)

View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum


Powered by phpBB © 2001-2007 phpBB Group
All times are GMT - 6 Hours
 
Forums ©