Ravens PHP Scripts: Forums
 

 

View next topic
View previous topic
Post new topic   Reply to topic    Ravens PHP Scripts And Web Hosting Forum Index -> General/Other Stuff
Author Message
grantb
Regular
Regular


Joined: Feb 16, 2005
Posts: 67
Location: Canada

PostPosted: Fri Jan 27, 2006 5:06 pm Reply with quote

Hello all, I havent posted much lately but I recently came back from holidays and found a huge drain on my site from a robot called picsearch.com. This site's robots used up over 1gig of bandwidth searching my amazon module for.. i guess pics..lol in one month!! Well any site that does not limit how much they pull from me is not a robot that I want.
I have added a entry to my .htacces to deny any hits from picsearch.com.. but my question is this.. How can i add this site and all it's robots to my .htaccess without doing a general entry like this:

Code:
deny from picsearch.com


What I am worried about is that apache will have to try and resolve to the domain name for every visitor to the site, and from what i understand is that this slows down apache. I can find an ip but will this block all spiders from picsearch?.. eg. spider1.picsearch.com, spider2.picsearch.com etc

Thanks in advance, oh and Raven if this is in the wrong place please move, thanks again.

_________________
Only registered users can see links on this board! Get registered or login! 
View user's profile Send private message
Raven
Site Admin/Owner


Joined: Aug 27, 2002
Posts: 17077

PostPosted: Fri Jan 27, 2006 5:28 pm Reply with quote

wrote:
Information for domain picsearch.com
IP: 217.212.245.198 [get RIPE/ARIN IP info] (Resolved: Only registered users can see links on this board! Get registered or login!
Reverse IP: there are 2 domains on this IP, click here to get them all
Name Server: dns5.telia.com Found 966 domains, hosted on this NS
ns3.picsearch.com Found 2 domains, hosted on this NS
(click to any NS server to get all domains, hosted on this NS)
IP Location: Sweden [SE] - Stockholm
IP owner: TeliaSonera AB
IP assigned to: Picsearch AB
Domain status: REGISTRAR-LOCK
Domain Registrar: NETWORK SOLUTIONS, LLC.
Created: 15-mar-1999
Expires: 15-mar-2009


Here are the IP's/CIDR assigned to that domain name: 217.212.224.0/19

I would add deny from 217.212.224.0/19 instead
 
View user's profile Send private message
Susann
Moderator


Joined: Dec 19, 2004
Posts: 3191
Location: Germany:Moderator German NukeSentinel Support

PostPosted: Fri Jan 27, 2006 5:33 pm Reply with quote

Add PicSpider/1.1 to your harvester blocker and write to htaccess.
This is only one 1 IP:inetnum: 217.212.224.128 - 217.212.224.255
netname: SE-PICSEARCH
More info: Only registered users can see links on this board! Get registered or login!
 
View user's profile Send private message
grantb
PostPosted: Sat Jan 28, 2006 12:21 am Reply with quote

thank you both.. Smile
 
djmaze
Subject Matter Expert


Joined: May 15, 2004
Posts: 719
Location: http://tinyurl.com/5z8dmv

PostPosted: Sat Jan 28, 2006 7:23 am Reply with quote

Raven wrote:
Here are the IP's/CIDR assigned to that domain name: 217.212.224.0/19

Incorrect ban because that bans the whole network

Susann wrote:
217.212.224.128 - 217.212.224.255


So the correct CIDR should be: 217.212.224.128/25
 
View user's profile Send private message Visit poster's website
Raven
PostPosted: Sat Jan 28, 2006 10:27 am Reply with quote

I did that intentionally, DJ, because their Lookup revealed that they could use any of the band, but it's his choice.
 
djmaze
PostPosted: Sat Jan 28, 2006 11:30 am Reply with quote

I understand but if you lookup ripe Only registered users can see links on this board! Get registered or login! that is owned by TradeDoubler AB

If you check the /19 CIDR you will notice you ban a lot more then the range specified.
The CIDR specified by RIPE is meant as being anything in sweden aka Top-Level

217.212.224.0/19 = 217.212.224.0 - 217.212.255.255

As you see /19 is the wrong here.

Here's some code you can use to check it yourself Wink
Code:
<?php

$ip = '217.212.224.0';
$cidr = 19;

$ip = ip2long($ip);
$mask = 0xffffffff << (32 - $cidr);
echo "$ip - ".long2ip($ip - $mask);
 
Raven
PostPosted: Sat Jan 28, 2006 12:19 pm Reply with quote

DJ, I completely understand how to calculate CIDR's, but thanks. Yours is the more precise/correct CIDR to use. When I started looking up the 900+ domains in their name servers, I just went for overkill. Use the /25 as DJ has mentioned.
 
Susann
PostPosted: Sun Jan 29, 2006 8:01 am Reply with quote

Thanks Djmaze
For those who have problems to understand the cidr´ s I found a helpful site with examples and some pictures for better understanding.
Only registered users can see links on this board! Get registered or login!
 
djmaze
PostPosted: Sun Jan 29, 2006 8:19 am Reply with quote

Here's some more info Wink
Only registered users can see links on this board! Get registered or login! Only registered users can see links on this board! Get registered or login!
 
Display posts from previous:       
Post new topic   Reply to topic    Ravens PHP Scripts And Web Hosting Forum Index -> General/Other Stuff

View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum


Powered by phpBB © 2001-2007 phpBB Group
All times are GMT - 6 Hours
 
Forums ©