How to accept libwww-perl from special sites ?

Susann · Posted: Sat Jul 23, 2005 6:30 pm

NukeSentinel banned systran when user tried to translate our downloads with the systran box, because libwww is in the harvester blocker. Possible we lost some interested user through this action, but doesn´t matter. There are only a few people who translated the downloads. So the question is:

How to block libwww-perl but except from systran ?

Code:

You have been blocked from duck ring this site.

You of acres using A possible Harvester on this site.
User agent: Mozilla/5.0 (Windows; U; Win 9x 4.90; De-DE; rv:1.7.10; libwww-perl/5.76; SYSTRAN) Gecko/20050717 Firefox/1.0.6
Remote ADDRESS: 66.185.171.204
Client IP: none
Forwarded For: 66.185.171.195

64bitguy · Posted: Tue Jul 26, 2005 11:02 pm

I'm not sure if you do one without the other.

Im my case, I usually leave the libwww-perl harvester disabled (removed from Sentinel Blocking). I personally haven't found any abuse because of that action. You have to keep in mind that W3C Validators also use that function, so on top of everything else, you can't do any web based validation with it enabled.

Susann · Posted: Wed Jul 27, 2005 5:15 pm

I know your opinion about libwww-perl and the Harvester Blocker, but really I´m not worried about the validator. I have my own opinion about this based onto different requests with UA libwww-perl.

Thats only one of our blocked IP´s. Maybe thats not the best example.

Code:

Datum & Uhrzeit: 2005-07-05 18:41:22 CEST GMT +0200
Gesperrte IP: 62.112.150.205
Benutzer-ID: Unbekannt (1)
Grund: Abuse-Harvest
String-Übereinstimmung: libwww-perl
--------------------
User-Agent: libwww-perl/5.69
Query-String: my-site.de/modules.php?name=Downloads&d_op=viewdownload&cid=5
Get String: my-site.de/modules.php?name=Downloads&d_op=viewdownload&cid=5
Post String: my-site.de/modules.php
Weitergeleitet für: none
Client-IP: none
Entfernte Adresse: 62.112.150.205

What are they looking for with this User Agent ?

Are you 100 % shure that there isn´t any risk as you always said ? I have my doubt.
As I wrote above I would prefer to remove libwww-perl from the Harvester Blocker but I would like then to know the rules for my .htaccess (incl.Systran).

If this works with one address it should probably also work with several adresses.

houstonguy · New Member Joined: Jul 26, 2005 Posts: 19

wow thats one strong blocker lol

64bitguy · Posted: Thu Jul 28, 2005 11:05 pm

libwww-perl is a user agent. It is used to scan for data and is used by some agent services and search engines. It doesn't inject data, it harvests it. Thus, it is not a danger to your domain.

hitwalker · Sells PC To Pay For Divorce Joined: Posts: 5661

well that leaves the question why its blocked standard in sentinel,i mean if it cant do any harm,specially now suddenly the w3 compliance is becoming a big deal.

Nukeum66 · Posted: Fri Jul 29, 2005 6:39 am

Maybe libwww-perl was viewed as being a domain resource abuser.

hitwalker · Sells PC To Pay For Divorce Joined: Posts: 5661

well i did some checking,weird thing is ...for fun i used the the validator and infact w3 got banned.
but weird thing is i cannot find its ip.

it sure can be usefull to have your site validated but personaly i dont think its a good idea for nuke.
We have to much stuff added and always have to look out for what we add to the site.
I know from when i was using wordpress.

Susann · Posted: Fri Jul 29, 2005 8:20 am

The question about libww-perl as an standard blocker was already answered in 2004 by Raven.

Only registered users can see links on this board!
Get registered or login to the forums!

I found an example for using htaccess I´ve only to test this. Than I`ll remove libww-perl from the Harvester Blocker.

Nukeum66 · Posted: Fri Jul 29, 2005 8:23 am

If you could find the ip of the validator you could protect it via NukeSentinel. In my opion having an HTML compliant site Nuke or not, is vary important. And we sould make it a priority to check all blocks, modules, theme, and even our content.
Well I'm running offtopic, so I'll stop now! ..... Smile

64bitguy · Posted: Fri Jul 29, 2005 8:34 am

Susann wrote:

The question about libww-perl as an standard blocker was already answered in 2004 by Raven.

Only registered users can see links on this board!
Get registered or login to the forums!

I found an example for using htaccess I´ve only to test this. Than I`ll remove libww-perl from the Harvester Blocker.

Yeah, back when I didn't know what the heck was going on, I had that problem with it.

Again, it is a harverster. It does not inject nor damage your site.

Regardless, I'm glad you've found what you are looking for.

hitwalker · Sells PC To Pay For Divorce Joined: Posts: 5661

well Nukeum66,im not that worried if a site is HTML compliant.
fact is,were using a cms in php and that will never validate unless you turn it around.
as for sentinel,making sure that w3 isnt banned is just a must for those that running HTML compliant nuke sites.
As for all the others,simple said doesnt make any difference.
but thats all personal... Smile