a Couple problems with NukeSentinel_242_71-79

Posted: Sun Dec 04, 2005 4:42 pm

I've basically had to disable two of NukeSentinel features because of issues with them.

I have the phrase "Alison Krauss and Union Station" as the topic of many articles. Users who try and add comments, vote or whatever always get the UNION block. Thus UNION is disabled to allow the site to be friendly to users.

I had to disable SCRIPT blocking due to JavaScript being allowed in Advertising banners. If you attempt to add a banner with this enabled, you'll get a SCRIPT block.

I would like to have these enabled but I can't find a way to do this.

DNSStuff.com has banned the site because of too many requests. How can I either change this or disable it? I changed the setting in the configuration but the site still tries to use DNSStuff for all messages back to the admin. Is there a work around or ???

Finally, is there anyway to keep search engine bots from trying to access admin.php?.... pages? GoogleBot is the worst but Yahoo! does too. They all seem to ignore the robots.txt file and since the standard for robots.txt doesn't really adequately address wildcards like "*admin*" or "*amazon*," there doesn't seem to be a way to keep them out. Thus, google always gets blocked and then their AdSense program fails.

I need to be protected but right now, Sentinel seems to be causing more problems than it is supposed to.

Any help or ideas would be greatly appreciated.

Bill

Site Admin/Owner Joined: Aug 27, 2002 Posts: 17088

First of all, I have an email from dnsstuff.com dated 12/1/2005 and it follows

Quote:

You are correct, this is a perfectly acceptable use of our site. Smile

-Scott

Gaylen wrote:
>
>
>
> Are any automated programs OK? What about extensions/widgets/addons?
>
> Some are OK. In general, if the program makes it clear that the information comes from [ Only registered users can see links on this board! Get registered or login! ] <http://www.DNSstuff.com>, the output is displayed to the user in some way, and the users of the program have to enter the information manually (so they can't, for example, get information on a whole list of IPs automatically), it is OK. But before distributing such a program, please contact us, so that we can verify that it is OK (and perhaps assist you, and add a link to your program).
>
>
>
> Thank you for a great service! I am one of the main authors and contributors to a product called NukeSentinel(tm) and admin/owner of the Main support site. It is the most popular security addon for phpNuke. We offer 2 options for whois lookup - dnsstuff.com and arin.net. There is absolutely no method to look up more than one IP at a time. The application does generate hyperlinks that will link directly to dnsstuff.com and/or arin.net. We make no attempt to screen scrape or any other such tactics that would mask where the information is coming from. The webmaster has the option of allowing the public to see the links or not. The admin, of course, always sees the links.
>
> Having read the above from your FAQ, it seems as if we ARE in compliance with your automated policy. But, I am requesting that you review this and advise. Obviously, the application drives many people to your site which does open up potential for referral clicks to Google and any other advertisers, so I hope that we can continue to offer your services to our users.
>
> Thank you for considering this,

So, if you have been banned, discuss it with them. I get many, many crack attempts and emails and have never been banned, so i'm not sure what your issue is.

I have script in some advertising banners and I don't have any problems with NukeSentinel(tm). I would suggest just emailing admin for script security. That way you can look at each one and it doesn't affect your clients.

Yes, the UNION would cause a problem. There is no work-around at the moment. UNION passed in a HTTP GET request will cause a trap. You could rewrite the nuke code to use POST instead of GET. That should fix the issue.

In robots.txt use these 3 lines. It works for me.
User-agent: *
Disallow: /admin/
Disallow: /admin.php

Posted: Sun Dec 04, 2005 6:54 pm

Thanks for the quick response!!

OK, for now I'll leave the UNION turned off.

My Robots.txt has the following:
User-agent: *
Disallow: /admin.php?*
Disallow: /modules.php?name=Amazon
Disallow: /admin/
but google slams me on all of these anyway. I've been trying to work it out with them for over a year. They fix it for a day or two and then it takes months for them to fix it again. I'll keep bugging them about it.

I am the admin and I get the script notices when attempting to enable javascript banners. For now, I just always turn off script blocking, insert the ad, turn it back on. I've set up the admins, protected my IP but I still get these. It is an annoyance but one I can live with.

I'll contact DNSStuff and see what they say. The following is on my admin e-mails from Sentinel:

Quote:

DNSStuffDNSStuffSorry, you have triggered our rate limiting system. If you are reading this in a web browser, we apologize -- we want you to use the site as much as you like. What we do not like is when people use automated programs with our free service. We have the addresses [ Only registered users can see links on this board! Get registered or login! ] and [ Only registered users can see links on this board! Get registered or login! ] here in case spammers are harvesting addresses from our site. Please go here for more details. Your IP is 209.68.1.137. Thanks!

Thanks a lot!
Bill

Posted: Sun Dec 04, 2005 7:02 pm

Run your robots.txt thru [ Only registered users can see links on this board! Get registered or login! ] . It is not valid. You have many errors Smile

Posted: Sun Dec 04, 2005 8:10 pm

Raven,

Just curious, do you see anything wrong with the following robots.txt or would you recommend removing or adding any statements?

User-agent: *
Disallow: /admin/
Disallow: /admin.php
Disallow: /images/
Disallow: /includes/
Disallow: /themes/
Disallow: /blocks/
Disallow: /modules/
Disallow: /language/

TIA,
montego

Posted: Sun Dec 04, 2005 8:27 pm

I'd add
/db/
/cgi-bin/

Add everything explicitly that you don't want advertised.

Posted: Sun Dec 04, 2005 10:48 pm

Yes, I was aware of these. All of those changes were suggested by Google in an attempt to keep their bots out of places they shouldn't be.

I've removed the offending lines but, I know it won't make any difference since the compliant file was also ignored by them.

Thanks again!
Bill

Posted: Sun Dec 04, 2005 10:58 pm

Make sure that you don't have any other lines other than this one User-agent: * at the top. Then after that, only your Disallow lines. Don't use any Google or Googlemedia* overrides.

Posted: Mon Dec 05, 2005 8:58 am

Thanks. I've reduced my robots.txt to:
User-agent: *
Disallow: /admin.php?
Disallow: /admin/
Disallow: /blocks/
Disallow: /modules.php?name=Amazon
Disallow: /db/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /Images/
Disallow: /includes/
Disallow: /themes/
Disallow: /language/
Disallow: /modules/

This is pretty much where I started. I added a lot of stuff per Google's request that didn't work so maybe this will.

Thanks!!
Bill

Posted: Mon Dec 05, 2005 10:02 am

I am suprised to learn google has ignored your robots.txt I have never had an issue with them at all in 2 years - unlike inktomi / slurp which, I have found, a scourge when it gets a grip.

Client Joined: Jul 18, 2003 Posts: 977

Raven wrote:

Make sure that you don't have any other lines other than this one User-agent: * at the top. Then after that, only your Disallow lines. Don't use any Google or Googlemedia* overrides.

So you are saying do NOT use this at the top?

Code:

User-agent: Mediapartners-Google*


Disallow:

Posted: Mon Dec 05, 2005 1:47 pm

blith, yes, that is what Raven is saying. I have had the same two lines in my robots.txt file for a long time now and was wondering why I haven't been showing up in Google! I am wondering now if these two lines have been screwing me up this whole time. I will see in the next few weeks if things improve any.