WebCrawler Access admin areas

Regular Joined: Feb 16, 2005 Posts: 67 Location: Canada

Ok I really appreciate all the help and security updates I have been getting from this site, you guys are the nuke-gods!..

Here is something that has me concerned about the fact that searchbots have been getting banned by sentinel for trying to access admin links. I have been getting searchbot bans everytime that I log in as administrator of a site.

My question is.. How can this searchbot be trying to access the links available to me as an admin when it shouldn't see these links(url's) at all?

Code:

Reason: Abuse-Admin 


-------------------- 


User Agent: Mediapartners-Google/2.1 


Query String: X.com/admin.php?op=ABBlockedIPDelete&xIPs=66.xxx.xx.52&min=0&am p;column=date&direction=desc&xop=ABBlockedIP 


Forwarded For: none 


Client IP: none 


Remote Address: 66.249.xx.xxx 


Remote Port: 44144 


Request Method: GET

And:

Code:

Reason: Abuse-Admin 


-------------------- 


User Agent: Mediapartners-Google/2.1 


Query String: X.com/admin.php?op=module_edit&mid=37 


Forwarded For: none 


Client IP: none 


Remote Address: 66.249.xx.xxx 


Remote Port: 37678 


Request Method: GET

And for my robots.txt (which is last on my mind considering admin links are somehow showing or being accessed by other users when I am logged in as admin)I have this for the admin area:

Quote:

User-agent: *
Disallow: admin.php
Disallow: /admin/

Anyways I would really appreciate any kind of suggestions as to what is going on with that site. Thanks

Client Joined: Jan 29, 2004 Posts: 624

I dunno how they 'see' it but try this in your .htaccess file both in your root and in your admin directory

Code:




RewriteEngine on


RewriteCond %{HTTP_USER_AGENT} ^.*Ants.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*attach.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*autoemailspider.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*BackWeb.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Bandit.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Buddy.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*CherryPicker.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*CherryPickerElite.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Copier.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Crawler.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Crescent\Internet.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Collector.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*DA.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*DiaGem.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*DISCo\Pump.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Downloader.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Drip.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*eCatch.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*EirGrabber.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*EmailCollector.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*EmailExtractor.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*EmailSiphon.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*EmailWolf.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*ExtractorPro.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*EyeNetIE.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*FileHound.*$


RewriteCond %{HTTP_USER_AGENT} ^.*FlashGet.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*GetRight.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Gets.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*gotit.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Grabber.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*GrabNet.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Grafula.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Go-Ahead-Got-It.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Go!Zilla.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*HMView.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*HTTrack.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*IBrowse.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*lftp.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*InterGET.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Internet\Ninja.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Iria.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*JetCar.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*JOC.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*JustView.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*leech.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*likse.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Likse.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*LNSpiderguy.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Magnet.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Mag-Net.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Memo.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Mirror.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Mister\PiX.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*MIDown\tool.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Navroad.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*NearSite.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Netprospector\JavaCrawler.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*NetSpider.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*NICErsPro.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Ninja.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Nutsack.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Offline.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Offline\Explorer.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*PageGrabber.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Papa\Foto.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*pcBrowser.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Pockey.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Pump.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Reaper.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Recorder.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*ReGet.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Siphon.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Slurp.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*SmartDownload.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Snagger.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Snake.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*SpaceBison.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Stripper.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Sucker.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*SuperBot.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*SuperHTTP.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Surfbot.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*tAkeOut.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Teleport.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Vacuum.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*WebAuto.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*WebCollector.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*WebCopier.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*WebFetch.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*WebReaper.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*WebSauger.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Webster.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*WebStripper.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*WebWhacker.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*WebZIP.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Web\Image\Collector.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Web\Magnet.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Web\Sucker.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Wget.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Whacker.*$ [OR]


RewriteCond %{HTTP_USER_AGENT} ^.*Widow.*$ [OR]


RewriteRule /* http://www.google.com [L,R]

The last line is where you want to redirect the bot. Just put the useragent of the bot you want to stop if its not listed.

Posted: Thu Feb 17, 2005 12:18 am

Thank you very much! Now why didn't I think of that.. lol

Posted: Thu Feb 17, 2005 10:26 am

Actually I got a Mediapartner blocked too the other day, got into my admin area... pesky little bots. Maybe I should use rat-a-way lol

Posted: Thu Feb 17, 2005 7:45 pm

Ya for a while it was everytime i logged in as admin on two sites and on mine was everyfew times.. especially if i stayed logged in for more than a few mins. Cheers! (nice site by the way, it is very familiar Wink

)

Posted: Thu Feb 17, 2005 9:13 pm

Why yes and so is yours, superficially. Smile

I like your content better tho!