PHP Web Host - Quality Web Hosting For All PHP Applications Sign up for PayPal and start accepting credit card payments instantly
  Login or Register
 • Home • Downloads • Your Account • Forums • 

View next topic
View previous topic


Google
 
Web RavenPHPScripts (This Site)
Post new topic   Reply to topic
Author Message
blith
Client


Joined: Jul 18, 2003
Posts: 977

PostPosted: Fri Sep 19, 2003 7:47 am Reply with quote Back to top

Why is it that when I have a wildcard in the user agent and one of the disallows is set at /modules/ I still have bots in the forums? Aren't the forums covered by the /modules/ if so why are they still in there and more importantly how do I get them out of the forums? I would like them to be on the rest of the site but not in the forums. Is there someplace I have to activate the robots.txt file? Thanks.
View user's profile Send private message Visit poster's website
Raven
Site Admin/Owner


Joined: Aug 27, 2002
Posts: 16987
Location: Kansas

PostPosted: Fri Sep 19, 2003 7:57 am Reply with quote Back to top

bots are not required to honor any of those directives. bots either choose to or do not.
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger
blith
Client


Joined: Jul 18, 2003
Posts: 977

PostPosted: Fri Sep 19, 2003 9:13 am Reply with quote Back to top

Well the one I am specifically dealing with is the Googlebot. At any one time I have about 20 or so of them in my forums. I believe they maybe contributing to my max_resource problem that I have with ipowerweb. I think you remember it... (When my current contract runs out I will look at hosting with you Raven Wink ) Anyway on the google website they say their bots honor the robots.txt file but it may take awhile... so after the standard searching I should have done first, I at least found an answer to the problem... now I will see if the do in deed honor it... Thanks again Raven....
View user's profile Send private message Visit poster's website
Display posts from previous:       
Post new topic   Reply to topic

View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Forums ©
 

All logos and trademarks in this site are property of their respective owner.
The comments are property of their posters, all the rest © 2002-2011 by Raven

You can syndicate our news using the file xml

CSE HTML Validator Helped Clean up This Page! [Valid RSS] valid RSS 2.0 Valid robots.txt Stop Spam Harvesters, Join Project Honey Pot

Website engines core code is © copyright by PHP-Nuke but has been heavily patched and modified by myself and others.
PHP-Nuke is a free software released under the GNU/GPL.


:: fisubice phpbb2 style by Daz :: PHP-Nuke theme by www.nukemods.com ::
:: fisubice Theme Modified by the RavenNuke™ Team ::

:: W3C CSS Compliance Validation :: W3C HTML 4.01 Transitional Compliance Validation ::

zerosum