Ravens PHP Scripts: Forums
 

 

View next topic
View previous topic
Post new topic   Reply to topic    Ravens PHP Scripts And Web Hosting Forum Index -> How To's
Author Message
blith
Client


Joined: Jul 18, 2003
Posts: 977

PostPosted: Fri Sep 19, 2003 7:47 am Reply with quote

Why is it that when I have a wildcard in the user agent and one of the disallows is set at /modules/ I still have bots in the forums? Aren't the forums covered by the /modules/ if so why are they still in there and more importantly how do I get them out of the forums? I would like them to be on the rest of the site but not in the forums. Is there someplace I have to activate the robots.txt file? Thanks.
 
View user's profile Send private message Visit poster's website
Raven
Site Admin/Owner


Joined: Aug 27, 2002
Posts: 17077

PostPosted: Fri Sep 19, 2003 7:57 am Reply with quote

bots are not required to honor any of those directives. bots either choose to or do not.
 
View user's profile Send private message
blith
PostPosted: Fri Sep 19, 2003 9:13 am Reply with quote

Well the one I am specifically dealing with is the Googlebot. At any one time I have about 20 or so of them in my forums. I believe they maybe contributing to my max_resource problem that I have with ipowerweb. I think you remember it... (When my current contract runs out I will look at hosting with you Raven Wink ) Anyway on the google website they say their bots honor the robots.txt file but it may take awhile... so after the standard searching I should have done first, I at least found an answer to the problem... now I will see if the do in deed honor it... Thanks again Raven....
 
Display posts from previous:       
Post new topic   Reply to topic    Ravens PHP Scripts And Web Hosting Forum Index -> How To's

View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum


Powered by phpBB © 2001-2007 phpBB Group
All times are GMT - 6 Hours
 
Forums ©