Why is it that when I have a wildcard in the user agent and one of the disallows is set at /modules/ I still have bots in the forums? Aren't the forums covered by the /modules/ if so why are they still in there and more importantly how do I get them out of the forums? I would like them to be on the rest of the site but not in the forums. Is there someplace I have to activate the robots.txt file? Thanks.
Well the one I am specifically dealing with is the Googlebot. At any one time I have about 20 or so of them in my forums. I believe they maybe contributing to my max_resource problem that I have with ipowerweb. I think you remember it... (When my current contract runs out I will look at hosting with you Raven ) Anyway on the google website they say their bots honor the robots.txt file but it may take awhile... so after the standard searching I should have done first, I at least found an answer to the problem... now I will see if the do in deed honor it... Thanks again Raven....
View next topic View previous topic
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
![[Valid RSS]](http://www.ravenphpscripts.com/images/valid-rss.png "Validate my RSS feed"){kind=small}
:: 
::
) Anyway on the google website they say their bots honor the robots.txt file but it may take awhile... so after the standard searching I should have done first, I at least found an answer to the problem... now I will see if the do in deed honor it... Thanks again Raven....