Yesterday on my website, I had 6 vistors, all with the same ip, expect the last number was different, sam spade trace route said they where all webcrawler/google bots, how can I prevent google bots from coming on my site? or should I not worry about it?, my downloads are set to registerd users only.
Joined: Aug 27, 2002 Posts: 16987 Location: Kansas
Posted:
Tue Sep 23, 2003 2:17 pm
robots.txt, for the bots that honor it, is the way to 'guide' the bots around your site. You could, of course, use .htaccess to blick the IP's, but it's a personal decision I suppose. I opt to try to guide them. Just my opinion.
Not sure I understand what you ment, do I really need the robots.txt? can I delete it from my root? or do i have to added something to it, to keep those bots off my site? like add a ban module or something?
View next topic View previous topic
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
![[Valid RSS]](http://www.ravenphpscripts.com/images/valid-rss.png "Validate my RSS feed"){kind=small}
:: 
::
