Locking the site - but not to search engines...

New Member Joined: Mar 01, 2006 Posts: 7

I'm somewhat of a newbie, so I beg your patience....

Is it possible to lock the site down (e.g., everything is restricted to Registered Users), yet still allowing spidering of the site? To ask from a different perspective, is there a way of either a) registering specific bots so that they login, or b) knowing a request is coming from a spiders an treating them as an authenticated user in the code?

Sells PC To Pay For Divorce Joined: Posts: 5661

no...you cant..
its not sure how a bot crawls a mod that is only for members.
but its likely that it will have problems figuring out what you have for content.
but generaly speaking ,if you lock your site then the best thing a bot has is your index and if relevant your metatags...
but some bot reads them,and others dont.

Posted: Wed Jul 05, 2006 8:50 pm

There might be a way to do it with some careful recoding of the is_user() function, but it may not even be limited to just that function.

To be honest, not sure its worth it. If its indexed, you would be sending people to your site only to be met with "Access is restricted" messages all over the place. In my opinion, that would just tick people off to no end.... it would for me!

Think about it. When you are looking for information, you really have no idea how "good the find" is until you go look at the content. If you can't get what you want quickly, would you really go through the registration process not really knowing for sure if you are going to get a "good hit"?

Not to mention, have you given consideration to the cached content? Google and other search engines cache these pages and anyone can get to the cached content anytime they wish without even going to your site.

Just good for thought.

Posted: Thu Jul 06, 2006 3:38 am

I appreciate the reply. This is not something that I would choose to do, but I have been asked to investigate it.

Our leadership seems to think that by resticting the content they can get more people to register. My thoughts are that we're more likely to drive people away and limit search engine exposure than anything else, but I need to do the due diligence in case I have to go on a forced march.

Can you give me an idea of what constructs I'm looking for in is_user(), and an overview of how they would need to change, please?