How many times does googlebot visit your site?

Posted: Thu Sep 16, 2004 8:10 am

I've noticed NC results are suffering from a lack of descriptions often lately too. I wonder if this is to do with efforts to crawl dynamic sites deeper?

storebuilder from a statistical stand point your view of compression may be true. But from a performance stand point every site should take advantage of compression and caching where ever possible and dynamic sites especially. On shared hosting it can make a huge increase in speed by reducing the number of database queries required to load a page. I've switched over to cached gzipped pages to anyone who isn't logged in and am caching several blocks even to users who are logged in. Even though these are just "junk sites" I really wouldn't want to go without either.

Now here is an anomaly to explain look at the first result for this search: [ Only registered users can see links on this board! Get registered or login! ]

The url is nukeresources but the description and cached link goes back to my site. The actual link goes back to nukeresources download page.

Posted: Thu Sep 16, 2004 8:14 am

Oh wait I get it now its the actual download link. So it redirects back to my site lol DOH!

registered · Posted: Thu Sep 16, 2004 8:16 am

I know it's a simple thing which people can say they've checked, but I think maybe some of you might want to make sure none of the IPs blocked by Sentinel have been googlebots I think the IPs are the 64.68.*.* range? But then again, my site hasn't been hit by a googlebot in 24 hours and usually they hit several times daily......or.....maybe it was just me overlooking my sentinel hits from google bots crawling cached pages and getting banned and me not realizing until days later (not good)...... where's the "loser" smiley with the L on the forehead?!?! *grin*

Posted: Thu Sep 16, 2004 9:36 am

At this point can we objectively say that the following is true:

The fiapple theme has the correct stylesheet info giving "storytitle" at the start of a story and is the default theme which a search engine would see.

If your descriptions in google carry the words

Quote:

Donate o Meter. Help Support RavenPhpScripts and NukeSentinel™. Make donations with
PayPal! Donat-o-Meter Stats. September´s Goal: $300.00. Due Date: Sep 30. ...

Which they do - then google can read the page and we can rule out GZip. Because it is reading the first block on the page. If it couldn't see through gzip then it wouldn't see this text?

Do we agree that it can't be the stylesheet and can't be Gzip?

Site Admin/Owner Joined: Aug 27, 2002 Posts: 17088

IMO, yes.

Posted: Thu Sep 16, 2004 12:34 pm

In which case can I suggest you make a simple HTML page with a page title called "googlebot woes give me the blues" or a page title of your choice - give it something obscure so that it will easily be a number one page.

Match it with a header

Oh boll£$%^ it's quicker to make it Wink

Quote:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<title>googlebot woes give me the blues</title>
<meta name="description" content="googlebot woes give me the blues">
<meta name="keywords" content="googlebot blues, freshbot">
</head>

<body>
<h1>Googlebot woes give me the blues
</h1>
 
They sure do those googlebot woes,
got me woeing into my beer,
woe, woe, is those googlebot blues,
gotta bring them right back, my dear. 
Now I know you are from kentucky right? But I mixed it up with Kansas for this one Wink

. Just pluck E, D, F. Wink

. 
 
<a href="http://www.ravenphpscripts.com">Back to index page
</a>
 
  
</body>
</html>

Now paste this into the root directory of your site and link to it from your home page.

Then we'll see if it gets indexed properly and start apportioning blame from there Wink

Posted: Thu Sep 16, 2004 1:12 pm

I see what you mean sixone. But it doesn't explain why the "descriptions" should deteriorate does it?

Deep crawling is just an expression for indexing the whole site. Wheras the freshbot indexes new content daily. So I don't see the correlation.

Let's see what Raven get's by putting up the HTML page and then we can all chew on the fat. Does that seem ok?

Posted: Thu Sep 16, 2004 1:18 pm

I placed the link in the Partners block Wink

Posted: Thu Sep 16, 2004 1:31 pm

Yep - you have put me up on the cross fairly and squarely. Laughing

Where is Chatserv? He normally comes and says "oh, you do this...." Very Happy

Posted: Thu Sep 16, 2004 3:50 pm

I think we're all concerned with the forums getting crawled since its the most active area of so many sites. Glad we have the 2 of ya on the job here! Wink

Posted: Mon Sep 20, 2004 6:00 am

I was just doing a search for a "center block links" when I noticed that the top result was ravenphpscripts and the title and description was displayed properly.

I did some more searching and started seeing your pages appearing in the top results all over!!!

Whatever it is that you have removed has done the trick.

Posted: Mon Sep 20, 2004 6:05 am

module copyrights not working

this search shows that your forum posts are now being indexed - looks like some major traffic is going to be headed your way!

Posted: Mon Sep 20, 2004 6:10 am

In that one, yes. But in the Old Articles block, when I searched for 'Chatserv has achieved fame in print', I'm not seeing that, exactly. Here's what came back

Code:

General purpose self-explanatory file with news headlines % ...... gif %% ChatterBox 2004 /modules.php?name=News&file=article&sid=515 2004-09-13 05:38:25 Raven Add-Ons 0 addon.gif %% Chatserv has achieved fame in print! ... 


ravenphpscripts.com/ultramode.txt - 2k - Cached - Similar pages

Is that what you would expect? And I still don't see the forums getting indexed or am I missing it?

Posted: Mon Sep 20, 2004 6:14 am

storebuilder wrote:

module copyrights not working

this search shows that your forum posts are now being indexed - looks like some major traffic is going to be headed your way!

Our posts must have crossed Smile

I'll keep looking.

Posted: Mon Sep 20, 2004 6:24 am

Right - but you just need to have a bit of patience. The pages that you get crawled today will appear in the index in a couple of days time, but effectively your whole site needs to be reindexed so it's going to happen over a month or so.

Posted: Mon Sep 20, 2004 6:25 am

also - the example that I quoted above is a forum post - and has only been indexed since you changed to dynamic titles.

Posted: Mon Sep 20, 2004 6:27 am

And a search for this [ Only registered users can see links on this board! Get registered or login! ]

Shows you up in spot 9. That shows you are being indexed correctly.

Posted: Mon Sep 20, 2004 6:30 am

Posted: Mon Sep 20, 2004 6:39 am

Posted: Tue Sep 21, 2004 2:50 pm

Almost 800 times today Smile

) And, a search for 'Googlebot woes give me the blues' comes up #1. I'm happy.

Edit: 865 times so far Wink

Client Joined: Apr 10, 2004 Posts: 649 Location: UK

After reading thru this thread with great interest, would it be possible for Storebuilder to put in one post what the rest of us have to do to get so many hits from the google bots, cos as the threads progressed it's got a bit confusing.

Not quite sure what dynamic titles are for a start or where to change them lol Sorry I'm only learning.

Thanks.

Posted: Fri Sep 24, 2004 11:33 am

Raven, did you decide what it was that was doing the damage?

Posted: Fri Sep 24, 2004 12:24 pm

No. I made so many changes. But the final change I made was to remove the google ads. However, before that I had added title tags to almost everything. So, somewhere amongst dynamic titles, theme changes, removing google ads, adding title tags, and offerring a blood sacrifice during the last full moon, it got fixed ROTFL

Posted: Fri Sep 24, 2004 12:35 pm

Yep - that's just what I would expect from a thoughtful, logical, programmer thinker. Laughing

Have you still got GZip turned off? Cause that's where my money goes....

Posted: Fri Sep 24, 2004 12:39 pm

Nope, it's on afaik.