Cant get Google to crawl Forums & tuning GT

Involved Joined: Oct 09, 2004 Posts: 293

Gt on 7.5.
Crawling Problem:
Only home appears in the gt block
Almost Everything appears to work and have html tags
Looksmart will crawl but others dont.
I have sitemap done [ Only registered users can see links on this board! Get registered or login! ]
I have gathered that it maybe something to do with

Quote:

The sessions ids are used in the forums and this is a problem for google but of course also for other crawlers they don´t like session ids.

see [ Only registered users can see links on this board! Get registered or login! ]
I dont know about or how to change session ids.
name=Forums&file=viewtopic&p=1784#1784
Essentually all I need is engines to crawl web links and the sticky posts in the forums.

When I have the mouse over the forums link on the front page (top toolbar)I get [ Only registered users can see links on this board! Get registered or login! ]
Once in the forums all the links appear as [ Only registered users can see links on this board! Get registered or login! ]
I sort of get the feeling that it may have something to do with the forms link on the front page not being html and search engines not able to follow that.???
If so how and where do I change the forum tool bar link?
I added a ' posts since last visited" link when loggen in in my user info...that has a html link..I have no idea how or why that is like that..thu I think thats good?

Tuning GT:
I see in the .htacess rewrite things for parts of the site

Code:

#Submit_News


RewriteRule ^submit.html submit.html [L]

Then corrosponding files in Root/Googletap/GT-Submit_News.php
As I only need googe to search some parts, Forums, web links, couple web link blocks... surveys, top ten etc are not used and/or dont need to be crawled, do I need those files/entries? can I delete them? is there anything else I need to delete or 'clean up'?
Adding the code for the web links I think I can work out once I know how to take the stuff that doesnt need to be crawled ot.
This is all the stuff that doesnt need to be crawled or not used
AvantGo, Encyclopedia, Feedback, FAQ, Journal, Members List, Private Messages, Recommend Us, Search, Sections, Statistics, Surveys, Submit_News, Top10, Webmail, Your Account

I have done the site map thing... [ Only registered users can see links on this board! Get registered or login! ] with just forums and think?? it would be good to have the site map, GT, robots.txt meta tags all doing the same thing. Like designing/ tuning up a Hot Rod so all the compents are 'matched' and doing the same thing.

Posted: Mon Aug 08, 2005 9:24 am

First of all. I don´t have the code for your sessions.php to remove the sids for Google &Co.
Your forum is full with session ids.If you try spiders eye, you should see all the sessions ids.If you check your logfiles you`ll probably find such requests with long sids: The sids aren´t search engine friendly.
65.54.188.138 - - [06/Aug/2005:00:00:50 +0200] "GET /ftopict-379.html?sid=515d8b646c01920be7296e8fedh527594 HTTP/1.0" 200 34416 "-" "msnbot/1.0

2004 was a discussion at phpBB.de about short urls also about session ids.This was the first time I was confronted with this problem. With a stand alone phpBB forum they all was able to remove the sids for spider.So if you install a new phpbb forum take a look at this great discussion. In the meantime there are about 105 pages or more about this topic.
But it´doesn´t work with the nuke forum because the sessions.php is different. I tried it and had no luck.phpBB had also problems before the used this function.e.g. you have 100.000 posts but with session ids are only 30 - 45 % in the search engine index.
But your problem is not only the forums session ids.Your link popularity isn´t high enough and the meta tags are not complete. You can use deeplinking to your forum,this should work.
And of course there are several ways to optimize a forum. I´m using different keywords and descr. and a lot of other things. I´m not talking about this, the most I´ve forgotten.
You can change your forums overall header and you can also try dynamic titles for better results.

Check your meta tags again. I found this:

Found no robots meta tag.
This tag is not really required but helps spiders to index the web page.

The web page contain adult words and poisons keywords .

Do you have also the language defined in your meta-tags ? You should have this too.
<META HTTP-EQUIV="Content-Language" CONTENT="">

About poison keywords we had a discussion before.The word "forum" is only one of this words. But doesn´t matter. I renamed only some words.
See also:
[ Only registered users can see links on this board! Get registered or login! ]

If you have webmail modules installed, delete it immediatly because its a high security risk !

Something to read and you can try the sessions.php from yoinda.When I tried this in 2004 I got too many 404 errors .It wasn´t my solution. But it´s working and this code helped many peoples.
[ Only registered users can see links on this board! Get registered or login! ]

So we are looking for this short piece of code: if you are a spider......

Posted: Mon Aug 08, 2005 6:27 pm

The 1st 4 paragraghs I have no idea what u are talking about...my ignorance lol

Quote:

Your link popularity isn´t high enough and the meta tags are not complete

For the subject we are rated on the front page....yes thanks, I was messing around the other day and // out a line I shouldnt have...the robots line.

Our meta tag

Code:

echo "<META HTTP-EQUIV=\"Content-Type\" CONTENT=\"text/html; charset="._CHARSET."\">\n";


echo "<META HTTP-EQUIV=\"EXPIRES\" CONTENT=\"0\">\n";


echo "<META NAME=\"RESOURCE-TYPE\" CONTENT=\"DOCUMENT\">\n";


echo "<META NAME=\"DISTRIBUTION\" CONTENT=\"GLOBAL\">\n";


echo "<META NAME=\"Title\" CONTENT=\"Kakariki, Care, Breeding, Ecology, and Conservation\">\n";


echo "<META NAME=\"AUTHOR\" CONTENT=\"Steptoe\">\n";


echo "<META NAME=\"COPYRIGHT\" CONTENT=\"Copyright (c) 2004 2005 by Kakariki.net\">\n";


echo "<META NAME=\"KEYWORDS\" CONTENT=\"kakariki, kakarikis, breeding, mutation, feeding, diet, food, care, aviary, aviaries, cage, breed, mutations, native_parrot , red, yellow, orange, blue, red_crown, red_fronted, yellow_crown, yellow_fronted, Orange_Fronted, orange_crown, New_Zealand, NZ, Antipodes, Enderby, Bollen, Tiritiri, Department_of_Conservation, DoC, flight, nest, parrot, cyanoramphus, auriceps, novaezelandiae, malherbi, unicolor\">\n";


echo "<META NAME=\"DESCRIPTION\" CONTENT=\"Experienced Members Have Created a Resource of Information on Kakariki Care, Aviaries, Breeding, Mutations, Diet, Health, Behaviour, Conservation and Related Subjects.\">\n"; 


echo "<META NAME=\"DESIGNER\" CONTENT=\"Steptoe\">\n";


echo "<META NAME=\"PUBLISHER\" CONTENT=\"Steptoe\">\n";


echo "<META NAME=\"ROBOTS\" CONTENT=\"ALL, INDEX, FOLLOW, \">\n";


///echo "<META NAME=\"ROBOTS\" CONTENT=\"all,\">\n";


echo "<META NAME=\"REVISIT-AFTER\" CONTENT=\"1 DAYS\">\n";


echo "<META NAME=\"RATING\" CONTENT=\"GENERAL\">\n";

Quote:

Do you have also the language defined in your meta-tags ? You should have this too.
<META HTTP-EQUIV="Content-Language" CONTENT="">

Althu our site is english, many of the members are European/ Asian etc should I still add it?

Quote:

The web page contain adult words and poisons keywords

No we dont have those sorts of words??

Quote:

[ Only registered users can see links on this board! Get registered or login! ]

Yes went there and refferd back here.

Quote:

If you have webmail modules installed, delete it immediatly because its a high security risk !

it is installed and set active, thu only visible to Admin, Im using postcast smtp server, do I still need web mail activated to have the email to work...It was my understanding web mail was 'sort of' not active in version 7.5 due to security issues along with couple other modules??
From my orginal post...

Quote:

When I have the mouse over the forums link on the front page (top toolbar)I get [ Only registered users can see links on this board! Get registered or login! ]
Once in the forums all the links (including the top task bar forum link changes) appear as html [ Only registered users can see links on this board! Get registered or login! ]
I sort of get the feeling that it may have something to do with the forms link on the front page not being html and search engines not able to follow that.???

I found this in my /deepblue/theme.php on the lines that make the buttons to forums etc in the task bar below the banner, front page

Code:

."<a href=\"index.php\"><img border=\"0\" src=\"themes/DeepBlue/images/home.gif\" width=\"140\" height=\"18\"></a>"


///   ."<a href=\"forums.html\"><img border=\"0\" src=\"themes/DeepBlue/images/Forum.gif\" width=\"140\" height=\"18\" alt=\"Forums\"></a>"  


////orginal line above


   ."<a href=\"forums.html\"><img border=\"0\" src=\"themes/DeepBlue/images/Forum.gif\" width=\"140\" height=\"18\" alt=\"Forums\"></a>"  


   ."<a href=\"modules.php?name=gallery\"><img border=\"0\" src=\"themes/DeepBlue/images/Gallery.gif\" width=\"140\" height=\"18\" alt=\"Gallery\"></a>"

Changing

Code:

a href=\"forums.html\

To

Code:

a href=\"forums.html\

Now makes the forum link on the front page appear as [ Only registered users can see links on this board! Get registered or login! ] and im not getting or can find any errors happening...could this be a soln to getting google to crawl forums from that link? could this be an oversigt in GT...or maybe mine lol.

I really appreciate your time
Steps

Posted: Mon Aug 08, 2005 7:02 pm

A side note..my boo boo in the meta tag mentioned above has been rather dramatic...within a couple days of making it we have dropped from the top 50 pages on google for every seach parameter except a cuople, and they (except seach "kakarki" ) have droped way down.
Previous to my boo boo on the 6th Aug theres where our ratings for search parameters [ Only registered users can see links on this board! Get registered or login! ]

The general comments around the place is " meta tags are now longer used or are not high on the list by many search engines for site rating"

I would conclude from this meta tags are more important than than is generally accepted.

Posted: Tue Aug 09, 2005 12:11 pm

Code:

meta tags are now longer used or are not high on the list by many search engines for site rating

Time changes.
There are a lot of search engines not only "G",Yahoo and MSN and of course the algorithm of the search engines isn `t always the same. But you don´t have to understand this.

Just use your meta tags and try to get more link pop !
For deeplinks you need links to your [ Only registered users can see links on this board! Get registered or login! ] instead of the main site. Have you ever checked Ravens linkpopularity ? It´s more than great its fantastic. Laughing

You can use this tool they have the poison keywords warning.
[ Only registered users can see links on this board! Get registered or login! ]

But its often better to use several tools:

"G" can also follow such links:

[ Only registered users can see links on this board! Get registered or login! ]

You ´ll find also such links in the Google index.
[ Only registered users can see links on this board! Get registered or login! ]
Without describtion.
I quess its normal, but Im not the Next Gen Google Tap expert, so if you need further information about this ask again in the forum.
Of course links without ? and session ids are better. For the 1. we are using GT Next Gen and the 2. we are looking for this piece of code if you are a spider........

Navigation:Of course you can change your links in the theme.php.

E.g
."<a href=\"downloads.html\"><img border=\"0\" src=\"themes/DeepBlue/images/kakariki.gif\"alt=\"Downloads\" width=\"140\" height=\"18\"></a>"
Metatags:
Language:Add your languages into the metatags.But of course if you don`t speak chinese you should`t add this language. This tag is interesting for multilingual websites.

See also:
[ Only registered users can see links on this board! Get registered or login! ]

<META HTTP-EQUIV="Content-Language" CONTENT="">

You can also use the page topic metatag.

<META NAME="page-topic" CONTENT="e.g. Internet">

See also the warning about Webmail Attack:
[ Only registered users can see links on this board! Get registered or login! ]

Posted: Tue Aug 09, 2005 4:09 pm

Great stuff
so u dont actually need web mail for replies to posts /pms /registarion etc....its gone lol
The web page contain adult words. xxx this is in the ip on user info block
Changed that to a 'y'

The web page contain poison keywords. 'forum, search engine'
"Search engine" is the name of a forum thread, changed that.
"forum" will have to change words in news items... and part of the page...will work on rewording that...thats a difficult one in some cases

Quote:

Have you ever checked Ravens linkpopularity

yes you hit the nail on the head here...when using google, seaching for and answer to a issue I may have, This and many other sites come up with the actual forum post....our site just comes up with the front page...forum posts is what I would like to achieve on our site, in particular the sticky threads.

The report from the sites u mention above report "This page contains too many URLs." can links like submit news, reconmend us, your a/c, lost pw, regist, Stats etc be 'blocked'?
I have tryed diff stuff in the robots.txt but doesnt seem to work.
Also see P.S. below.

Quote:

and try to get more link pop !
For deeplinks you need links to your [ Only registered users can see links on this board! Get registered or login! ] instead of the main site

I see there are sort of 'commercial' link exchange sites one can use...some how these dont 'feel right' to use. We have been link exchanging with related sites, the draw back to many of these is they have poor meta tags, or none at all. I didnt think to submitt the forums page...in retropect would have been far more logical.
I try to get 2 links per week..a time thing lol

P.S just checked the site logs, Google hit us last night.... 2262 hits in the forums (there is about 200 threads and 1000 posts)
Many are like /modules.php?name=Forums&file=posting&mode=reply&t=246
I assume these are follwing the links off the reply/quote buttons ???
Can google be helped out a bit by bocking these in the robots.txt, edit/delete parts of GT, etc
As far as editing GT Im having trouble finding info, and what is there is not clear.
EG in the /Google tap/ folder there are files like GT-Statistics.php can these just be deleted, along with the corrosponding enties in /.htaccess like

Code:

RewriteRule ^index.html index.php


#Statistics


RewriteRule ^stats-([0-9]*)-([0-9]*)-([0-9]*).html modules.php?name=Statistics&op=DailyStats&year=$1&month=$2&date=$3 [L]


RewriteRule ^stats-([0-9]*)-([0-9]*).html modules.php?name=Statistics&op=MonthlyStats&year=$1&month=$2 [L]


RewriteRule ^stats-([0-9]*).html modules.php?name=Statistics&op=YearlyStats&year=$1 [L]


RewriteRule ^advstats.html modules.php?name=Statistics&op=Stats [L]


RewriteRule ^stats.html stats.html [L]

Sorry for all the questions....thu I do have to blame you for that, your answers are in a lay forumat and the 1st time I can get my head around things. lol
Im sure this thread would be of help to many others in the future.

Cheers Steps

Posted: Wed Aug 10, 2005 2:03 pm

Quote:

Some Search Engines have problems with more than 100 urls on a page.

To reduce the links is only a proposal but of course you can remove some links from your start page e.g. stats.html.
About the robots.txt You ´ll also find at Google several information for webmaster.
There is nothing to delete in the Google Tap folder or in your htaccess.About the real logic behind links from and to a website and how it works, I feel we are in the wrong forum.
We can discuss this at [ Only registered users can see links on this board! Get registered or login! ]

Deeplinks are sometimes very helpful, believe me. If you need more information how it works send me a PM.
Of course sitemaps are also a useful way.

The important question how to remove the forum sids has been not answered yet. Where are the php experts ?

Btw:(Did´t you know that your block background is black and the font color is also black. But doesn´t matter black is beautiful and "G" can follow such links Rolling Eyes

)

Posted: Wed Aug 10, 2005 3:32 pm

You8r answer just came after just as I finished going over everything...did find a code error ..mine in GT doing the GT 4a update from some months ago.
I didnt have the 'edit box in the Amin/cp/modules...I do now but when I edit to say
forums.html the go back to check the entry is not there??
Yes I do have the url sql entry in the db.
Also the GT modules block (mentioned in a post above) is now working thu it is visible to Admin only

Quote:

To reduce the links is only a proposal but of course you can remove some links from your start page e.g. stats.html

Stats link is only in the custom user info block (needs finishing [ Only registered users can see links on this board! Get registered or login! ] and cant figure out how to do last bit.. in...) althu stats turned visible to regist only still shows from user info
The links that can be removed are all to user proflies in
last 10 post block (10)
news items posted by xxxx (10)
in user info new members and members logged on (3 + 1 or 2 for members)
Log out/my account on the top task bar and user info (2)
Plus these links
Links in ...these are visible when logged in.
weather channel link in weather blocks (2)
links in quick search block.. (5)
members sites block (11)
conservation sites block (10)
Links in news items body (6)
Total approx (62) links

I have know idea how to do these.

Sitemaps...yep done that the other day [ Only registered users can see links on this board! Get registered or login! ]

Quote:

Did´t you know that your block background is black and the font color is also black.

yep. its a customised deep blue...the black loads then the .gifs load over them...I did that custonising sortly after 1st publishing the site, this is my 1st and only attempt at a web site...I have never actually opened Front page or ever coded anything before except the odd simple batch file...everything is cut paste and try. That and a couple other things, basically cosmetic, I will go back later and figure out...Im sure it makes the site load a lot slower than it should??

Quote:

About the real logic behind links from and to a website and how it works, I feel we are in the wrong forum.
We can discuss this at [ Only registered users can see links on this board! Get registered or login! ]

you mean the

Quote:

Can google be helped out a bit by bocking these in the robots.txt, edit/delete parts of GT, etc ................

Posted: Wed Aug 10, 2005 5:53 pm

I meant the best way to link to other sites.e.g. from A to B from C to A.

And also helpful is the Google information for webmaster.

[ Only registered users can see links on this board! Get registered or login! ]

See also my PM.

Posted: Mon Aug 15, 2005 7:14 pm

update...
I was using a little free prog from [ Only registered users can see links on this board! Get registered or login! ] to check ratings BUT if one uses it too often then Google considers that they are getting spammed from that IP...Sooo, all my ratings google ratings for search parameters went low, too low to find, except for kakariki.
Now after a week most parameters have suddenly appeared...sigh of realf lol
Google has done a big several 1000 page craw of the site a while back, but not since...and when runing a google kakariki search then hit 'similar pages' nothing lol
Yet MNS lycos, etc have 100s of pages with links to forums etc.
Have got a few links to us with [ Only registered users can see links on this board! Get registered or login! ] (sry another plug) with other related subject sites...yet google doesnt go there.
As far as other seach engines google rating content compared to other seach engines is bad...extreme example want to design and build a kike aviary....all the other engines rate well , the 1000 or so in front of us have no info lol
Maybe google isnt coded for kakariki lol

Posted: Tue Aug 16, 2005 2:29 pm

Hi Steptoe,

a week isn´t so long time.Maybe Google was lazy or busy with our site. Very Happy

You should really change the links in the navigation to weblinks.html, downloads.html, And of course your site isn´t WC3 valid Html Neutral

The session ids are also no pleasure for googlebots. [ Only registered users can see links on this board! Get registered or login! ]
You can also place a link to your forums.html into your text and there between the first 200 words.
And if you don´t need all those funktions search or the translations in the navigation remove this. I´m sure the sitemap is a good thing. Be careful with the word kike is this also new zealand accent ? (kike aviary) !

Posted: Tue Aug 16, 2005 7:11 pm

Quote:

Be careful with the word kike is this also new zealand accent ? (kike aviary) !

The generally accepted nic has been kak...but as pionted out some time by a member, kak to many European/Sth African members looses something in transalation...so I used the Kiwi (NZer) term kike..I checked kike out as a bad word for serach engines and came back ok. Kike is now becoming a generally accepted nic for a kakariki. lol
I did have in the user info visitor ips with the last numbers hidden with xxx that got changed to zzz
Downloads..not worried about those...members only
most of the web links are on the members sites and conservation sites blocks, google and other engines hit those np

Edit: from the links u provided analyis said to many links on the page, so eliminated the links to author delails of news items..dont need those
Also have a bit of code from Chris at karakas to take out similar links in the user info so only visble when logged in ...dont need those either and going to look at taking similar links in forums...thread starter and last poster.