Ravens PHP Scripts: Forums
 

 

View next topic
View previous topic
This forum is locked: you cannot post, reply to, or edit topics.   This topic is locked: you cannot edit posts or make replies.    Ravens PHP Scripts And Web Hosting Forum Index -> NukeSentinel™ v2.4.x
Author Message
mrix
Client


Joined: Dec 04, 2004
Posts: 757

PostPosted: Sun Sep 11, 2005 3:27 am Reply with quote

Hi all, I have been slowly updating sentinal over a period of about a year, I updated this time and was receiving emails that google bot was getting blocked so I added some google ip`s to the protected bit in sentinal. Unfortunately today I find my main keyword that has been ranked 2nd to top in google is know where to be seen anymore ouch!.
why has sentinal decided to do this now I have been ok for like over a year it also tries to block yahoo and msn. has anyone got the ip ranges for just these search engines please. Sad , also I am a it worried now about my robots.txt could someone give me an idea what I should have in my robots.txt to allow the 2 search engines in?
thanks for any help
mrix Sad


Last edited by mrix on Sun Sep 11, 2005 5:38 am; edited 1 time in total 
View user's profile Send private message Visit poster's website
hitwalker
Sells PC To Pay For Divorce


Joined:
Posts: 5661

PostPosted: Sun Sep 11, 2005 4:29 am Reply with quote

to get the ip ranges you can view this... Only registered users can see links on this board! Get registered or login!

And as for your robot.txt ,
just start your file with..
Code:


User-agent: Mediapartners-Google*
Disallow:
 
View user's profile Send private message
mrix
PostPosted: Sun Sep 11, 2005 4:34 am Reply with quote

Hi I was looking at the yahoo google and msn ip`s but see masses of them to be honest I am not sure what I need and what I dont need?
Cheers
mrix
 
hitwalker
PostPosted: Sun Sep 11, 2005 4:41 am Reply with quote

well in my case i only have google in protected range ,others are not banned....
thing is that you have to look up the ip range ...
 
mrix
PostPosted: Sun Sep 11, 2005 5:36 am Reply with quote

To be totally honest I am at a loss what you are saying here, all I want is to be indexed by most of the search engines at the very least Google MSN and Yahoo I am suprised there is not basic info anywhere to allow these engines bit not only that surely sentinal should add these as default?
could you please tell me if my robots.txt file looks ok this is the contents at the moment

User-agent: Mediapartners-Google*
Disallow:

Disallow: /admin.php
Disallow: /admin/
Disallow: /images/
Disallow: /includes/
Disallow: /themes/
Disallow: /blocks/
Disallow: /modules/
Disallow: /language/

would you say this is ok?
I do appriciate your time
thanks
mrix
 
Raven
Site Admin/Owner


Joined: Aug 27, 2002
Posts: 17077

PostPosted: Sun Sep 11, 2005 6:42 am Reply with quote

NukeSentinel does protect these by default. Something else appears to be going on because I, as well as many others, use NS and am properly indexed by the search engines.
 
View user's profile Send private message
mrix
PostPosted: Sun Sep 11, 2005 8:25 am Reply with quote

I see, all I know is that I have been ok all this time and now since the update of sentinal I have had these problems I have not added anything to my site apart from sentinal for a few months. its really dissapointing to be honest to have my main keyword kicked from google as you can imagine Sad .
Cheers
mrix
 
grantb
Regular
Regular


Joined: Feb 16, 2005
Posts: 67
Location: Canada

PostPosted: Sun Sep 11, 2005 9:57 am Reply with quote

try changing the robots txt to:

User-agent: *
Disallow: /admin.php
Disallow: /admin/
Disallow: /images/
Disallow: /includes/
Disallow: /themes/
Disallow: /blocks/
Disallow: /modules/
Disallow: /language/

Not a big diff but better syntax ps. This wont help you with Sentinel, sorry

_________________
Only registered users can see links on this board! Get registered or login! 
View user's profile Send private message
BobMarion
Former Admin in Good Standing


Joined: Oct 30, 2002
Posts: 1037
Location: RedNeck Land (known as Kentucky)

PostPosted: Sun Sep 11, 2005 11:02 pm Reply with quote

When you get an email from NukeSentinel(tm) telling you it blocked a Google search what does it have in it. We can sit here a guess all day at why it now blocks them and didn't before. My best guess is your Harvester list has something in it that is triggered by the search engine user agent string.

_________________
Bob Marion
Codito Ergo Sum
Only registered users can see links on this board! Get registered or login! 
View user's profile Send private message Send e-mail Visit poster's website
BobMarion
PostPosted: Sun Sep 11, 2005 11:06 pm Reply with quote

On the ip list ( Only registered users can see links on this board! Get registered or login! ), I ran the first ip thru an IP Whois and it has nothing at all to do with google:
Code:
inetnum:      164.69.0.0 - 164.71.255.255

netname:      FUJITSU
country:      JP
descr:        Fujitsu Ltd.
admin-c:      DY215-AP
tech-c:       DY215-AP
status:       ALLOCATED PORTABLE
mnt-by:       MAINT-JPNIC
mnt-lower:    MAINT-JPNIC
changed:      **********@apnic.net 20050408
source:       APNIC
 
mrix
PostPosted: Mon Sep 12, 2005 12:04 am Reply with quote

Well as I said a couple of days ago I was getting emails explaing google had been blocked these messages went to my junk email and now are deleted unfortunately and I havnt any since, my site under the keyword "Sea Fishing" has ranked no2 in google for 2 years and now I find my site even though it has a 5 rank is know where to be seen? it appears its only my index page thats not getting cached as most of my others are not sure if this is a clue? but it all happened a couple of days adter I installed Sentinal. but like i say I am also getting blocks from yahoo Slurp and MSN but they only seem to be my links and download pages for some reason? I have some details here where a page is getting blocked from one of my 2 sites.
Cheers
mrix


Date & Time: 2005-09-11 23:45:35 EDT GMT -0400
Blocked IP: 207.46.98.138
User ID: Anonymous (1)
Reason: Abuse-Script
--------------------
User Agent: msnbot/1.0 (+http://search.msn.com/msnbot.htm)
Query String: Only registered users can see links on this board! Get registered or login!"derby=titleA
Get String: Only registered users can see links on this board! Get registered or login!"derby=titleA
Post String: Only registered users can see links on this board! Get registered or login!
Forwarded For: none
Client IP: none
Remote Address: 207.46.98.138
Remote Port: 44585
Request Method: GET
 
BobMarion
PostPosted: Mon Sep 12, 2005 1:16 am Reply with quote

The reason is easy for me to see. It is tring to index pages with ( and ) in them. This is a leading character found in scripting attacks.

Check all your Download/Web Links titles for those two characters and replace them with something like [ and ].
 
mrix
PostPosted: Tue Sep 13, 2005 1:17 am Reply with quote

I only actually see them characters in a description of a link which I did change and still am being blocked I clicked on the link to the downloads page in the email to only see those brackets used by default in the actual downloads modules, where it ses search by (A/B) (C/D) etc ,surely this must be sentinal playing up? I am suprised no one else is getting any problems with this sentinal upgrade?
Cheers
mrix
 
mrix
PostPosted: Tue Sep 20, 2005 2:37 am Reply with quote

This is getting me down I find that I can submit a Gogle Sitmap to my subdomain website no problem at all but find I cannot submit a site map for my main site it comes up as an error??? this is stuffing my site I think in some way sentinal is blocking it as I havnt done anything to my site in months just the sentinal upgrade.
Cheers
mrix
 
storebuilder
PHP-Portal Project


Joined: Mar 09, 2004
Posts: 169
Location: Telford UK

PostPosted: Tue Sep 20, 2005 5:43 am Reply with quote

Before you do anything else.

Do a "site:www.yoursite.com" in the google search box. Are all your pages still indexed?

Do you still rank for secondary keywords?

It's quite possibly a coincidence that you lost your search engine listing and installed sentinel within the same time frame.

Do you have msanalysis installed? If you can give the site details then I can take a look for you.

Tony

_________________
Only registered users can see links on this board! Get registered or login! Only registered users can see links on this board! Get registered or login! 
View user's profile Send private message Visit poster's website MSN Messenger
mrix
PostPosted: Tue Sep 20, 2005 6:23 am Reply with quote

Hello I have had my site high ranked under the term "Sea Fishing" well second place in google for about 2 years etc. I installed well upgraded to the new sentinal and started to recieve blocking emails that I never had before from like Yahoo Slurp MSN and google. then about 3 days later I noticed a massive drop in my site hits from like 700 to under 300 unique hits a day etc so I checked my pages and found my index was know longer cached? I tried other pages but they were still cached? but after that day by day almost all my pages have been lost...ouch, I`ll install this msanalysis and get back to you .... many thanks for your time on this
Cheers
mrix
 
grantb
PostPosted: Tue Sep 20, 2005 12:35 pm Reply with quote

Taking a quick look at your source code I see you have double <he ad></he ad> tagsin the index, and in the forums section the source shows double head tags then it has the closing </b ody> and </ht ml> tags with code after, like here:
Quote:
</T R>
</T ABLE>
</B ODY>
</H TML>
<t able width="100%" cellpadding="0" cellspacing="0" border="0" align="center">
<t r valign="top">

Then further down the page another <he ad> tag:
Quote:
<t d background="themes/ClanMtS/images/leftside.gif" width="15"><img src="themes/ClanMtS/images/leftside.gif" width="15" height="4" border="0"></td>
<t d bgcolor="#242424" width="100%">
<!DO CTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<ht ml dir="ltr">
<he ad>
<me ta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<me ta http-equiv="Content-Style-Type" content="text/css">



You can almost bet the search engine is only reading the first section of the page, if at all.

Checking with the W3C validator shows an error for the forums page:
This page is not Valid (no Doctype found)!

My guess for the overall problems with search engines is not Sentinal but inccorect page layout coding. In the past maybe Google has been more tolerant of errors but it changes all the time. Check for yourself: Only registered users can see links on this board! Get registered or login!

By the way, I really like your site!
 
Display posts from previous:       
This forum is locked: you cannot post, reply to, or edit topics.   This topic is locked: you cannot edit posts or make replies.    Ravens PHP Scripts And Web Hosting Forum Index -> NukeSentinel™ v2.4.x

View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum


Powered by phpBB © 2001-2007 phpBB Group
All times are GMT - 6 Hours
 
Forums ©