Ravens PHP Scripts: Forums
 

 

View next topic
View previous topic
Post new topic   Reply to topic    Ravens PHP Scripts And Web Hosting Forum Index -> General/Other Stuff
Author Message
magnum
Client


Joined: Jun 23, 2006
Posts: 83

PostPosted: Sun Apr 01, 2007 1:30 pm Reply with quote

Im just wondering how much bandwidth your sites uses a month i have 3 gigs and now , every month hitting its limit so i got 3 more gigs total 6 gigs now and its still going to hit the limit in bandwidth so im wondering how much gigs do your pages use a month am i to low at 6 gigs? whats a normal use a month.

_________________
Nukes real friend is a big cup of Java with a valium stirred in. Only registered users can see links on this board! Get registered or login! 
View user's profile Send private message
jakec
Site Admin


Joined: Feb 06, 2006
Posts: 3048
Location: United Kingdom

PostPosted: Sun Apr 01, 2007 2:21 pm Reply with quote

Wow, that sounds like a lot. Can you give us some stats? Number of users etc.
 
View user's profile Send private message
magnum
PostPosted: Sun Apr 01, 2007 3:59 pm Reply with quote

i have about 20 users that are regular every day logons let me get the stats and ill post it up
 
magnum
PostPosted: Sun Apr 01, 2007 4:01 pm Reply with quote

does this help?

Monthly Statistics for March 2007

Total Hits 792362
Total Files 472342
Total Pages 5834
Total Visits 3269
Total KBytes 4237645

Total Unique Sites 2193
Total Unique URLs 1633
Total Unique Referrers 104
Total Unique User Agents 225

. Avg Max

Hits per Hour 1065 9959
Hits per Day 25560 38995
Files per Day 15236 21318
Pages per Day 188 311
Visits per Day 105 132
KBytes per Day 136698 214621

Hits by Response Code

Undefined response code 1
Code 200 - OK 472342
Code 206 - Partial Content 1904
Code 301 - Moved Permanently 2
Code 302 - Found 27433
Code 304 - Not Modified 227312
Code 403 - Forbidden 9111
Code 404 - Not Found 54247
Code 405 - Method Not Allowed 2
Code 406 - Not Acceptable
 
Gremmie
Former Moderator in Good Standing


Joined: Apr 06, 2006
Posts: 2415
Location: Iowa, USA

PostPosted: Sun Apr 01, 2007 5:51 pm Reply with quote

I usually do about 13 Gigs a month but have gotten as high as 20 before.

_________________
Only registered users can see links on this board! Get registered or login! - An Event Calendar for PHP-Nuke
Only registered users can see links on this board! Get registered or login! - A Google Maps Nuke Module 
View user's profile Send private message
djmaze
Subject Matter Expert


Joined: May 15, 2004
Posts: 719
Location: http://tinyurl.com/5z8dmv

PostPosted: Sun Apr 01, 2007 6:09 pm Reply with quote

magnum wrote:
Code 206 - Partial Content 1904
Code 301 - Moved Permanently 2
Code 302 - Found 27433
Code 404 - Not Found 54247


Those are strange since phpnuke never sends those.

Code 304 could be your theme images and css files though and that is a good thing.

_________________
$ mount /dev/spoon /eat/fun auto,overclock 0 1
ERROR: there is no spoon Only registered users can see links on this board! Get registered or login! 
View user's profile Send private message Visit poster's website
Raven
Site Admin/Owner


Joined: Aug 27, 2002
Posts: 17086

PostPosted: Sun Apr 01, 2007 10:52 pm Reply with quote

magnum,

Check your AWSTATS and see what referrer is chewing up your bandwidth. It may be a bad bot or something like that.
 
View user's profile Send private message
Susann
Moderator


Joined: Dec 19, 2004
Posts: 3191
Location: Germany:Moderator German NukeSentinel Support

PostPosted: Mon Apr 02, 2007 5:18 am Reply with quote

Check for bad bots, unknown bots and when you find bots from China,Korea or Japan and you feel you don´t need results in asian searchengines ban them.
I`ve got a lot of spam from asian server in the past. So this can also reduce the spam flood.
Only registered users can see links on this board! Get registered or login!
 
View user's profile Send private message
magnum
PostPosted: Mon Apr 02, 2007 10:10 am Reply with quote

Last Update: 02 Apr 2007 - 02:43

Reported period: Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec - Year - 2006 2007


Back to main page

Links from an Internet Search Engine
7 different refering search engines Pages Percent Hits Percent
Google 176 59.8 % 176 59.8 %
Yahoo 72 24.4 % 72 24.4 %
MSN 26 8.8 % 26 8.8 %
Unknown search engines 10 3.4 % 10 3.4 %
AOL 4 1.3 % 4 1.3 %
AltaVista 3 1 % 3 1 %
Netscape 3 1 % 3 1 %

--------------------------------------------------------------------------------------

Robots/Spiders visitors (Top 25) - Full list - Last visit
8 different robots* Hits Bandwidth Last visit
Googlebot 139719+60 1.46 GB 31 Mar 2007 - 23:16
Inktomi Slurp 2911+1471 28.93 MB 31 Mar 2007 - 23:58
Unknown robot (identified by 'crawl') 517+27 5.19 MB 31 Mar 2007 - 20:51
Unknown robot (identified by hit on 'robots.txt') 0+152 84.50 KB 31 Mar 2007 - 23:14
Unknown robot (identified by 'spider') 6+66 294.90 KB 31 Mar 2007 - 21:37
Unknown robot (identified by 'robot') 32+16 772.19 KB 30 Mar 2007 - 23:44
MSNBot 0+26 14.55 KB 31 Mar 2007 - 23:07
AskJeeves 9+9 311.55 KB 28 Mar 2007 - 09:54

* Robots shown here gave hits or traffic "not viewed" by visitors, so they are not included in other charts. Numbers after + are successful hits on "robots.txt" files

how do i stop googlebot or do i want too?


Last edited by magnum on Mon Apr 02, 2007 10:37 am; edited 1 time in total 
Susann
PostPosted: Mon Apr 02, 2007 10:36 am Reply with quote

Use site:islandtitanz.org and you ll find all your indexed sites at google.
I have the same results for Slurp montly thats normal but 1.46 GB for Googlebot ?

Check this:http://www.google.com/support/webmasters/

but I would not disallow Googlebot general, because its an important search engine.
 
magnum
PostPosted: Mon Apr 02, 2007 11:01 am Reply with quote

would this slow googlebot down in my robots.txt or is something else better?

The following block of code DISALLOWS the crawling of all files and directories to the following spiders/robots: Slurp msnbot
Line 1 User-agent: Slurp
Line 2 Crawl-delay: 20
Line 3
Line 4
Line 5 User-agent: msnbot
Line 6 Disallow: /
Line 7


The following block of code DISALLOWS the crawling of all files and directories to the following spiders/robots: Inktomi Slurp
Line 8 User-agent: Inktomi Slurp
Line 9 Disallow: /
Line 10
Line 11
Line 12


The following block of code DISALLOWS the crawling of the following files and directories: /modules.php?name=Top&querylang=union%20select%200,pwd?id=honeytrap /abuse/ /admin/ /blocks/ /cgi-bin/ /db/ /images/ /includes/ /language/ /modules/ /public/ /themes/ /tmp/ /admin.php /config.php /demohack.php /cplogin.php /hackattempt.php /login.php to the following spiders/robots: googlebot
Line 13 User-agent: googlebot
Line 14 Disallow: /modules.php?name=Top&querylang=union%20select%200,pwd?id=honeytrap
Line 15 Disallow: /abuse/
Line 16 Disallow: /admin/
Line 17 Disallow: /blocks/
Line 18 Disallow: /cgi-bin/
Line 19 Disallow: /db/
Line 20 Disallow: /images/
Line 21 Disallow: /includes/
Line 22 Disallow: /language/
Line 23 Disallow: /modules/
Line 24 Disallow: /public/
Line 25 Disallow: /themes/
Line 26 Disallow: /tmp/
Line 27 Disallow: /admin.php
Line 28 Disallow: /config.php
Line 29 Disallow: /demohack.php
Line 30 Disallow: /cplogin.php
Line 31 Disallow: /hackattempt.php
Line 32 Disallow: /login.php
Line 33
 
jakec
PostPosted: Mon Apr 02, 2007 11:08 am Reply with quote

If you have an account setup with Google, you can set a crawl delay in there.
 
Susann
PostPosted: Mon Apr 02, 2007 6:31 pm Reply with quote

Inktomi Slurp is from Yahoo. So your robots.txt isn´t correctly.
Check this link and the redirect:


If you use crawl-delay for Slurp you can try it also with 30 or 60 and afterthat check your logfiles and make sure that your robots.txt is valid.
Don´t know if that works for top.html?.
But like Jakec said you can use crawl-delay and also check different things with an account.


Last edited by Susann on Wed Aug 29, 2007 9:25 am; edited 1 time in total 
magnum
PostPosted: Mon Apr 02, 2007 8:25 pm Reply with quote

ok cool thank you all for your help i hope the bandwidth drops from all this.
 
evaders99
Former Moderator in Good Standing


Joined: Apr 30, 2004
Posts: 3221

PostPosted: Tue Apr 03, 2007 8:06 pm Reply with quote

My server uses about 300-400 GB a month.. but that's because we do host several large downloads. The HTML text size is usually nothing.. its the downloads that cause significant bandwidth usage.

_________________
- Only registered users can see links on this board! Get registered or login! -

Need help? Only registered users can see links on this board! Get registered or login! 
View user's profile Send private message Visit poster's website
wiz
Involved
Involved


Joined: Oct 09, 2006
Posts: 413
Location: UK

PostPosted: Tue Aug 28, 2007 8:16 pm Reply with quote

Im suffering from the bandwidth issue right now. I see a huge increase this month and im now starting to wonder about robots.
Unfortuntaly Susann the link is dead now, any help would be great.

I see most of my robots bandwidth is from an unknown robot, is this bad?
 
View user's profile Send private message Visit poster's website AIM Address
evaders99
PostPosted: Tue Aug 28, 2007 10:10 pm Reply with quote

Could be someone leaching your downloads. Scrutinize whatever statistics packages you have to see where they are taking things from
 
Susann
PostPosted: Wed Aug 29, 2007 9:31 am Reply with quote

I removed the dead link.
Check this:
Only registered users can see links on this board! Get registered or login!

To find out what or who is behind the unknown bot just go trough your logfiles.
 
wiz
PostPosted: Wed Aug 29, 2007 7:34 pm Reply with quote

Thanks for the link Susann
 
wiz
PostPosted: Thu Aug 30, 2007 9:22 am Reply with quote

it seems the heaviest traffic is from a bot called Twiceler
 
Susann
PostPosted: Thu Aug 30, 2007 10:45 am Reply with quote

Do you use .htaccess to ban bad bots ? I added between other lines for Twiceler this:

RewriteCond %{HTTP_USER_AGENT} ^.*\(twiceler-0\.9.*$ [NC,OR]
 
wiz
PostPosted: Thu Aug 30, 2007 6:46 pm Reply with quote

Cheers again Susann. I havent currently written anything extra into htaccess.
So i was kind of hoping someone would post somrthing like you have.

Thanks greatly
 
Susann
PostPosted: Thu Aug 30, 2007 6:58 pm Reply with quote

wiz search here for an example of a .htaccess to ban bad bots and add this line into it. You can use robots.txt but I can tell you robots.txt doesn´t work for twiceler. Maybe ban Nutch also.
 
wiz
PostPosted: Thu Aug 30, 2007 7:31 pm Reply with quote

htaccess is another thing that im sadly lacking in knowledge on. However i do know that it sits at root, and therefore anything denied there has got to be final.

I usually search for the info, but as you know, the info is there; but the topic title sometimes does not reflect that. so the search is duff. Said again, i found this topic via search, thats why i posted here and didnt make a new one Idea

Thankyou again Susann, great sentinel (ish) support as always Wave
 
montego
Site Admin


Joined: Aug 29, 2004
Posts: 9457
Location: Arizona

PostPosted: Thu Aug 30, 2007 7:55 pm Reply with quote

By the way, rather than carrying about the actual version number, I have used this since April:

RewriteCond %{HTTP_USER_AGENT} ^Twiceler [NC,OR]

The NC means don't worry about the case and the OR is because this is one of about 160+ lines blocking various user agents.... BTW, I have posted this list in these forums before in case you want to start building up your repertoire... Wink However, do keep in mind that the more you stuff into .htaccess, the slower your site will become. Yes, I HAVE tested by removing my .htaccess before and the speed difference is very noticeable.

_________________
Only registered users can see links on this board! Get registered or login!
Only registered users can see links on this board! Get registered or login! 
View user's profile Send private message Visit poster's website
Display posts from previous:       
Post new topic   Reply to topic    Ravens PHP Scripts And Web Hosting Forum Index -> General/Other Stuff

View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum


Powered by phpBB © 2001-2007 phpBB Group
All times are GMT - 6 Hours
 
Forums ©