Bandwidth

magnum · Client Joined: Jun 23, 2006 Posts: 83

Im just wondering how much bandwidth your sites uses a month i have 3 gigs and now , every month hitting its limit so i got 3 more gigs total 6 gigs now and its still going to hit the limit in bandwidth so im wondering how much gigs do your pages use a month am i to low at 6 gigs? whats a normal use a month.

jakec · Posted: Sun Apr 01, 2007 2:21 pm

Wow, that sounds like a lot. Can you give us some stats? Number of users etc.

magnum · Client Joined: Jun 23, 2006 Posts: 83

i have about 20 users that are regular every day logons let me get the stats and ill post it up

magnum · Client Joined: Jun 23, 2006 Posts: 83

does this help?

Monthly Statistics for March 2007

Total Hits 792362
Total Files 472342
Total Pages 5834
Total Visits 3269
Total KBytes 4237645

Total Unique Sites 2193
Total Unique URLs 1633
Total Unique Referrers 104
Total Unique User Agents 225

. Avg Max

Hits per Hour 1065 9959
Hits per Day 25560 38995
Files per Day 15236 21318
Pages per Day 188 311
Visits per Day 105 132
KBytes per Day 136698 214621

Hits by Response Code

Undefined response code 1
Code 200 - OK 472342
Code 206 - Partial Content 1904
Code 301 - Moved Permanently 2
Code 302 - Found 27433
Code 304 - Not Modified 227312
Code 403 - Forbidden 9111
Code 404 - Not Found 54247
Code 405 - Method Not Allowed 2
Code 406 - Not Acceptable

Gremmie · Posted: Sun Apr 01, 2007 5:51 pm

I usually do about 13 Gigs a month but have gotten as high as 20 before.

djmaze · Posted: Sun Apr 01, 2007 6:09 pm

magnum wrote:

Code 206 - Partial Content 1904
Code 301 - Moved Permanently 2
Code 302 - Found 27433
Code 404 - Not Found 54247

Those are strange since phpnuke never sends those.

Code 304 could be your theme images and css files though and that is a good thing.

Raven · Posted: Sun Apr 01, 2007 10:52 pm

magnum,

Check your AWSTATS and see what referrer is chewing up your bandwidth. It may be a bad bot or something like that.

Susann · Posted: Mon Apr 02, 2007 5:18 am

Check for bad bots, unknown bots and when you find bots from China,Korea or Japan and you feel you don´t need results in asian searchengines ban them.
I`ve got a lot of spam from asian server in the past. So this can also reduce the spam flood.

Only registered users can see links on this board!
Get registered or login to the forums!

magnum · Client Joined: Jun 23, 2006 Posts: 83

Last Update: 02 Apr 2007 - 02:43

Reported period: Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec - Year - 2006 2007

Back to main page

Links from an Internet Search Engine
7 different refering search engines Pages Percent Hits Percent
Google 176 59.8 % 176 59.8 %
Yahoo 72 24.4 % 72 24.4 %
MSN 26 8.8 % 26 8.8 %
Unknown search engines 10 3.4 % 10 3.4 %
AOL 4 1.3 % 4 1.3 %
AltaVista 3 1 % 3 1 %
Netscape 3 1 % 3 1 %

--------------------------------------------------------------------------------------

Robots/Spiders visitors (Top 25) - Full list - Last visit
8 different robots* Hits Bandwidth Last visit
Googlebot 139719+60 1.46 GB 31 Mar 2007 - 23:16
Inktomi Slurp 2911+1471 28.93 MB 31 Mar 2007 - 23:58
Unknown robot (identified by 'crawl') 517+27 5.19 MB 31 Mar 2007 - 20:51
Unknown robot (identified by hit on 'robots.txt') 0+152 84.50 KB 31 Mar 2007 - 23:14
Unknown robot (identified by 'spider') 6+66 294.90 KB 31 Mar 2007 - 21:37
Unknown robot (identified by 'robot') 32+16 772.19 KB 30 Mar 2007 - 23:44
MSNBot 0+26 14.55 KB 31 Mar 2007 - 23:07
AskJeeves 9+9 311.55 KB 28 Mar 2007 - 09:54

* Robots shown here gave hits or traffic "not viewed" by visitors, so they are not included in other charts. Numbers after + are successful hits on "robots.txt" files

how do i stop googlebot or do i want too?

Susann · Posted: Mon Apr 02, 2007 10:36 am

Use site:islandtitanz.org and you ll find all your indexed sites at google.
I have the same results for Slurp montly thats normal but 1.46 GB for Googlebot ?

Check this:http://www.google.com/support/webmasters/

but I would not disallow Googlebot general, because its an important search engine.

magnum · Client Joined: Jun 23, 2006 Posts: 83

would this slow googlebot down in my robots.txt or is something else better?

The following block of code DISALLOWS the crawling of all files and directories to the following spiders/robots: Slurp msnbot
Line 1 User-agent: Slurp
Line 2 Crawl-delay: 20
Line 3
Line 4
Line 5 User-agent: msnbot
Line 6 Disallow: /
Line 7

The following block of code DISALLOWS the crawling of all files and directories to the following spiders/robots: Inktomi Slurp
Line 8 User-agent: Inktomi Slurp
Line 9 Disallow: /
Line 10
Line 11
Line 12

The following block of code DISALLOWS the crawling of the following files and directories: /modules.php?name=Top&querylang=union%20select%200,pwd?id=honeytrap /abuse/ /admin/ /blocks/ /cgi-bin/ /db/ /images/ /includes/ /language/ /modules/ /public/ /themes/ /tmp/ /admin.php /config.php /demohack.php /cplogin.php /hackattempt.php /login.php to the following spiders/robots: googlebot
Line 13 User-agent: googlebot
Line 14 Disallow: /modules.php?name=Top&querylang=union%20select%200,pwd?id=honeytrap
Line 15 Disallow: /abuse/
Line 16 Disallow: /admin/
Line 17 Disallow: /blocks/
Line 18 Disallow: /cgi-bin/
Line 19 Disallow: /db/
Line 20 Disallow: /images/
Line 21 Disallow: /includes/
Line 22 Disallow: /language/
Line 23 Disallow: /modules/
Line 24 Disallow: /public/
Line 25 Disallow: /themes/
Line 26 Disallow: /tmp/
Line 27 Disallow: /admin.php
Line 28 Disallow: /config.php
Line 29 Disallow: /demohack.php
Line 30 Disallow: /cplogin.php
Line 31 Disallow: /hackattempt.php
Line 32 Disallow: /login.php
Line 33

jakec · Posted: Mon Apr 02, 2007 11:08 am

If you have an account setup with Google, you can set a crawl delay in there.

Susann · Posted: Mon Apr 02, 2007 6:31 pm

Inktomi Slurp is from Yahoo. So your robots.txt isn´t correctly.
Check this link and the redirect:

If you use crawl-delay for Slurp you can try it also with 30 or 60 and afterthat check your logfiles and make sure that your robots.txt is valid.
Don´t know if that works for top.html&.
But like Jakec said you can use crawl-delay and also check different things with an account.

magnum · Client Joined: Jun 23, 2006 Posts: 83

ok cool thank you all for your help i hope the bandwidth drops from all this.

evaders99 · Posted: Tue Apr 03, 2007 8:06 pm

My server uses about 300-400 GB a month.. but that's because we do host several large downloads. The HTML text size is usually nothing.. its the downloads that cause significant bandwidth usage.

wiz · Client Joined: Oct 09, 2006 Posts: 394 Location: UK

Im suffering from the bandwidth issue right now. I see a huge increase this month and im now starting to wonder about robots.
Unfortuntaly Susann the link is dead now, any help would be great.

I see most of my robots bandwidth is from an unknown robot, is this bad?

evaders99 · Posted: Tue Aug 28, 2007 10:10 pm

Could be someone leaching your downloads. Scrutinize whatever statistics packages you have to see where they are taking things from

Susann · Posted: Wed Aug 29, 2007 9:31 am

I removed the dead link.
Check this:

Only registered users can see links on this board!
Get registered or login to the forums!

To find out what or who is behind the unknown bot just go trough your logfiles.

wiz · Client Joined: Oct 09, 2006 Posts: 394 Location: UK

Thanks for the link Susann

wiz · Client Joined: Oct 09, 2006 Posts: 394 Location: UK

it seems the heaviest traffic is from a bot called Twiceler

Susann · Posted: Thu Aug 30, 2007 10:45 am

Do you use .htaccess to ban bad bots ? I added between other lines for Twiceler this:

RewriteCond %{HTTP_USER_AGENT} ^.*\(twiceler-0\.9.*$ [NC,OR]

wiz · Client Joined: Oct 09, 2006 Posts: 394 Location: UK

Cheers again Susann. I havent currently written anything extra into htaccess.
So i was kind of hoping someone would post somrthing like you have.

Thanks greatly

Susann · Posted: Thu Aug 30, 2007 6:58 pm

wiz search here for an example of a .htaccess to ban bad bots and add this line into it. You can use robots.txt but I can tell you robots.txt doesn´t work for twiceler. Maybe ban Nutch also.

wiz · Client Joined: Oct 09, 2006 Posts: 394 Location: UK

htaccess is another thing that im sadly lacking in knowledge on. However i do know that it sits at root, and therefore anything denied there has got to be final.

I usually search for the info, but as you know, the info is there; but the topic title sometimes does not reflect that. so the search is duff. Said again, i found this topic via search, thats why i posted here and didnt make a new one Idea

Thankyou again Susann, great sentinel (ish) support as always Wave

montego · Posted: Thu Aug 30, 2007 7:55 pm

By the way, rather than carrying about the actual version number, I have used this since April:

RewriteCond %{HTTP_USER_AGENT} ^Twiceler [NC,OR]

The NC means don't worry about the case and the OR is because this is one of about 160+ lines blocking various user agents.... BTW, I have posted this list in these forums before in case you want to start building up your repertoire... Wink

However, do keep in mind that the more you stuff into .htaccess, the slower your site will become. Yes, I HAVE tested by removing my .htaccess before and the speed difference is very noticeable.