PHPBB Problem Taking Down My Site 7.5

Regular Joined: Nov 09, 2003 Posts: 73

About 4 days ago my site started Dedicated Server Started having problems and would get overwhelmed with connections that would cause it to crash..

Now i had no changed or added anything like modules or addons to PHPBB to my system for at least a week prior to the to problem.. Anyways I have network Administrator that i pay to monitor my system as i have no idea about that stuff.. So he was working on it for days trying to figure what the problem was..

Quote:

After a long night trying to figure out what is going on, I think I have it at least narrowed down sufficiently.

It seems that when you restart the web server the self referrals go away. Let me show you what I am talking about first:

Netstat is the command that sows all current connections to the server and their startus. Here is a sample of the self referral that I am referring to:

tcp 0 0 ***.**.10.196:80 ***.**.10.196:46852 ESTABLISHED 9866/httpd

The first IP is the target IP, that is the IP of the site. The second IP is the IP of the client. Notice its the same? That is not the way it is supposed to be. There are, during period of the problem, literally 100-1000 of these connections at a time.

The server can in no way keep up with that number of active requests at a time. No server could. This causes apache and other processes to spiral out and crash.

So, then it became a question of where is it coming from? The server is somehow doing this to itself, the question again is how. This morning, when an attack was going on, I left apache up, but shut off the mysql server. This way, the web server would still server pages, but only the database problem page, not your normal content. As soon as I shut off the database, the self-connects disappeared. I turned the database back on and they reappeared within an hour.

So, that leads me to believe that someone somewhere has added a bad string to an article or board post. The site is fine until someone reads that article or post or shout or whatever, the bad code gets launched and the server struggles until it crashes.

So, This is about the end of what I can do for you. It is not a problem with the server, and it is not an external force. Your site is being triggered somehow to eat itself. I have read a little about PHPNuke and it seems that it is quite vulnerable to attack. One article I read seemed to affect an earlier version,. But through a http refer problem you could make it recursivle include itself which cause a similar effect as what we are seeing now. I am not implying that that specifically is the problem, but merely to point out that PHPNuke has some problems from time to time.

I would at a minimum check all the articles and items posted in the last couple of days and see if you can find the bad one that is in there. It is clearly something in the database since the connections go away as soon as the database server is down.

1. So after recieving I instantly installed the 2.0.11 patch which did not seem to help and problem persisted

2. Then i went ahead and installed the Nuke Sentinel 2.11 and put almost all the security on... Well all the main ones.. That didnt help either... Sad

3. My server was just crashing as everything would spike out of control, Connections, CPU, PROCESSES, all go throught the roof.. Then i would restart apache and mysql.. and almost immediately the problem would happen ago...

So then i disabled the FORUMS and the problem seem to stop instantly.. So i left the FORUMS off for a little over 24 hour period to see what would happen... And server ran Flawless.. Now this morning i turn the Forum Module Back on and it ran for about an hour than the same thing happened again...

Now I am lost on what I should do next??

My site is heavy traffic 50,000 hits a day..
PHPNuke 7.5

Any help on what the next step should be also..

Also now when I try logging into the FORUMS from the ADMIN PANEL i am getting this message???

Quote:

Information
You are not authorised to administer this board

Now that i upgraded to 2.0.11 and Nuke Sentinel i would believe it would have something to do with that as i never had thing problem before.. But just one more thing to add to my head ache.. Embarassed

Any help appreciated..
[ Only registered users can see links on this board! Get registered or login! ] is my site..

Regards
Rick
[/code]

Site Admin/Owner Joined: Aug 27, 2002 Posts: 17088

There is nothing at all in NukeSentinel that should relate to this. To prove one way or another, comment out the mainfile.php lines and see what happens.

Posted: Tue Dec 07, 2004 3:24 pm

yeah i commented out the Sentinel out of the mainfile.php and still the same problem.. must of been something when i updated with the 2.0.11.. i will restore my original files and see what happens..

any ideas on the major problem Sad

Posted: Tue Dec 07, 2004 3:56 pm

Unfortunately, no. But NukeSentinel is not involved. By commenting it out, no code ever gets called.

Useless Joined: Aug 23, 2002 Posts: 213 Location: Chicago

dirtbag, Did you buy any chance installed any additional mods to PHPbb? I have noticed the same thing when forums get close to 30-40 users, it crashes.

CPG-Nuke team has figured it out from my understanding and so has NukeCops but I can't find any information about that.

Steven (a user who used to post here and at NukeCops) did talk about few things but before he finished those hacks, he stoped posting.

Posted: Tue Dec 07, 2004 10:15 pm

yeah i did install addtional mods but its been in the past.... and it seem to be running fine..but lately my site has been picking up much more traffic...

i installed the

attachement mod
quick reply mod
Advanced_Username_Color_1.0.3
BBCode Mod

yeah the thing i am lookng at next is reinstalling the just the fresh default 7.5 Forums module over and the the additional files that were changed and see what happens...

the thing is it was fine until a couple of days ago. was wondering if it coule be some bad code somewhat put in post .. an exploit..

it sucks but i need to find a solution...

Posted: Wed Dec 08, 2004 6:16 am

I had attachment mod, quick reply and BB Code mod... I think it's somehow related to attachment mod.

Posted: Wed Dec 08, 2004 10:18 am

okay i will try removing it and see what happens.. you think it could be something someone attached?? i will removed today and fire the forums back up and see what happens

Posted: Wed Dec 08, 2004 12:21 pm

I think I found one of the problems... You do not have 404 setup in your .htaccess file and as you know, people upload attachments, use them in one post and sometime "shadow" them in other posts too but then they delete it for some reason and forget to change the shadow topics.
Your server searches for a 404 page but can't find it but this process is happening so many times in once (let's say one topic has 3 broken links and 4 people are watching it at a time, that's 12 404s at a time) that it's starts to lag your server and at some point crashes it.

On the other hand, I had 404 set up but had too many broken links+Googletap, I guess do not go well too much. Maybe Raven or Bob who know about servers better than us can tell us if that could be one of the reasons.

Posted: Wed Dec 08, 2004 12:27 pm

That should not matter. It is subsequent to the problem. In other words, the nuke/phpbb system is throwing a 404 before it discovers that there is not a server 404 file.

Posted: Wed Dec 08, 2004 1:20 pm

And what about the GoogleTap?

Posted: Wed Dec 08, 2004 2:55 pm

Yeah i have my site GoogleTap.. including the the forums... you think i should turn it off for the Forums??? As that could somehow be causing the crash...??

Posted: Wed Dec 08, 2004 5:31 pm

Is there anything showing in your server logs? I would have thought that a url that was being recursed would make a definite impression somewhere - but then if you are googletaped, it might not show the true url.
At least you seem to have narrowed it down to the forums so aapainfull, slow process of re-introducing the mods over a period of time should help you pin point it - good look mate.

Posted: Sat Dec 11, 2004 5:58 pm

well after 4 or 5 days I finally did track it down to something in the Forum after process of elimination.. And after further investiagtion one of the posts that should had like 400 reads... had about 75,000 so somehow it got caught in a loop.... That post had a lot of avatar picture that were attached and also some audio files as i was using the newest Attachment mod from Portedmods.com. anyways i am not going to reinstall it..

So basically i replaced all the forums file to default ones and removed all traces or the attachment mod including the tables from my database and all seems well..

thanks for the help as also patched my site with to 2.8 so everything should be secure and my sentinel is running...

Posted: Sat Dec 11, 2004 10:54 pm

Great to see you got it sorted dirtbag - I'm wondering now if someone was hotlinking to the avatar or audio file that your user uploaded????

Good detective work though.

Posted: Sun Dec 12, 2004 1:26 am

no.. it was something that would get caught it a loop... and my active connections would go from 100 to 2000 in a matter or minutes... and all the connections were coming from my own site...

the more i read deeper into articles over at portedmods.com the attachment module seems to have problems.. like i said i have ran in the past and on my site for year no problem until now.. anyways maybe someone attached something that would cause an exploit.. site been running fine since i got rid of it...

so i can live without it...

Posted: Tue Dec 14, 2004 10:05 am

Check this out man, [ Only registered users can see links on this board! Get registered or login! ]

Posted: Tue Dec 14, 2004 1:38 pm

thanks...

as that was probably the problem and i was one of the first to get hit... like i said I am just going to keep that off and stick to the basics..

Posted: Tue Dec 14, 2004 2:01 pm

There has to be a way to make it work right. I mean it is a great add-on and a lot of site owners need it.

Subject Matter Expert Joined: Feb 23, 2004 Posts: 358

First thing you'd have to do is to get your board upgraded to 2.0.11. Then you can use the fix file for the attachment mod and repair your Nuke port. If you're not handy enough with PHP and the phpBB/port process, then you'll have to wait for the port author to release a fixed version for Nuke. But again, the fixed version will obviously require your phpBB to be 2.0.11 if you haven't done that yet (and you should for other reasons, not just the attachment mod fix).

PHrEEk

Worker Joined: Aug 06, 2004 Posts: 192

[ Only registered users can see links on this board! Get registered or login! ] The updated attachment mod is there.

Dirtbag, you should check this post out. [ Only registered users can see links on this board! Get registered or login! ]

Code:

                                          ////////////////////////////////////////////////////////


                                  // Resize Remote Avatars mod


                                 // Make sure that both dimensions of remote avatars conform to the limits


                                  // set in the admin control panel. Use the width and height attributes 


                                  // on the <img> tag to control the image size in the browser. 


                                  // Reduce the largest dimension of the remote image to the maximum allowed 


                                 // in the ACP, then reduce the other dimension to maintain the aspect ratio.


                                  //


                                  // phpbb 2.0.6 impl


                                  // $poster_avatar = ( $board_config['allow_avatar_remote'] ) ? '<img src="' . $postrow[$i]['user_avatar'] . '" alt="" border="0" />' : '';





                                  // new impl


                                 list($width, $height) = @getimagesize($postrow[$i]['user_avatar']);


                                  $width_attr = '';


                                  $height_attr = '';


                                  // resize the avatar in the browser if either dimension is too large


                                  $resize = $width > $board_config['avatar_max_width'] || $height > $board_config['avatar_max_height'];





                                  // set max dimension and adjust the other according to the ratio


                                  if ( $resize )


                                  {


                                      if ( $width == $height ) 


                                      {


                                            $width_attr = ' width="' . $board_config['avatar_max_width'] . '"';


                                          $height_attr = ' height="' . $board_config['avatar_max_height'] . '"';


                                      }


                                         else if ( $width > $height )


                                      {


                                            $width_attr = ' width="' . $board_config['avatar_max_width'] . '"';


                                          $height_attr = ' height="' . $board_config['avatar_max_width'] * $height / $width . '"';


                                      }


                                         else // $height > $width


                                      {   


                                          $width_attr = ' width="' . $board_config['avatar_max_height'] * $width / $height . '"';


                                          $height_attr = ' height="' . $board_config['avatar_max_height'] . '"';


                                      }


                                  }


                                  $poster_avatar = ( $board_config['allow_avatar_remote'] ) ? '<img src="' . $postrow[$i]['user_avatar'] . '" alt="" border="0"' . $width_attr . $height_attr . '/>' : '';


                                  // end Resize Remote Avatars mod


                                 ////////////////////////////////////////////////////////>' : '';

That code was causing a similar problem to yours on another users site.