Try adjusting the following settings:

define('CAPTCHA_WIDTH', 150);
define('CAPTCHA_MAX_FONT_SIZE', 16);

There is a good discussion on this at Login error due to Captcha/Security image

You are using v-e-r-y old code.

Look in mainfile.php and at or around line 218 and you should see this block of code. Read the comments and make a decision as to which way you want to go. We recommend updating your code to use the $db logic instead but the choice is yours, at least for this release :)

/*

 * The following two lines of code were moved and commented out in RN 2.30.00 to "test the waters" on

 * finding out what old modules/blocks/hacks/etc. are still using this SQL layer that is so

 * old it should be obsoleted/removed.  If you really need these, uncomment them.  We will keep this

 * code this way for one more major release as well as keep the includes/sql_layer.php script.

 */

//@require_once(INCLUDE_PATH.'includes/sql_layer.php');

//$dbi = sql_connect($dbhost, $dbuname, $dbpass, $dbname);