Akismet Module

Involved Joined: Apr 06, 2008 Posts: 484

So i have dug out the akismet module I was working on a long time back and decided to investigate it a little further ( gives me something to do while I'm away Smile

).

basically I'm after some thoughts, idea's, abuse on what should be there, shouldn't be there.Basically any thoughts that may help me. (and don't be shy to yell out if your interested in helping)

easiest way is a list of thoughts so far i suppose.

a way to submit missed spam
a way to submit false positives (ham)
Key Verification
Include all $_SERVER i.e akismet would like this

I was using a class I found but decided to start fresh with functions, I have got as far as;

checking a connection is possible to akismet servers.
checking key verification
checking if a comment is spam or ham (all $_SERVER is included)

Sample Output;

Code:




resource(3) of type (stream)





string(243) "POST /1.1/verify-key HTTP/1.0


Host: rest.akismet.com


Content-Type: application/x-www-form-urlencoded; charset=utf-8


Content-Length: 52


User-Agent: Ravennuke/2.4.0 | akismet_module/1.0





key=6c6f385dd095&blog=http://localhost/test/akismet/"





Connection OK





resource(4) of type (stream)





string(243) "POST /1.1/verify-key HTTP/1.0


Host: rest.akismet.com


Content-Type: application/x-www-form-urlencoded; charset=utf-8


Content-Length: 52


User-Agent: Ravennuke/2.4.0 | akismet_module/1.0





key=6c6f385dd095&blog=http://localhost/test/akismet/"





Key Verification Successfull





resource(5) of type (stream)





string(1094) "POST /1.1/comment-check HTTP/1.0


Host: 6c6f385dd095.rest.akismet.com


Content-Type: application/x-www-form-urlencoded; charset=utf-8


Content-Length: 886


User-Agent: Ravennuke/2.4.0 | akismet_module/1.0





blog=http://localhost/test/akismet/&HTTP_USER_AGENT=Mozilla%2F5.0+%28Windows%3B+U%3B+Windows+NT+5.1%3B+en-GB%3B+rv%3A1.9.1.5%29+Gecko%2F20091102+Firefox%2F3.5.5+%28.NET+CLR+3.5.30729%29&HTTP_ACCEPT=text%2Fhtml%2Capplication%2Fxhtml%2Bxml%2Capplication%2Fxml%3Bq%3D0.9%2C%2A%2F%2A%3Bq%3D0.8&HTTP_ACCEPT_LANGUAGE=en-gb%2Cen%3Bq%3D0.5&HTTP_ACCEPT_ENCODING=gzip%2Cdeflate&HTTP_ACCEPT_CHARSET=ISO-8859-1%2Cutf-8%3Bq%3D0.7%2C%2A%3Bq%3D0.7&SERVER_NAME=localhost&SERVER_ADDR=127.0.0.1&REMOTE_ADDR=127.0.0.1&REMOTE_PORT=1755&REQUEST_METHOD=GET&REQUEST_URI=%2Ftest%2Fakismet%2Findex.php&SCRIPT_NAME=%2Ftest%2Fakismet%2Findex.php&user_ip=127.0.0.1&user_agent=Mozilla%2F5.0+%28Windows%3B+U%3B+Windows+NT+5.1%3B+en-GB%3B+rv%3A1.9.1.5%29+Gecko%2F20091102+Firefox%2F3.5.5+%28.NET+CLR+3.5.30729%29&referer=&comment_author=Testy1&comment_author_email=&comment_author_url=&comment_content=viagra-test-123"





Comment Is SPAM

Posted: Sat Nov 07, 2009 5:48 am

i've thought of it and of a simple class to implement that but failed miserably.

if you have something good , I'm interrested for my Simple_Forum module because I don't want client side regulations Smile

Posted: Sat Nov 07, 2009 8:45 am

That would be a very nice addon. Are you integrating it into the whole site of just the forums? Either way it would be a major plus.

Posted: Sat Nov 07, 2009 10:57 pm

nuken wrote:

That would be a very nice addon. Are you integrating it into the whole site of just the forums? Either way it would be a major plus.

the idea was primarily for raven as there was talk of the forums becoming modular

ok according to akismet

Quote:

If it is at all possible, please modify the user agent string you request with to be of the following format:

Application Name/Version | Plugin Name/Version

As per akismet we would need the following user agent

Code:




define('AKISMET_API_USERAGENT', 'Ravennuke/'. RAVENNUKE_VERSION_FRIENDLY .' | akismet_module/'. AKISMET_MODULE_VERSION);

Outputs

Quote:

Ravennuke/02.40.00 | akismet_module/1.0

This function sends the POST request to akismet with a timeout of 3 seconds (The timeout could be made configurable???). The HTTP request is constructed with full headers. The response headers are discarded and the function returns the body of the response.

Code:




function rnAkismet_http_post($request, $host, $path) {





   $http_request  = "POST $path HTTP/1.0\r\n";


   $http_request .= "Host: $host\r\n";


   $http_request .= "Content-Type: application/x-www-form-urlencoded; charset=utf-8\r\n";


   $http_request .= "Content-Length: " . strlen($request) . "\r\n";


   $http_request .= "User-Agent: " . AKISMET_API_USERAGENT . "\r\n";


   $http_request .= "\r\n";


   $http_request .= $request;





  $response = '';


  if (false !== ($fs = @fsockopen($host, AKISMET_API_PORT, $errno, $errstr, 3))) {


    fwrite($fs, $http_request);


    while (!feof($fs)) {


      $response .= fgets($fs, 1160); // One TCP-IP packet


    }


    fclose($fs);


    $response = explode("\r\n\r\n", $response, 2);


  }


  


  return $response;


}

Key Verification

Quote:

Key Verification — rest.akismet.com/1.1/verify-key

The key verification call should be made before beginning to use the service. It requires two variables, key and blog.

key (required)
The API key being verified for use with the API
blog (required)
The front page or home URL of the instance making the request. For a blog, site, or wiki this would be the front page. Note: Must be a full URI, including [ Only registered users can see links on this board! Get registered or login! ]

The call returns "valid" if the key is valid. This is the one call that can be made without the API key subdomain.

Raven Key Verification Function

Code:




function rnAkismet_verify_key($key, $ip = null) {


   global $nukeurl;


  if (!empty($key)) {


    $request = 'key='. $key .'&blog='. $nukeurl;


    $response = rnAkismet_http_post($request, AKISMET_API_HOST, '/'. AKISMET_API_VERSION .'/verify-key', $ip);


  }


  if (isset($response[1]) && 'valid' == $response[1]) {


      return true;


  } else {


      return false;


  }


}

Comment Check

ok this is where Im having design difficulties.Akismet request the following;

Quote:

This is basically the core of everything. This call takes a number of arguments and characteristics about the submitted content and then returns a thumbs up or thumbs down. Almost everything is optional, but performance can drop dramatically if you exclude certain elements. I would recommend erring on the side of too much data, as everything is used as part of the Akismet signature.

blog (required)
The front page or home URL of the instance making the request. For a blog or wiki this would be the front page. Note: Must be a full URI, including [ Only registered users can see links on this board! Get registered or login! ]
user_ip (required)
IP address of the comment submitter.
user_agent (required)
User agent information.
referrer (note spelling)
The content of the HTTP_REFERER header should be sent here.
permalink
The permanent location of the entry the comment was submitted to.
comment_type
May be blank, comment, trackback, pingback, or a made up value like "registration".
comment_author
Submitted name with the comment
comment_author_email
Submitted email address
comment_author_url
Commenter URL.
comment_content
The content that was submitted.
Other server enviroment variables
In PHP there is an array of enviroment variables called $_SERVER which contains information about the web server itself as well as a key/value for every HTTP header sent with the request. This data is highly useful to Akismet as how the submited content interacts with the server can be very telling, so please include as much information as possible.

This call returns either "true" or "false" as the body content. True means that the comment is spam and false means that it isn't spam. If you are having trouble triggering you can send "viagra-test-123" as the author and it will trigger a true response, always.

Idea: - use function akismet_prepare_comment_data to prepare the data like user ip, referrer, username etc, the function also checks if a user is registered and inserts some info automatically. We could then send the data via the function rnAkismet_comment_check.The $slink would be the link to the particular content. e.g. if you were looking at a news article here

Code:




http://localhost/dfwmods/modules.php?name=News&file=article&sid=1

the slink would be;

Code:




article1.html


or probably more like


article.html'.$sid.'

The function;

Code:




function akismet_prepare_comment_data($sAuthor = 'Anonymous', $sEmail = '', $sLink, $sComment) {


   global $user;


   if (is_user($user)) {


   $userinfo = getusrinfo($user);


   } else {


      $userinfo = '';


   }


   


  // Prepare data that is common to nodes/comments.


  $comment_data = array();


    // IP address of the comment submitter.


   


    $comment_data['user_ip'] = $_SERVER['REMOTE_ADDR'];


    // User agent information of the comment submitter.


    $comment_data['user_agent'] = $_SERVER['HTTP_USER_AGENT'];


    // The content of the HTTP_REFERER header should be sent here.


    $comment_data['referer'] = isset($_SERVER['HTTP_REFERER']) ? $_SERVER['HTTP_REFERER'] : '';;


    // Submitted name with the comment.


   $comment_data['comment_author'] = (isset($userinfo['username']) ? $userinfo['username'] : $sAuthor);





    $comment_data['permalink'] = $nukeurl . $sLink;


    $comment_data['comment_author_email'] = (isset($userinfo['user_email']) ? $userinfo['user_email'] : $sEmail);


    $comment_data['comment_author_url'] = (isset($userinfo['user_website']) ? $userinfo['user_website'] : '');


    $comment_data['comment_content'] = $sComment;


  return $comment_data;


}

The following function would send the data via the function rnAkismet_http_post.The rnAkismet_comment_check function also ties in the server variables akismet requested (see further down).

Code:




function rnAkismet_comment_check($comment_data) {


  global $akismet_cfg;;


  if (!empty($akismet_cfg['api_key'])) {


    $comment_data = array_merge(rnAkismet_include_request(), $comment_data);


    $query_string = rnAkismet_build_string($comment_data);


    $host = $akismet_cfg['api_key'] .'.'. AKISMET_API_HOST;


    $response = rnAkismet_http_post($query_string, $host, '/'. AKISMET_API_VERSION .'/comment-check');


  }


  if (!isset($response[1])) {


    return AKISMET_API_RESULT_ERROR;


  }


  return ('true' == $response[1] ? AKISMET_API_RESULT_IS_SPAM : AKISMET_API_RESULT_IS_HAM);


}

Akismet also ask;

Quote:

Other server enviroment variables
In PHP there is an array of enviroment variables called $_SERVER which contains information about the web server itself as well as a key/value for every HTTP header sent with the request. This data is highly useful to Akismet as how the submited content interacts with the server can be very telling, so please include as much information as possible.

we could achieve this by the following function i borrowed from egroupware.

Code:




function rnAkismet_include_request() {


  // You may add more elements here, but they are often related to internal server


  // data that makes little sense to check whether a comment is spam or not.


  // Be sure to not send HTTP_COOKIE as it may compromise your user's privacy!


  static $safe_to_send = array(


    'CONTENT_LENGTH',


    'CONTENT_TYPE',


    'HTTP_ACCEPT',


    'HTTP_ACCEPT_CHARSET',


    'HTTP_ACCEPT_ENCODING',


    'HTTP_ACCEPT_LANGUAGE',


    'HTTP_REFERER',


    'HTTP_USER_AGENT',


    'REMOTE_ADDR',


    'REMOTE_PORT',


    'SCRIPT_URI',


    'SCRIPT_URL',


    'SERVER_ADDR',


    'SERVER_NAME',


    'REQUEST_METHOD',


    'REQUEST_URI',


    'SCRIPT_NAME'


  );





  // The contents of $_SERVER doesn't change between requests,


  // so we can have this cached in static storage.


  static $server_data;


  if (!$server_data) {


    $server_data = array();


    foreach ($_SERVER as $key => $value) {


      if (in_array($key, $safe_to_send)) {


        $server_data[$key] = $value;


      }


    }


  }


  return $server_data;


}

Next would be all comments classified as spam would be held in the database so an administrator could resubmit the comment as ham (not spam) in case of false positive's.Accordingly each comment should have a new link that will allow administrators to mark the comment as sapm in case it gets through.

I guess we could now tie all this in with a function in mainfile.

anything im missing....or can anyone see any problems with this.

Posted: Sat Nov 07, 2009 11:58 pm

ok here is the mainfile function so far

Code:




function check_spam($sAuthor = '', $sEmail = '', $sLink, $sComment) {


  global $akismet_cfg, $nukeurl, $user;


  include_once (INCLUDE_PATH . 'includes/akismet.inc.php');


  // First check Server Connectivity





  $Connect = akismet_check_server_connectivity();


  if ($Connect === true) {


    // Next Verify The Key


    $VerifyKey = rnAkismet_verify_key($akismet_cfg['api_key']);


    if ($VerifyKey === true) {


      if (is_user($user)) {


        $userinfo = getusrinfo($user);


      }


      $sAuthor = isset($userinfo['username']) ? $userinfo['username'] : 'Anonymous';


      $sEmail = isset($userinfo['user_email']) ? $userinfo['user_email'] : $sEmail;


      $sWebsite = isset($userinfo['user_website']) ? $userinfo['user_website'] : '';


      // The comment


      $user = '[a-zA-Z0-9_\-\.\+\^!#\$%&*+\/\=\?\`\|\{\}~\']+';


      $domain = '(?:(?:[a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\-]*[a-zA-Z0-9])\.?)+';


      $ipv4 = '[0-9]{1,3}(\.[0-9]{1,3}){3}';


      $ipv6 = '[0-9a-fA-F]{1,4}(\:[0-9a-fA-F]{1,4}){7}';


      $VerifyEmail = (bool)preg_match("/^$user@($domain|(\[($ipv4|$ipv6)\]))$/", $sEmail);


      $sEmail = (isset($VerifyEmail) && $VerifyEmail === true) ? $sEmail : '';


      $sLink = htmlentities($sLink, ENT_QUOTES);


      $sComment = htmlentities(strip_tags($sComment), ENT_QUOTES);


      // Next Prepare the Data


      $Prepare = akismet_prepare_comment_data($sAuthor, $sEmail, $sWebsite, $sLink, $sComment);


      $CheckComment = rnAkismet_comment_check($Prepare);


      if ($CheckComment === 1) {


        echo 'Comment Is SPAM!<br />';


      } else {


        echo 'Comment Is HAM!<br />';


      }


    } else {


      echo 'Key Verification Failed<br />';


    }


  } else {


    echo 'Connection Failed<br />';


  }


}

Output Test 1;

Code:




Ravennuke/02.40.00 | akismet_module/1.0





array(8) {


  ["user_ip"]=>


  string(9) "127.0.0.1"


  ["user_agent"]=>


  string(109) "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 (.NET CLR 3.5.30729)"


  ["referer"]=>


  string(45) "http://localhost/dfwmods/admin.php?op=akismet"


  ["comment_author"]=>


  string(6) "Testy1"


  ["permalink"]=>


  string(72) "http://localhost/dfwmodsmodules.php?name=News&file=article&sid=1"


  ["comment_author_email"]=>


  string(16) "egmade@local.com"


  ["comment_author_url"]=>


  string(23) "http://www.spamsite.com"


  ["comment_content"]=>


  string(29) "hi there people nice function"


}





Comment Is HAM!

Output Test 2;

Code:




Ravennuke/02.40.00 | akismet_module/1.0





array(8) {


  ["user_ip"]=>


  string(9) "127.0.0.1"


  ["user_agent"]=>


  string(109) "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 (.NET CLR 3.5.30729)"


  ["referer"]=>


  string(45) "http://localhost/dfwmods/admin.php?op=akismet"


  ["comment_author"]=>


  string(6) "Testy1"


  ["permalink"]=>


  string(72) "http://localhost/dfwmodsmodules.php?name=News&file=article&sid=1"


  ["comment_author_email"]=>


  string(16) "egmade@local.com"


  ["comment_author_url"]=>


  string(23) "http://www.spamsite.com"


  ["comment_content"]=>


  string(15) "viagra-test-123"


}





Comment Is SPAM!

Posted: Sun Nov 08, 2009 5:33 am

This is what I have been using [ Only registered users can see links on this board! Get registered or login! ]

Posted: Sun Nov 08, 2009 5:44 am

I was using similar but decided to go with a rewrite.do you think a class is the way to go?

Posted: Sun Nov 08, 2009 5:55 am

I would because you can just 'include' it whenever you need it, you don't have to have it loaded all the time. All the codes in one place so if Akismet ever changes, it's easier to find the bit of code you need to change.
I used it in a couple of modules but since I have not had any spam since RN 2.x it didn't seem worthwhile incorporating it any more.

Posted: Sun Nov 08, 2009 4:10 pm

ok Ill check it out thanks.

Site Admin Joined: Jun 04, 2004 Posts: 6432

Sorry I missed this originally. I was looking at this for a series of standalone form-to-email applications. I reviewed 4 libraries (either class, include functions, or both).

I remembered that Guardian was working on this, and asked him about it. His response might help others, so I am re-posting part of it here (thanks, G):

Guardian2003 wrote:

I think I would probably look at Project Honeypot as an initial filter and then add a manually updated blacklist as it would be more viable in terms of the man hours spent updating the blacklist once a month I should think.

If you're using a Class or developing your own code to access the Akismet API use the http1.0 protocol NOT http1.1 - 1.0 is much faster and works just as well."

Since Project Honeypot is IP-based, and IPs are so easy to spoof, I'm wondering how effective that would be. As far as stopping feedback / comment / forum spam, a content analysis approach like Akismet seems more effective. Checking the IP first might weed out a few spammers, but it would not be long before even those bottom-feeders circumvent that. Of course, Guardian has been looking at this MUCH longer than I - so I'd be especially interested in further discussion on this (hence this post)...

Also, I'm interested in further feedback on Akismet 1.0 vs. 1.1 - everything I've seen uses 1.1, though that could be just a "latest-is-greatest" mentality. Is there any different in the interface - or just the Akismet version parameter that gets passed?

Posted: Sun Jul 11, 2010 7:04 am

I think it would be a nice pre registration mod on any site. Which also takes real time operation out of the equation to some extent. I've been to lazy to pursue it as its not been an issue for me either.

Posted: Sun Jul 11, 2010 7:47 am

Six, to clarify, by "it" do you mean Project Honeypot or Akismet? And what do you mean by real-time operation? Thanks...

Posted: Sun Jul 11, 2010 8:42 am

I'd say either could apply but I was thinking of Akismet.
Realtime as in checking validity of a posters email on an anon reply to an active comment thread. Operation could be way to slow. Where a onsite database would be feasible as an option.
(not that allowing anon posters doesn't lead to trolls and flame wars but... it is done frequently)

Where registration confirmation delay would hardly be noticed.

Am I just confusing the use of Akismet and topic?

Could be just too tired to comprehend anything Exclamation

Edit some hours of sleep later:
Ok so someone registered on my site using a suspected spammer name/email so I decided to look at another option for fun.

Much like nukesentinel is [ Only registered users can see links on this board! Get registered or login! ] I'm testing it now.
Likes:
Easy to update csv file for ip bans.
Fast load

Dislikes:
Installer seems quirky easier to upload files to live site after setting them up locally.
Not a native solution duplicates some native checking.

Off topic will start a new thread after some testing.