« MT Plugin Network v1.0 | Main | FastSearch search relevance »

Spam Firewall v1.0 - Reduce the MT Spam Load

Two months ago I posted about the CPU load effect of Comment and Trackback spam. In short, even when most of the spam gets correctly marked as junk, it can still have a major impact on the CPU load and memory usage of the server. This can make everything else run slower, and some web hosts may threaten to disable your account for hogging CPU resources.

Spam Firewall is a plugin for Movable Type that can help reduce this load. The goal of Spam Firewall is to block 80% of spam attempts before they reach MT's comment and trackback scripts. As the name suggests, it acts like a firewall protecting your MT scripts.

How Spam Firewall Works

Spam Firewall creates a PHP script that acts as a firewall. Once installed, when comments or trackbacks are posted, they get posted to the PHP firewall script. The firewall script will then scan the the request against a "Top 40" list of common spam patterns. If the request matches, things stop right there. If the request passes the test, it gets forwarded to MT for processing in the normal manner. Most spam is blocked quickly, without consuming large amounts of CPU and memory.

The Top 40 List

The key to Spam Firewall is a "Top 40" list of common spam patterns. The 40 spam patterns (rules) released today represent 80% of the spam that I have received in the past 8 days. Of a total of 14,405 spam attempts, this Top 40 list matches 11,541 of them (80.1%). The list is based on real spam attempts from the past week. I plan on updating the Top 40 list regularly.

Requirements:

  • MT 3.2+
  • PHP (Note: you do not need to enable MT's dynamic publishing to use Spam Firewall, but your server must support PHP scripts -- most do)

Instructions:

  1. Download the zip file and extract its contents.
  2. Within the main 'Spam Firewall' directory, there are two folders, one names 'plugins' and one named 'php'. Upload both of these folders (including sub folders and files) into the same directory as Movable Type (often /cgi-bin/mt).
  3. Go to 'Settings' on the blog you want to setup, then choose 'Plugins', then choose the 'Settings' link under "Spam Firewall". Then click the "Click here to install the Spam Firewall Template for this blog" link. This will install the Spam Firewall index template and rebuild it. Advanced users can change the Output File name if desired, but you should not change the Name of the template.
  4. (Optional, but highly recommended) Rename your comment and trackback scripts [More Info]. Because Spam Firewall hides the name of these MT scripts, now is a very good time to rename them.
  5. To setup Spam Firewall for comments, you need to change the comments submission form to point to the firewall script. Go to your Individual Archive Template and look for the following:

    action="<$MTCGIPath$><$MTCommentScript$>"

    and replace it with:

    action="<$MTBlogURL$><$MTFirewallCommentScript$>"


  6. To setup Spam Firewall for trackbacks, look for the <$MTEntryTrackbackData$> and replace it with <$MTFirewallTrackbackData$>. Also, look for the <$MTEntryTrackbackLink$> and replace it with <$MTFirewallTrackbackLink$>.

  7. Rebuild all individual archives.

Download Spam Firewall

Non-commercial use - FREE ( In exchange for the free use of this plugin, I ask that you install the MT Plugin Network plugin. Plugin Network will help keep you up to date on the latest version (and rules) of Spam Firewall and other useful plugins. [more info])

Also, donations are appreciated:

Commercial use - $97.00

Download Now

Membership required, please sign-in or register:

As always, suggestions and feedback are appreciated. Please reply to this entry.

Rate this entry:

  • Currently 5/5
  • 1
  • 2
  • 3
  • 4
  • 5
Rating: 5/5 (1 votes cast). Powered by the Ajax Rating plugin.

TrackBack

TrackBack URL for this entry:

Listed below are links to weblogs that reference Spam Firewall v1.0 - Reduce the MT Spam Load:

» スパム対策いろいろ from オラオラ
スパム対策については、CGIリネームが一番効果がある。 [Read More]

» Movable Type plugin - SpamFirewall from LUX.ET.UMBRA
Lately, there hasn't been a lot superb plugins for MT. There's a couple here and there, but this one stood out. In fact a whole bunch by Mark Carey stood out. SpamFirewall basically hides your commenting/trackback ability from the slower... [Read More]

» Fathom Five Page Proofs Arrive from Bow. James Bow.
Before I talk about my other news for the day, I’d like to apologize for the slow pace of posts these past couple of days. I’ve been busy on a number of fronts, including the technical backend work on this... [Read More]

Comments (45)

Matt [TypeKey Profile Page]:

It'd be very cool if there was an option for this to integrate with Akismet, that would solve a lot of people's MT spam and load problems at the same time.

Jake [TypeKey Profile Page]:

Could this also work with MTDisguiseCommentURL? Just trying to lock things down as much as possible :-)

Jake, the answer is 'yes but'. ;)

There are 2 ways that it can be used with MTDiguiseCommentURL. The first is point the form at the firewall script, but then also use MTDisguiseCommentURL. With this approach, people with javascript enabled will post directly toi the MT comment script, and those without JS will post to the firewall script. This approach will appeal to people who want to keep the form \"accessible\", since javascript is not required to submit the form.

The second method is for those who want maximium spam protection and who are willing to "require" javascript. For this, you would follow the instructions for MTDisguiseCommentURL and point the form to a non-existant file. Then use the javascript to point the form to the firewall script. Note: I need to update MTDisguiseCommentURL to make this "just work". In the meantime you could replace the MTDisguiseCommentURL tag with similar javascript, but pointing to the firewall URL. That said, I will try to update MTDisguiseCommentURL and MTDisguiseTrackabckURL in the coming days, to have more direct support for Spam Firewall.

saahov [TypeKey Profile Page]:

If it is switched on authentication TypeKey at sending comment there is a mistake:
- Name and email address are required.
How it is possible to correct?

saahov, can you provide some additional details. Specifically, is the blog domain and the MT domain different? If so, you may need to install the firewall script on the same domain as MT, then point your comment scripts to that location. It sounds like the issue relates to the TypeKey cookie not being available to the firewall script because it is on a different domain.

My web server error_log is full of these:

35: [Mon Dec 25 04:39:19 2006] [error] [client 62.163.39.183] Subroutine MT::Template::Context::add_global_filter redefined at /chroot/home/conserva/conservativethinking.com/html/cgi-bin/blog/extlib/bradchoate/postproc.pm line 22.

I'm seeing the following problem in an MT 3.33 activity log:

Plugin error: /home/.../cgi-bin/mt/plugins/SpamFirewall/SpamFirewall.pl Can't locate object method "add_tag" via package "MT::Template::Context" at /home/.../cgi-bin/mt/plugins/SpamFirewall/SpamFirewal

I edited the paths in the error message above so that it wouldn't identify the blog.

--Dave Aiello

I solved the problem I mentioned above by adding the following line of code to SpamFirewall.pl:

use MT::Template::Context;

Can someone tell me why was this necessary? I'm a Perl programmer, but I don't have any experience writing MT plugins.

Thanks,

--Dave Aiello

Dafyd [TypeKey Profile Page]:

Mark, this is a great idea... I've been meaning to do something like this for a while, but don't have the skills.

One thing, though: I may be wrong, but there doesn't seem to be any PHP versions of the tags... meaning that it doesn't work on my dynamically-published archive pages. Am I doing something wrong, or is this a future feature?

Cheers,

Dafyd Jones

Toni [TypeKey Profile Page]:

Alogblog's CCode and TCode do this already. I have been using the plugin for about a year and comment and trackback spam have virtually ceased.

Toni, thanks for the pointer to CCode and TCode, but those do something completely different. While the goal may be the same, Spam Firewall and CCode take very different approaches.

Dafyd, yes, PHP versions of the tags may be included in a future version.

It should be noted here though that the tags are for convenience only. Meaning, you can use this plugin without the tags if you edit the templates appropriately. For example, for comments, you would use:

action="<$MTBlogURL$>spamfw.php"

(or whatever filename you chose for the firewal index template)

Admittedly, doing trackbacks without the tags is a little more complex, so I will try to add PHP versions of the tags in a future version.

Chris Short: I have seen errors like that for several years -- but I don't think they have anything to do with this plugin. It relates to a module released by Brad Choate a long time ago. I think I get this wanring because I use Brad's MTMacro plugin.

Dave Aiello: I am not sure why you were getting that error and others were not -- but I have added that line for the next version, just in case others have the same problem.

I tried to subscribe to this post (so that I could get a heads-u when the plugin had PHP versions of the tags, since I use dynamic publishing), but I got:

"The requested URL /cgi-bin/mt/mt-notifier.cgi was not found on this server."

FYI.

cbom [TypeKey Profile Page]:

After uploading the files, i can't find any word about "setting" to launch under spam firewall in plugins setting of my mt 3.34.

and then i tried to change the scripts in individul archive and rebuilt, it caused a error message-"Can't call method "outfile" on an undefined value at /home/onumhjov/public_html/cgi-bin/plugins/SpamFirewall/SpamFirewall.pl line 216."

was there any mistake or problem in mt3.34 or my step?

cheers~

cbom wu, taiwan

cbom,

The "settings" are on a per-blog basis. Browse to the blog you want, then choose Settings, then choose the Plugin tab, scroll down to Spam Firewall and click Settings. Note that there is only a link to install the spam firewall template -- there are no additional settings to choose.

The error you had sounds like the MT could not locate the spam firewall template. So go to the settings for the blog (as described above) and install the Spam Firewall index template. After that, it should work.

I just installed spam firewall and found it instantly helpful greatly reducing the spam load. My only issue so far. Many users have reported getting a long pause after hitting the submit button followed by a blank page. I too have seen this issue occasionally.

I just installed the firewall and found it instantly helpful greatly reducing the spam load. My only issue so far: Many users have reported getting a long pause after hitting the submit button followed by a blank page. I too have seen this issue occasionally.

raul, I haven't heard about such an issue previously. When they get a blank page like that, does the comment get submitted okay?

Is the page completely blank? Note that when a comment is blocked by the filters, a white page will be displayed will a single word: "blocked". That page can seem blank if you look really quickly -- just wondering whether this is what people are seeing. However, when comments get blocked by the firewall rules, that usually happens very quickly.

raul:

The comment is submitted... in fact when this happens most users tend to hit the back button and resubmit and I end up with duplicate comments... The page is completely blank. I didn't think to look at the source to see what was going on.

And to give you an idea of frequency I've had 2 users report it over the last 3 days since I installed...About 30 comments were submitted in that time period. (as a sidenote on both posts I've made here I've gotten an "Error: Your reply was not posted. Error Code: 500" the first time I tried to post.)

Just tested it out again after more complaints from users... the blank page is just that there is no html whatsoever when you view source... and yet the comment is submitted normally. Usually hitting the back button and submitting again leads to a normal comment approved for moderation dialog and a second submitted comment.

Hi Raul, that is odd. Without an error message, it is hard to figure out why this might happen. I have read that PHP will display a blank page if there is an error. Errors will be displayed in the browser only if this option in set in the PHP config file. This page explains two way that you may be able to turn on the display of errors. Also, there may be error in your web server error log.

Running the spamfirewall and finding that the trackbacks aren't working. It gives me an internal error 500.

When you look at the apache logs, it says:
Premature end of script headers: /spamfw.php

Never mind. Figured out the issue. Dynamic bootstrapper.

Found another issue. Trackbacks don't show up. At all. When you try to trackback any post, creates a new file in the template_c directory. But unfortunately the way it's trying to create the file, it hoses up when you try to access it.

darkmon, Spam Firewall does not have anything to do with creating the file or displaying the trackbacks. That seems to be an issue with MT dynamic publishing or your .htaccess file.

The 500 error seems to be related to to this plugin, but without furtehr detail, it is hard to track down.

It doesn't?

http://life.firelace.com/spamfw.php?tb_id=4274

Gives me an error of:
Parse error: syntax error, unexpected '}' in /home/darkmoon/public_html/templates_c/%%B2^B2E^B2E38223%%mt%3A61.php on line 8

I had MT support look at my dynamic publishing and .htaccess. It's working fine now, since I have fastsearch running so I know it works.

Thanks, darkmon. That link and error message provided additional detail. The spam firewall index (spamfw.php) must be built as a static template, not as a dynamic one.

Thanks for pointing this out, as this is something I should add to the documentation.

darkmoon [TypeKey Profile Page]:

Hrm. Still gives me the same error. I'm not building spamfw.php as dynamic. It's being built as a static template.

Any other thoughts as to how to fix this?

Mark Carey [TypeKey Profile Page]:

darkmon, that error indicates that MT is trying to build the file dynamically. Try rebuilding the file manually -- browse to the spam firewall template and click save and rebuild. Then to verify that the file was built, FTP to your blog root folder and look for the spamfw.php. If you see it there, then it was built statically -- if you still get the error, it must mean that there is an .htaccess problem.

darkmoon [TypeKey Profile Page]:

Mark, could you post an example of a good htaccess file? I'm using the one created by MT in 3.34 and I don't see what I need to change in it to make it work.

darkmoon [TypeKey Profile Page]:

Never mind. Hrm. There has to be a syntax error in the spamfw.php. The generated php in template_c has a syntax error. Tracking it down right now.

darkmon, that syntax error is likely caused by the fact that MT is trying to publish it dynamically, but that won't work. There are/is MT tags in that template that don't have PHP equivalents. Your server shouldn't be looking in templates_c at all for the file, it should find it in your blog root directory. Do you see the spamfw.php in the root directory when you look via FTP (or SSH)?

The important part of the .htacess is the following:

# don't serve mtview.php if the request is for a real file
# (allows the actual file to be served)
RewriteCond %{REQUEST_FILENAME} !-f
# anything else is handed to mtview.php for resolution
RewriteRule ^(.*)$ /mtview.php [L,QSA]

The "RewriteCond %{REQUEST_FILENAME} !-f" part says "check to see if the requested file actually exists as a static file, if so, serve the static file". If not, it will try to dynamically generate the page, via the last line above.

darkmoon [TypeKey Profile Page]:

okay.... who knows why it was messed up. I moved it to a new server and now it works. hah... man. frustrating but I think I'm finally done with it.

Glad you got it working, darkmon. ;)

I've installed Spam Firewall and really appreciate it. It seems to have cut down my spam load dramatically. However, I'm having a problem with TypeKey identification. I have a blog which is posting to a different domain from the MT installation, and when I try to log into Typekey and post a comment, my name and e-mail address doesn't carry over. What's the best way to fix this problem?

Hi James,

Glad the plugin is working for you.

The particular case you describe is currently a limitation. The PHP firewall script has access to the cookies on blog domain but not on the MT domain. So when the blog domain is different that the MT domain, there's no easy way to pass those TypeKey cookies across. One way to work around is to build the Spam Firewall template as you normall would, and then manually move (or copy) the built PHP file over to a location on the MT domain. Then, update the comment form on your Individual Archive template to point to the new location of of the firewall script. To be honest, I haven't tested this but I believe it should work. Please reply here and let me know either way.

Your workaround does the trick, although I have to create separate files for each blog that sits on a different domain. This isn't a problem, but it might complicate things if this plugin is updated.

But on the whole I'm really pleased. I also particularly like BlogJanitor.

smoothmomma [TypeKey Profile Page]:

I'm having a problem where many commenters are just getting a blank screen after they hit submit. The comments are going through, but the page isn't redirected. It doesn't happen every time and of course it doesn't happen when I'm trying to troubleshoot. error reporting is turned on in php.ini...

it seems that the page is just sometimes not redirecting... is there any simple way to leave a link on the page back to the post in case the re-direct doesn't work?

Also is there anything that can be done about the pause between submit and the the post showing up. The other problem I have is people hitting submit again after a few seconds because they think the blog is hung

smoothmomma, you are not the only person to run into the blank page issue. This has been reported by one other person, and I noticed this once myself. Same symptoms, but no errors seem to get reported anywhere, making it difficult to find the source of the problem. If you see anything odd in your error logs, please let me know. Look for both the spamfw.php as well as the comment script, since both get called in the process. Obviously, the comment posting is working here, but for some reason the firewall is not able to capture and display the result, and I am not sure why. The fact that this happens only occasionally makes it even more difficult to pinpoint. Maybe there is a timeout issue? I will keep trying to figure it out...

Mark Carey [TypeKey Profile Page]:

I think I answered my own question in my previous reply. Seems to be a timeout issue. Should be resolved in version 1.2, just released. Please reply to the 1.2 thread if you still get this problem.

Is there any way to strip the slashes that PHP is putting in front of apostrophes (i.e. \'). Can your script be edit as such that the slashes are stripped from the comment or trackback body?

Fabian, the latest version addresses this issue. See link for 1.2 above...

I've installed Spam Firewall

but Gives me an error of


(Smarty error: [in mt:16 line 18]: syntax error: unrecognized tag 'MTFirewallTrackbackData' (Smarty_Compiler.class.php, line 580

Post a comment

Gift idea: Buy Seinfeld DVD box set, complete 9 seasons!