Home

Forum

Archives

Tags



Plugins:


AddCategory

AjaxRating

Blacklist32

BlacklistStats

BlogJanitor

CommentRegistration

DefineCommentPending

EntryPost

FastSearch

FlickrPhotos

FormatList

MTCategoryCommentCount

MTDisguiseCommentURL

MTDisguiseTrackbackURL

MTDynamic

MTTrackbackEntry

PluginNetwork

SmartRebuild

SpamFirewall

TemplateInheritance

TrustedCommenters







Spam Firewall v1.0 - Reduce the MT Spam Load

Keep track of this discussion by subscribing to the RSS feed for this discussion thread
Author Message
Mark







PostPosted: December 19, 2006 3:51 PM 

Two months ago I posted about the CPU load effect of Comment and Trackback spam. In short, even when most of the spam gets correctly marked as junk, it can still have a major impact on the CPU load and memory usage of the server. This can make everything else run slower, and some web hosts may threaten to disable your account for hogging CPU resources.

Spam Firewall is a plugin for Movable Type that can help reduce this load. The goal of Spam Firewall is to block 80% of spam attempts before they reach MT's comment and trackback scripts. As the name suggests, it acts like a firewall protecting your MT scripts.

How Spam Firewall Works

Spam Firewall creates a PHP script that acts as a firewall. Once installed, when comments or trackbacks are posted, they get posted to the PHP firewall script. The firewall script will then scan the the request against a "Top 40" list of common spam patterns. If the request matches, things stop right there. If the request passes the test, it gets forwarded to MT for processing in the normal manner. Most spam is blocked quickly, without consuming large amounts of CPU and memory.

The Top 40 List

The key to Spam Firewall is a "Top 40" list of common spam patterns. The 40 spam patterns (rules) released today represent 80% of the spam that I have received in the past 8 days. Of a total of 14,405 spam attempts, this Top 40 list matches 11,541 of them (80.1%). The list is based on real spam attempts from the past week. I plan on updating the Top 40 list regularly.

Requirements:

  • MT 3.2+
  • PHP (Note: you do not need to enable MT's dynamic publishing to use Spam Firewall, but your server must support PHP scripts -- most do)

Instructions:

  1. Download the zip file and extract its contents.
  2. Within the main 'Spam Firewall' directory, there are two folders, one names 'plugins' and one named 'php'. Upload both of these folders (including sub folders and files) into the same directory as Movable Type (often /cgi-bin/mt).
  3. Go to 'Settings' on the blog you want to setup, then choose 'Plugins', then choose the 'Settings' link under "Spam Firewall". Then click the "Click here to install the Spam Firewall Template for this blog" link. This will install the Spam Firewall index template and rebuild it. Advanced users can change the Output File name if desired, but you should not change the Name of the template.
  4. (Optional, but highly recommended) Rename your comment and trackback scripts [More Info]. Because Spam Firewall hides the name of these MT scripts, now is a very good time to rename them.
  5. To setup Spam Firewall for comments, you need to change the comments submission form to point to the firewall script. Go to your Individual Archive Template and look for the following:

    action="<$MTCGIPath$><$MTCommentScript$>"

    and replace it with:

    action="<$MTBlogURL$><$MTFirewallCommentScript$>"


  6. To setup Spam Firewall for trackbacks, look for the <$MTEntryTrackbackData$> and replace it with <$MTFirewallTrackbackData$>. Also, look for the <$MTEntryTrackbackLink$> and replace it with <$MTFirewallTrackbackLink$>.

  7. Rebuild all individual archives.

Download Spam Firewall

Non-commercial use - FREE ( In exchange for the free use of this plugin, I ask that you install the MT Plugin Network plugin. Plugin Network will help keep you up to date on the latest version (and rules) of Spam Firewall and other useful plugins. [more info])

Also, donations are appreciated:

Commercial use - $97.00

Download: SpamFirewall.zip

As always, suggestions and feedback are appreciated. Please reply to this entry.

Matt [TypeKey Profile Page]

Posts: 1

Reply: 1



PostPosted: December 19, 2006 3:47 PM 

It'd be very cool if there was an option for this to integrate with Akismet, that would solve a lot of people's MT spam and load problems at the same time.

Jake [TypeKey Profile Page]

Posts: 10

Reply: 2



PostPosted: December 19, 2006 4:33 PM 

Could this also work with MTDisguiseCommentURL? Just trying to lock things down as much as possible Smile

Mark Carey [TypeKey Profile Page]

Posts: 150

Reply: 3



PostPosted: December 19, 2006 4:59 PM 

Jake, the answer is 'yes but'. Wink

There are 2 ways that it can be used with MTDiguiseCommentURL. The first is point the form at the firewall script, but then also use MTDisguiseCommentURL. With this approach, people with javascript enabled will post directly toi the MT comment script, and those without JS will post to the firewall script. This approach will appeal to people who want to keep the form \"accessible\", since javascript is not required to submit the form.

The second method is for those who want maximium spam protection and who are willing to "require" javascript. For this, you would follow the instructions for MTDisguiseCommentURL and point the form to a non-existant file. Then use the javascript to point the form to the firewall script. Note: I need to update MTDisguiseCommentURL to make this "just work". In the meantime you could replace the MTDisguiseCommentURL tag with similar javascript, but pointing to the firewall URL. That said, I will try to update MTDisguiseCommentURL and MTDisguiseTrackabckURL in the coming days, to have more direct support for Spam Firewall.

saahov [TypeKey Profile Page]

Posts: 1

Reply: 4



PostPosted: December 20, 2006 2:24 AM 

If it is switched on authentication TypeKey at sending comment there is a mistake:
- Name and email address are required.
How it is possible to correct?

Mark Carey [TypeKey Profile Page]

Posts: 150

Reply: 5



PostPosted: December 20, 2006 6:22 AM 

saahov, can you provide some additional details. Specifically, is the blog domain and the MT domain different? If so, you may need to install the firewall script on the same domain as MT, then point your comment scripts to that location. It sounds like the issue relates to the TypeKey cookie not being available to the firewall script because it is on a different domain.

Chris Short [TypeKey Profile Page]

Posts: 1

Reply: 6



PostPosted: December 25, 2006 6:25 AM 

My web server error_log is full of these:

35: [Mon Dec 25 04:39:19 2006] [error] [client 62.163.39.183] Subroutine MT::Template::Context::add_global_filter redefined at /chroot/home/conserva/conservativethinking.com/html/cgi-bin/blog/extlib/bradchoate/postproc.pm line 22.

Dave Aiello [TypeKey Profile Page]

Posts: 2

Reply: 7



PostPosted: January 1, 2007 12:37 PM 

I'm seeing the following problem in an MT 3.33 activity log:

Plugin error: /home/.../cgi-bin/mt/plugins/SpamFirewall/SpamFirewall.pl Can't locate object method "add_tag" via package "MT::Template::Context" at /home/.../cgi-bin/mt/plugins/SpamFirewall/SpamFirewal

I edited the paths in the error message above so that it wouldn't identify the blog.

--Dave Aiello

Dave Aiello [TypeKey Profile Page]

Posts: 2

Reply: 8



PostPosted: January 3, 2007 3:34 PM 

I solved the problem I mentioned above by adding the following line of code to SpamFirewall.pl:

use MT::Template::Context;

Can someone tell me why was this necessary? I'm a Perl programmer, but I don't have any experience writing MT plugins.

Thanks,

--Dave Aiello

Dafyd [TypeKey Profile Page]

Posts: 2

Reply: 9



PostPosted: January 6, 2007 1:01 PM 

Mark, this is a great idea... I've been meaning to do something like this for a while, but don't have the skills.

One thing, though: I may be wrong, but there doesn't seem to be any PHP versions of the tags... meaning that it doesn't work on my dynamically-published archive pages. Am I doing something wrong, or is this a future feature?

Cheers,

Dafyd Jones

Toni [TypeKey Profile Page]

Posts: 1

Reply: 10



PostPosted: January 7, 2007 5:16 AM 

Alogblog's CCode and TCode do this already. I have been using the plugin for about a year and comment and trackback spam have virtually ceased.

Mark Carey [TypeKey Profile Page]

Posts: 150

Reply: 11



PostPosted: January 10, 2007 8:38 AM 

Toni, thanks for the pointer to CCode and TCode, but those do something completely different. While the goal may be the same, Spam Firewall and CCode take very different approaches.

Mark Carey [TypeKey Profile Page]

Posts: 150

Reply: 12



PostPosted: January 10, 2007 8:44 AM 

Dafyd, yes, PHP versions of the tags may be included in a future version.

It should be noted here though that the tags are for convenience only. Meaning, you can use this plugin without the tags if you edit the templates appropriately. For example, for comments, you would use:

action="<$MTBlogURL$>spamfw.php"

(or whatever filename you chose for the firewal index template)

Admittedly, doing trackbacks without the tags is a little more complex, so I will try to add PHP versions of the tags in a future version.

Mark Carey [TypeKey Profile Page]

Posts: 150

Reply: 13



PostPosted: January 10, 2007 8:52 AM 

Chris Short: I have seen errors like that for several years -- but I don't think they have anything to do with this plugin. It relates to a module released by Brad Choate a long time ago. I think I get this wanring because I use Brad's MTMacro plugin.

Dave Aiello: I am not sure why you were getting that error and others were not -- but I have added that line for the next version, just in case others have the same problem.

*** Dave [TypeKey Profile Page]

Posts: 3

Reply: 14



PostPosted: January 16, 2007 1:34 AM 

I tried to subscribe to this post (so that I could get a heads-u when the plugin had PHP versions of the tags, since I use dynamic publishing), but I got:

"The requested URL /cgi-bin/mt/mt-notifier.cgi was not found on this server."

FYI.

cbom [TypeKey Profile Page]

Posts: 1

Reply: 15



PostPosted: January 20, 2007 1:33 PM 

After uploading the files, i can't find any word about "setting" to launch under spam firewall in plugins setting of my mt 3.34.

and then i tried to change the scripts in individul archive and rebuilt, it caused a error message-"Can't call method "outfile" on an undefined value at /home/onumhjov/public_html/cgi-bin/plugins/SpamFirewall/SpamFirewall.pl line 216."

was there any mistake or problem in mt3.34 or my step?

cheers~

cbom wu, taiwan

Mark Carey [TypeKey Profile Page]

Posts: 44

Reply: 16



PostPosted: January 20, 2007 5:00 PM 

cbom,

The "settings" are on a per-blog basis. Browse to the blog you want, then choose Settings, then choose the Plugin tab, scroll down to Spam Firewall and click Settings. Note that there is only a link to install the spam firewall template -- there are no additional settings to choose.

The error you had sounds like the MT could not locate the spam firewall template. So go to the settings for the blog (as described above) and install the Spam Firewall index template. After that, it should work.

raul Gutierrez

Posts: 4

Reply: 17



PostPosted: January 26, 2007 10:03 AM 

I just installed spam firewall and found it instantly helpful greatly reducing the spam load. My only issue so far. Many users have reported getting a long pause after hitting the submit button followed by a blank page. I too have seen this issue occasionally.

raul Gutierrez [TypeKey Profile Page]

Posts: 4

Reply: 18



PostPosted: January 26, 2007 10:04 AM 

I just installed the firewall and found it instantly helpful greatly reducing the spam load. My only issue so far: Many users have reported getting a long pause after hitting the submit button followed by a blank page. I too have seen this issue occasionally.

Mark Carey [TypeKey Profile Page]

Posts: 44

Reply: 19



PostPosted: January 26, 2007 11:30 AM 

raul, I haven't heard about such an issue previously. When they get a blank page like that, does the comment get submitted okay?

Is the page completely blank? Note that when a comment is blocked by the filters, a white page will be displayed will a single word: "blocked". That page can seem blank if you look really quickly -- just wondering whether this is what people are seeing. However, when comments get blocked by the firewall rules, that usually happens very quickly.

raul

Posts: 3

Reply: 20



PostPosted: January 26, 2007 11:46 AM 

The comment is submitted... in fact when this happens most users tend to hit the back button and resubmit and I end up with duplicate comments... The page is completely blank. I didn't think to look at the source to see what was going on.

And to give you an idea of frequency I've had 2 users report it over the last 3 days since I installed...About 30 comments were submitted in that time period. (as a sidenote on both posts I've made here I've gotten an "Error: Your reply was not posted. Error Code: 500" the first time I tried to post.)

raul Gutierrez [TypeKey Profile Page]

Posts: 3

Reply: 21



PostPosted: January 30, 2007 1:16 PM 

Just tested it out again after more complaints from users... the blank page is just that there is no html whatsoever when you view source... and yet the comment is submitted normally. Usually hitting the back button and submitting again leads to a normal comment approved for moderation dialog and a second submitted comment.

Mark Carey [TypeKey Profile Page]

Posts: 44

Reply: 22



PostPosted: January 31, 2007 8:48 AM 

Hi Raul, that is odd. Without an error message, it is hard to figure out why this might happen. I have read that PHP will display a blank page if there is an error. Errors will be displayed in the browser only if this option in set in the PHP config file. This page explains two way that you may be able to turn on the display of errors. Also, there may be error in your web server error log.

darkmoon

Posts: 23

Reply: 23



PostPosted: February 6, 2007 1:30 PM 

Running the spamfirewall and finding that the trackbacks aren't working. It gives me an internal error 500.

When you look at the apache logs, it says:
Premature end of script headers: /spamfw.php

darkmoon

Posts: 23

Reply: 24



PostPosted: February 7, 2007 6:10 AM 

Never mind. Figured out the issue. Dynamic bootstrapper.

darkmoon [TypeKey Profile Page]

Posts: 23

Reply: 25



PostPosted: February 7, 2007 6:52 AM 

Found another issue. Trackbacks don't show up. At all. When you try to trackback any post, creates a new file in the template_c directory. But unfortunately the way it's trying to create the file, it hoses up when you try to access it.

Mark Carey [TypeKey Profile Page]

Posts: 44

Reply: 26



PostPosted: February 7, 2007 8:05 AM 

darkmon, Spam Firewall does not have anything to do with creating the file or displaying the trackbacks. That seems to be an issue with MT dynamic publishing or your .htaccess file.

The 500 error seems to be related to to this plugin, but without furtehr detail, it is hard to track down.

darkmoon [TypeKey Profile Page]

Posts: 23

Reply: 27



PostPosted: February 7, 2007 8:32 AM 

It doesn't?

http://life.firelace.com/spamfw.php?tb_id=4274

Gives me an error of:
Parse error: syntax error, unexpected '}' in /home/darkmoon/public_html/templates_c/%%B2^B2E^B2E38223%%mt%3A61.php on line 8

I had MT support look at my dynamic publishing and .htaccess. It's working fine now, since I have fastsearch running so I know it works.

Mark Carey [TypeKey Profile Page]

Posts: 44

Reply: 28



PostPosted: February 7, 2007 10:03 AM 

Thanks, darkmon. That link and error message provided additional detail. The spam firewall index (spamfw.php) must be built as a static template, not as a dynamic one.

Thanks for pointing this out, as this is something I should add to the documentation.

darkmoon [TypeKey Profile Page]

Posts: 23

Reply: 29



PostPosted: February 7, 2007 11:37 AM 

Hrm. Still gives me the same error. I'm not building spamfw.php as dynamic. It's being built as a static template.

Any other thoughts as to how to fix this?

Mark Carey [TypeKey Profile Page]

Posts: 44

Reply: 30



PostPosted: February 7, 2007 3:12 PM 

darkmon, that error indicates that MT is trying to build the file dynamically. Try rebuilding the file manually -- browse to the spam firewall template and click save and rebuild. Then to verify that the file was built, FTP to your blog root folder and look for the spamfw.php. If you see it there, then it was built statically -- if you still get the error, it must mean that there is an .htaccess problem.

darkmoon [TypeKey Profile Page]

Posts: 23

Reply: 31



PostPosted: February 8, 2007 6:47 AM 

Mark, could you post an example of a good htaccess file? I'm using the one created by MT in 3.34 and I don't see what I need to change in it to make it work.

darkmoon [TypeKey Profile Page]

Posts: 23

Reply: 32



PostPosted: February 8, 2007 8:06 AM 

Never mind. Hrm. There has to be a syntax error in the spamfw.php. The generated php in template_c has a syntax error. Tracking it down right now.

Mark Carey [TypeKey Profile Page]

Posts: 44

Reply: 33



PostPosted: February 8, 2007 3:33 PM 

darkmon, that syntax error is likely caused by the fact that MT is trying to publish it dynamically, but that won't work. There are/is MT tags in that template that don't have PHP equivalents. Your server shouldn't be looking in templates_c at all for the file, it should find it in your blog root directory. Do you see the spamfw.php in the root directory when you look via FTP (or SSH)?

Mark Carey [TypeKey Profile Page]

Posts: 44

Reply: 34



PostPosted: February 8, 2007 3:37 PM 

The important part of the .htacess is the following:

# don't serve mtview.php if the request is for a real file
# (allows the actual file to be served)
RewriteCond %{REQUEST_FILENAME} !-f
# anything else is handed to mtview.php for resolution
RewriteRule ^(.*)$ /mtview.php [L,QSA]

The "RewriteCond %{REQUEST_FILENAME} !-f" part says "check to see if the requested file actually exists as a static file, if so, serve the static file". If not, it will try to dynamically generate the page, via the last line above.

darkmoon [TypeKey Profile Page]

Posts: 23

Reply: 35



PostPosted: February 8, 2007 4:34 PM 

okay.... who knows why it was messed up. I moved it to a new server and now it works. hah... man. frustrating but I think I'm finally done with it.

Mark Carey [TypeKey Profile Page]

Posts: 44

Reply: 36



PostPosted: February 9, 2007 9:07 AM 

Glad you got it working, darkmon. Wink

James Bow [TypeKey Profile Page]

Posts: 11

Reply: 37



PostPosted: February 19, 2007 5:17 PM 

I've installed Spam Firewall and really appreciate it. It seems to have cut down my spam load dramatically. However, I'm having a problem with TypeKey identification. I have a blog which is posting to a different domain from the MT installation, and when I try to log into Typekey and post a comment, my name and e-mail address doesn't carry over. What's the best way to fix this problem?

Mark Carey [TypeKey Profile Page]

Posts: 150

Reply: 38



PostPosted: February 19, 2007 5:49 PM 

Hi James,

Glad the plugin is working for you.

The particular case you describe is currently a limitation. The PHP firewall script has access to the cookies on blog domain but not on the MT domain. So when the blog domain is different that the MT domain, there's no easy way to pass those TypeKey cookies across. One way to work around is to build the Spam Firewall template as you normall would, and then manually move (or copy) the built PHP file over to a location on the MT domain. Then, update the comment form on your Individual Archive template to point to the new location of of the firewall script. To be honest, I haven't tested this but I believe it should work. Please reply here and let me know either way.

James Bow [TypeKey Profile Page]

Posts: 11

Reply: 39



PostPosted: February 19, 2007 9:41 PM 

Your workaround does the trick, although I have to create separate files for each blog that sits on a different domain. This isn't a problem, but it might complicate things if this plugin is updated.

But on the whole I'm really pleased. I also particularly like BlogJanitor.

smoothmomma [TypeKey Profile Page]

Posts: 3

Reply: 40



PostPosted: February 21, 2007 1:59 AM 

I'm having a problem where many commenters are just getting a blank screen after they hit submit. The comments are going through, but the page isn't redirected. It doesn't happen every time and of course it doesn't happen when I'm trying to troubleshoot. error reporting is turned on in php.ini...

it seems that the page is just sometimes not redirecting... is there any simple way to leave a link on the page back to the post in case the re-direct doesn't work?

Also is there anything that can be done about the pause between submit and the the post showing up. The other problem I have is people hitting submit again after a few seconds because they think the blog is hung

Mark Carey [TypeKey Profile Page]

Posts: 150

Reply: 41



PostPosted: February 21, 2007 8:05 AM 

smoothmomma, you are not the only person to run into the blank page issue. This has been reported by one other person, and I noticed this once myself. Same symptoms, but no errors seem to get reported anywhere, making it difficult to find the source of the problem. If you see anything odd in your error logs, please let me know. Look for both the spamfw.php as well as the comment script, since both get called in the process. Obviously, the comment posting is working here, but for some reason the firewall is not able to capture and display the result, and I am not sure why. The fact that this happens only occasionally makes it even more difficult to pinpoint. Maybe there is a timeout issue? I will keep trying to figure it out...

Mark Carey [TypeKey Profile Page]

Posts: 150

Reply: 42



PostPosted: February 21, 2007 9:56 AM 

I think I answered my own question in my previous reply. Seems to be a timeout issue. Should be resolved in version 1.2, just released. Please reply to the 1.2 thread if you still get this problem.

Fabian

Posts: 1

Reply: 43



PostPosted: February 28, 2007 8:12 PM 

Is there any way to strip the slashes that PHP is putting in front of apostrophes (i.e. \'). Can your script be edit as such that the slashes are stripped from the comment or trackback body?

Mark Carey [TypeKey Profile Page]

Posts: 150

Reply: 44



PostPosted: March 1, 2007 4:59 PM 

Fabian, the latest version addresses this issue. See link for 1.2 above...

laatland

Posts: 1

Reply: 45



PostPosted: April 30, 2007 11:14 AM 

I've installed Spam Firewall

but Gives me an error of


(Smarty error: [in mt:16 line 18]: syntax error: unrecognized tag 'MTFirewallTrackbackData' (Smarty_Compiler.class.php, line 580



Subscribe to this discussion: Email | RSS

Join the conversation:









Remember personal info?





Very Happy Smile Sad Surprised
Shocked Confused Cool Laughing
Mad Razz Embarassed Crying or Very Sad
Evil or Very Mad Twisted Evil Rolling Eyes Wink
Powered by MTSmileys
Check to Subscribe to this Comment:
(email field must be filled in)



Subscribe Without Commenting
















Creative Commons License
This weblog is licensed under a Creative Commons License.