Black Raven Dragoons

[Zancarius]

Some of you may already be aware that the goon sites came under attack briefly. This post elaborates on that.

Earlier today, the goon forums became unresponsive, and as I understand from Josh, the TS server was providing somewhat limited connectivity earlier in the day with momentary service interruption. Initially, I ssh'd into the box (with the first connection timing out and refusing my password), but discovered nothing terribly out of the ordinary. The only exception to this was the unusual high load averages, suggestive of someone beating the daylights out of the server.

My first thought was to presume it was the result of an OpenSSL exploit of which there were many for the specific version the server had been running at that time. Several of these exploits provide attackers with an avenue for running a denial of service attack against services that link to affected libraries. Since SSH was acting rather unusual, this was my first assumption, and I updated OpenSSL and OpenSSH both. Apache followed suit shortly thereafter since I had some vague recollection that it held dependencies on the OpenSSL libraries, and would therefore no longer run if it weren't also updated (the old version of OpenSSL would be unavailable).

I then assumed the problem was likely resolved, and following a restart of the sshd service, I left it alone for a few hours.

Sometime later, I went to read the forums only to discover they were unavailable. That's when it clicked that something weird was going on. At this point, it was a suspected DoS attack on the web server, and my immediate reaction was to shut down Apache and start digging. I discovered that one of the Wordpress installs on this box were under repeated attack (presumably via an attempt to exploit known vectors in the Wordpress xmlrpc script), and since it was getting late--I needed to cook burgers--I left it offline. By that point, I'd contacted Josh and informed him of the presumed problem, possible solutions, etc., and left to eat dinner. I expected Apache wouldn't restart without some effort, but what I didn't expect was the substantial changes made to the out-of-the-box configuration, syntax changes, etc., and now I was in a dilemma: Fiddle with Apache for a few hours or just bite the bullet and migrate to PHP-FPM + nginx. All of my personal servers run nginx, rather than Apache, and we'd discussed migrating at some point in the future.

This evening, Josh returned my call and we discussed the situation. Our conclusion was to migrate to nginx, partially for performance reasons, and partially because the configuration is incredibly simple. It also helps that I've not run an Apache-based web service in a number of years and all of my tools are now focused on nginx, so for the sake of making things a bit easier and reducing overall labor, Josh preferred the idea of running nginx on this system. This also means that we can do a few neat things at some point in the future. (It also means Josh doesn't have to fuss with Apache-specific nonsense any longer--definitely a bonus when you consider how awful mod_rewrite is.)

So, long story short, the goon sites are slowly being migrated to a new platform, and though you shouldn't notice any changes, nginx is more performant and less susceptible to some of the unfortunate issues that still plague Apache. Of course, all of these sites are still PHP powered (via the FPM daemon), so whatever benefits gained by the move to nginx are lost in the PHP interpreter. On the plus side, FPM is somewhat faster than mod_php and has a better set of controls over its performance characteristics, so you might still notice a slight reduction in overall latency. I don't expect it'll be much, though, because once you reduce response latency around or below the average jitter that affects most networks, it becomes impossible to judge.

As far as the attack goes, it appears that it was an automated script repeatedly attacking a Wordpress installation in such a manner as to force Apache to continue fork()ing new processes, bringing the machine to a crawl. I suspect the attacker's script was broken in some manner because of the nature of the attack. I did not collect a tcpdump of the incoming traffic at the time (the server was not yet updated), but I'll be monitoring the attempts, collecting further data as needed. Note that the attackers were not successful in exploiting Wordpress since it appears the bug they're using may have been in a newer version (?), but we have taken some further precautions to mitigate this in the future. All of the sites are now isolated.

At the present time, I'm still migrating some of the remaining sites. There are a few problems since the configurations were ported from my own, and phpBB requires some special needs, so you may notice that certain images (avatars) aren't loading.

I'll keep you all posted. For now, good night!

[Grimblast]

You certainly can feel the snappiness in how fast the forums load up. It's crazy.

[Zancarius]

Definitely!

I don't think I ever did a direct comparison between Apache and nginx myself. Or maybe I did and it was long enough ago that I've since forgotten. Granted, it's slightly more complicated (two separate services to deal with instance of just one), but I'm no longer of the frame of mind that loading modules into a web service to support a scripting language is a good idea. Sometimes having a tool that does one thing--and does it well--is much better than having a swiss army knife that does everything poorly.

[Zancarius]

Pat noticed some links were returning an HTTP 404 which was due to an oversight on my behalf with the nginx rewrite rules. These should be fixed.

Let me know if you run into anything else that doesn't appear to be working correctly. Most of the nginx rules were ported over from the mod_rewrite equivalents, but there's undoubtedly a few that I may have missed.