So that McCain/Cotton image? Yeah … linked to all over the frakkin ‘net (hotlinked sometimes, too; bastards).

This was perhaps a bad time to try moving the site from mod_php to mod_fcgid, or, depending on how you look at it, the perfect time. More on that later.

So the first 8K hits came from StumbleUpon, which I forgot existed. Then when that was done the forums picked it up and started hotlinking it all over the place (and I’ve resisted the urge to drop something evil in its place only because I have the bandwidth to take it — go Quantact!).

That was a nice test of my new arrangement for PHP, to say the least. The server started to choke early on and then I discovered that Tracker.module was logging an error to the database on every page hit. Turned off the module and replaced it with the Views equivalent and it came right back up (no more writing to the DB).

Then it appears Reddit picked it up (as a hotlink, of course). Funny thing about getting high up on Reddit: the top 30 or so links are at the bottom of frickin-Wired-dot-com. Yep, Wired. So I’m getting that as well.

I looked at the traffic stats tool on Quantact and saw that I went from near-nothing to 600Kb/s, then 1.1Mb/s, then 2.1Mb/s and it was going up even more. What the hell? I looked at the file … I’d saved it as a PNG. A 500KB PNG.

I made a JPEG version and added the proper redirect and updated the blog entry. Now everyone gets the 100K file, my usage has fallen to 300-400Kb/s, and the site is very responsive now (compared to when it was at 2.1Mb/s at least).

If it sounds like I’m complaining, I’m not. I’m just a little shocked. This round is much worse than when Slashfood picked up the Krispy Kreme Luthers a few weeks ago and I’m loving the lesson in how to keep a server going in 384MB of RAM.

We’re at 30K hits to the actual page and an additional 25K for the images.

This is fun.

The Setup

As I mentioned earlier, I changed how the site worked last night. I had a thinking-while-in-the-shower moment where I realized that having mod_php loaded meant that every single hit was forking Apache just to protect PHP from itself. Very inefficient. So I pondered using LigHTTPd as a reverse proxy for a while. That would work by forwarding any non-file requests to Apache for processing. I was looking into that until I realized I’d have to have every site I host in two config files and make sure they were pointing at the right places and somehow determine which files existed and which didn’t in Lighty, which isn’t easy (best solution out there? A Lua script. For every hit).

So, ditching that, I realized that Apache isn’t too heavyweight when mod_php isn’t sucking up 50MB of RAM per process, so I looked into FastCGI again and managed to get it working after some false starts. Now I’m on Apache 2 with MPM-worker. This way, anything that needs Ruby or PHP is sent to the fork with the FCGI children and everything else is handled by the lightweight forks with their dozen theads each. PHP processing is slow when being hammered, but not slow like it was (where it wouldn’t load). Just slow like “wait a couple of seconds for a FCGI instance to free up”. Much better. As a side effect, file serving has sped up dramatically. This is overall a much better solution for a page powered by a large PHP application (Drupal) with some small files on the page (CSS + images).

To do the setup (in Ubuntu), I first disabled and removed mod_php5 and then installed the FCGI version of PHP (php-cgi). Then I fetched the mod_fcgid and apache2-mpm-worker packages (which will remove the apache2-mpm-prefork package, but not the config files for it). This got everything ready, but I had to tell it how to handle PHP files, so I created /etc/apache2/conf.d/fcgid.conf and put in it:

Then I just made sure that every directory I served PHP out of had +ExecCGI on it and I was home-free.

Update: Almost home-free. One last thing I did that made a huge difference: I changed the thread stack size. Linux allocates 8MB per thread for the stack. That’s excessive in this case. So I told Apache to make it 1MB instead. My memory usage dropped considerably and I’ve had no ill effects from it.