On Sunday, March 6th of 2005 I was messing around on my workstation retagging some Mp3’s so I could add them into my iPod playlists when I got a GMail notification of a new comment on my blog. I went to check it out and it was fairly basic comment “prepare to be slashdotted. I went to check-out Slashdot and didn’t see anything there, so I assumed it was likely just a spam comment. It wasn’t long after that when a link to an entry I’d wrote a couple of months previous appeared at the top of their front page.
I was actually quite shocked—being linked to from Slashdot is a very big deal in the geek community. For those not familiar with Slashdot, the best comparison I can make is if you were an amateur author and had your work published in Newsweek or something like that. Slashdot does not have nearly the readership of Newsweek, but it’s certainly one of the most trafficked international geek news sites that I know of offhand– I’m a regular reader myself.
Once I knew for sure I was getting Slashdotted, I actually decided to get-up from the computer and go do something else while I thought about how I could prepare for a load like that rather than freaking out and breaking things. My server at the time was a temporary server built out of spare parts (I host my site at home) which was over 6 years old, so I had no idea what I’d be able to do with it. After remembering articles several years ago about something to the effect of “A Pentium 90 can easily put out enough bandwidth to saturate a T1�?, I kept that in mind and started to think of ways to eliminate bottlenecks. I eventually came back, saw that my server was non-responsive via HTTP. I then blocked port 80 at the firewall, and went to work.
I was not a bit surprised the server was non-responsive; my site at the time was running on a Celeron 466 with 192 MB of RAM. I also of course used Apache HTTPD Server, as well as PHP5 and MySQL to power my Wordpress blog in which the entry was in that was being linked to.
My first course of action was to figure-out how to make the page static so that it wouldn’t have to hit MySQL each time someone requested that page. I assumed there was probably a Wordpress plug-in to do this with, so I googled for one and found one which Matt Mullenweg himself was using, Staticize Reloaded. (Note: I’ve since switched to an upgraded version of the same plugin.) This was a major help, but not quite enough given that I hadn’t really touched this particular FreeBSD install much as it was temporary until my new server came in.
My second major course of action was to change the settings on Apache to keep it from dying. It’s been quite some time since all of this happened so I don’t remember the exact settings I used, but basically I upped the number of min and max servers, set a server timeout of 15 seconds, and made sure new server processes were started every 500 requests or so to keep runaway processes from killing the server.
One last thing I did was kill all unnecessary processes such as SMTP (I knew my mail would queue on external SMTP servers trying to reach mine, and I get more spam than legitimate mail anyhow), made sure nothing in cron was intensive, etc. Had bandwidth also been a problem after these changes, I planned on moving my site to the default Wordpress template which did not have any graphics—just XHTML & CSS.
From there I decided to reboot the box just to clean it up and open port 80 back-up again to accept HTTP traffic again. Surprisingly FreeBSD never once crashed the whole time even after the initial Slashdotting before I started blocking traffic.. it was just very bogged down with requests (mostly MySQL was dying). I rebooted it after changing the configuration as a precautionary measure. This was a very stock install of FreeBSD– I didn’t even make a custom kernel for it or anything.
Once I put the box back in, it took a bit more Apache tweaking to get it perfect, but eventually my spare parts Celeron 466 was handling a Slashdotting at only 30% CPU utilization! Here is an example of the number of connections it was getting (which is crazy for me to see on a personal machine) as the Slashdot effect was winding down:
Active Internet connections Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 0 791 grimmace.http puddle189.drizzl.3055 ESTABLISHED tcp4 0 33580 grimmace.http c-67-166-235-247.4364 ESTABLISHED tcp4 0 0 grimmace.http 213.58.2.244.1227 ESTABLISHED tcp4 0 32120 grimmace.http puddle189.drizzl.3054 ESTABLISHED tcp4 0 26018 grimmace.http c-67-166-235-247.4363 ESTABLISHED tcp4 0 0 grimmace.http planetlab2.georg.55584 TIME_WAIT tcp4 0 0 grimmace.http c-67-170-48-141..61898 ESTABLISHED tcp4 0 0 grimmace.http tide505.microsof.39548 ESTABLISHED tcp4 0 0 grimmace.http tide532.microsof.54754 ESTABLISHED tcp4 0 0 grimmace.http unassigned.purec.1361 ESTABLISHED tcp4 0 0 grimmace.http c-67-170-48-141..61897 ESTABLISHED tcp4 0 0 grimmace.http tide508.microsof.30208 ESTABLISHED tcp4 0 0 grimmace.http tide70.microsoft.53537 ESTABLISHED tcp4 0 0 grimmace.http c-24-19-42-108.c.4436 ESTABLISHED tcp4 0 0 grimmace.http tide85.microsoft.40545 ESTABLISHED tcp4 0 0 grimmace.http S01060050baa51fa.33235 ESTABLISHED tcp4 0 0 grimmace.http c-24-19-42-108.c.4434 ESTABLISHED tcp4 0 0 grimmace.http S01060050baa51fa.33234 ESTABLISHED tcp4 0 0 grimmace.http tide85.microsoft.40500 ESTABLISHED tcp4 0 0 grimmace.http tide531.microsof.36377 FIN_WAIT_2 tcp4 0 0 grimmace.http tide517.microsof.12997 FIN_WAIT_2 tcp4 0 0 grimmace.http tide518.microsof.18692 FIN_WAIT_2 tcp4 0 0 grimmace.http tide510.microsof.52133 ESTABLISHED tcp4 0 0 grimmace.http c-67-182-8-198.c.60914 FIN_WAIT_2 tcp4 0 0 grimmace.http 68-113-20-232.wa.37925 ESTABLISHED tcp4 0 0 grimmace.http c-67-182-8-198.c.60913 FIN_WAIT_2 tcp4 0 0 grimmace.http 68-113-20-232.wa.37924 ESTABLISHED tcp4 0 0 grimmace.http msnbot.msn.com.37805 TIME_WAIT tcp4 0 12427 grimmace.http dial-148-240-184.3124 ESTABLISHED tcp4 0 0 grimmace.http pool-68-163-42-6.4368 TIME_WAIT tcp4 0 0 grimmace.http unassigned.purec.1359 ESTABLISHED tcp4 0 32327 grimmace.http dial-148-240-184.3122 ESTABLISHED tcp4 0 0 grimmace.http relax15.idiap.ch.40978 TIME_WAIT tcp4 0 0 grimmace.http relax15.idiap.ch.40977 TIME_WAIT tcp4 0 0 grimmace.http pool-68-163-42-6.4366 TIME_WAIT tcp4 0 0 grimmace.http relax15.idiap.ch.40975 TIME_WAIT tcp4 0 0 grimmace.http 24.125.148.80.1176 TIME_WAIT tcp4 0 0 grimmace.http 213.58.2.244.1208 TIME_WAIT tcp4 0 0 grimmace.http CPE0050da6ab5da-.3371 TIME_WAIT tcp4 0 0 grimmace.http 24.125.148.80.1175 TIME_WAIT tcp4 0 0 grimmace.http adsl-70-240-237-.1266 FIN_WAIT_2 tcp4 0 0 grimmace.http CPE0050da6ab5da-.3370 TIME_WAIT tcp4 0 0 grimmace.http bbox.ne.client2..47283 TIME_WAIT tcp4 0 0 grimmace.http msnbot.msn.com.36671 TIME_WAIT tcp4 0 0 grimmace.http host-148-244-150.46757 TIME_WAIT tcp4 0 0 grimmace.http hannibal.lr-s.tu.50584 TIME_WAIT tcp4 0 0 grimmace.http rdu25-17-055.nc..54974 TIME_WAIT tcp4 0 0 grimmace.http 163.80-202-31.ne.57984 TIME_WAIT tcp4 0 0 grimmace.http bbox.ne.client2..47282 TIME_WAIT tcp4 0 0 grimmace.http rdu25-17-055.nc..54973 TIME_WAIT tcp4 0 0 grimmace.http hannibal.lr-s.tu.50578 TIME_WAIT tcp4 0 0 grimmace.http c-67-174-111-123.3682 TIME_WAIT tcp4 0 0 grimmace.http c-67-174-111-123.3681 TIME_WAIT tcp4 0 0 grimmace.http c-67-174-111-123.3680 TIME_WAIT tcp4 0 0 grimmace.http relax15.idiap.ch.40952 TIME_WAIT tcp4 0 0 grimmace.http relax15.idiap.ch.40953 TIME_WAIT tcp4 0 0 grimmace.http c-67-174-111-123.3679 TIME_WAIT tcp4 0 33304 grimmace.http relax15.idiap.ch.40943 LAST_ACK tcp4 0 0 grimmace.http 163.80-202-31.ne.57983 TIME_WAIT tcp4 0 33304 grimmace.http relax15.idiap.ch.40931 CLOSE_WAIT tcp4 0 0 grimmace.http ool-4570bffb.dyn.54562 TIME_WAIT tcp4 0 0 grimmace.http 208-59-174-198.c.1771 TIME_WAIT tcp4 0 0 grimmace.http xdsl-81-173-248-.3833 TIME_WAIT tcp4 0 0 grimmace.http 24-117-147-243.c.60104 TIME_WAIT tcp4 0 0 grimmace.http pcp658486pcs.cap.29623 TIME_WAIT tcp4 0 0 grimmace.http pcp658486pcs.cap.29622 TIME_WAIT tcp4 0 0 grimmace.http ool-4570bffb.dyn.54558 TIME_WAIT tcp4 0 0 grimmace.http 208-59-174-198.c.1770 TIME_WAIT tcp4 0 0 grimmace.http 24-117-147-243.c.60103 TIME_WAIT tcp4 0 0 grimmace.http xdsl-81-173-248-.3825 TIME_WAIT tcp4 0 0 grimmace.http pool-68-238-49-1.50253 TIME_WAIT tcp4 0 0 grimmace.http pool-68-238-49-1.50252 TIME_WAIT tcp4 0 0 grimmace.http pool-68-238-49-1.50251 TIME_WAIT tcp4 0 0 grimmace.http pool-68-238-49-1.50250 TIME_WAIT tcp4 0 24300 grimmace.http ip160-108.om.apa.14426 CLOSING tcp4 0 30640 grimmace.http ip160-108.om.apa.14403 FIN_WAIT_1 tcp4 0 33304 grimmace.http 209.237.230.104.21469 ESTABLISHED tcp4 0 33580 grimmace.http S010600067b078af.1423 ESTABLISHED tcp4 0 29326 grimmace.http bidwell.textdriv.55107 FIN_WAIT_1 tcp4 0 0 grimmace.http c68.115.108.235..3517 FIN_WAIT_1 tcp4 0 0 grimmace.http c68.115.108.235..3516 FIN_WAIT_1 tcp4 0 0 grimmace.http cust1392.vic01.d.20836 FIN_WAIT_1 tcp4 0 0 grimmace.http node-423a2a62.df.62359 FIN_WAIT_1 tcp4 0 0 grimmace.http cust1392.vic01.d.20835 FIN_WAIT_1 tcp4 0 0 grimmace.http node-423a2a62.df.62358 FIN_WAIT_1 tcp4 0 0 grimmace.http 66.89.66.69.ptr..57168 FIN_WAIT_1 tcp4 0 0 grimmace.http 81.104.95.80.ip..13863 FIN_WAIT_2 tcp4 0 0 grimmace.http 66.89.66.69.ptr..57166 FIN_WAIT_1 tcp4 0 0 grimmace.http 81.104.95.80.ip..13854 FIN_WAIT_1 tcp4 0 0 grimmace.http lata228-05-c136..29431 FIN_WAIT_2 tcp4 0 0 grimmace.http lata228-05-c136..29430 FIN_WAIT_2 tcp4 0 24300 grimmace.http ip160-108.om.apa.14400 CLOSING tcp4 0 0 grimmace.http 207-171-180-101..44370 FIN_WAIT_1 tcp4 0 0 grimmace.http 207-171-180-101..44302 FIN_WAIT_1 tcp4 0 0 grimmace.http lata228-05-c136..29429 FIN_WAIT_2 tcp4 0 0 grimmace.http lata228-05-c136..29428 FIN_WAIT_2 tcp4 0 0 grimmace.http ausisapc102-dmz..22690 FIN_WAIT_2 tcp4 0 0 grimmace.http h000625d8511c.ne.4303 FIN_WAIT_2 tcp4 0 0 grimmace.http 209.237.230.104.28587 FIN_WAIT_2 tcp4 0 0 grimmace.http 66.193.93.3.47306 FIN_WAIT_2 tcp4 0 0 grimmace.http 66.193.93.3.47302 FIN_WAIT_2 tcp4 0 0 grimmace.http adsl-6-96-227.ms.46767 FIN_WAIT_2 tcp4 0 33580 grimmace.http c-24-6-68-25.cli.1107 ESTABLISHED
Unfortunately I’d estimate I was offline for around 2 hours while I fixed the machine, etc, so there is no telling how much traffic I would of seen while my site was the top link on Slashdot, but here are a few graphical examples of what type of traffic I saw once I got the server working (I think it would have been significantly more had it not been on a Sunday afternoon.):
Usually I do around 2-2.5 GB of web traffic per month, but in one day I did over 4 GB! Granted this isn’t impressive compared to a real production environment, but not bad for an old Celeron on a DSL line. I’ve also had one of my entries end up on del.ici.ous/popular which upped my traffic to about 4 GB+ of bandwidth in a month, but it was much more steady rather than one big rush of requests all at once.
I had other stuff to do on Sunday other than sit around on the computer, but I did spend a considerable amount of time responding to comments both on Slashdot as well as my own site. Surprisingly I didn’t get any comments which I was forced to delete. I did however have some other strange experiences which I recently wrote about.
I’d say my main highlights to pass on from this experience are:
- Content based sites are easy to scale (having previous work experience with this on a platform that ran about 200 mall websites using a single software platform on just a handful of servers, it was just another data point for said observation)
- FreeBSD is amazing, and should get more hype though it’s unfortunately shadowed by Linux (but I already knew this as well)
- Wordpress is a great, extensible blogging platform, and can stand-up to load without breaking a sweat
- Though it’s not my language of choice to work in when doing web scripting, the combination of PHP and MySQL is amazingly fast while requiring the least amount of money and resources of other web scripting languages
- You may not always be able to do the “academically sound” process when faced with a conflict, but you can certainly hack your way through an unexpected situation to get you by in a pinch (though hacking is toxic in the long term)
All in all it was a very interesting experience, and hopefully it will happen again sometime. The server which was on order at the time is currently serving the site, and should be able to handle much more load without me needing to touch it if it were to happen again, but we’ll see…
40 Responses
Stay in touch with the conversation, subscribe to the RSS feed for comments on this post.
Brandon…
Great to meet you at CFUnited… hope to see you at Max.
I only hope I get to be slashdotted one day. Perhaps eventually I shall be worthy.
Laterz!
Brandon, also good to meet you at CFUNITED. Thanks for posting this stuff, interesting…
Good to meet both of you as well. I actually seriously doubt I’ll be at Max, but who knows I guess.
After talking to Pete at CFUNITED about this and other various things I remembered I never got around to blogging about what happened and so forth, so this was one item on my post-conference to-do list.
I’ll add one to the list of highlights: Pay attention when you are warned of slashdotting!
Great read.
Well it’s certainly not a Slashdotting, but you’ve been dugg… or is it digged? Take yer pick
Shame you didnt survive the Digg effect…
oh well, a Slashdotting (?) is still pretty impressive
Are you using a Dynamic or Static IP system for your DSL line?
Is your DSL line 1.5Mbps up, down or both?
It is pretty ironic that this page didn’t survive the Digg effect.
Great story, though!
HMTKSteve:
I have a single static IP, and my DSL line is only 1.5 Mbps down.
Good post, and I have to second your wp-cache-2 recommentation. It’s a great plugin. My lowly VPS survives lots of spikes in traffic because of it!
Tom & junger:
I was seeding Torrents and was sleeping when this hit the front page at Digg. I killed the torrents and it seems to be okay now (being that I’m hitting it from work and all).
Yeah, FreeBSD is pretty awesome and blows Linux away even now. http://ftp.freebsd.org use to move 700gb a day on one server back in the mid-90’s. I can’t imagine what it moves now.
prepare to be Dugg!
So, you published something interesting and it appeared on Slashdot.
Congratulations, you can be proud.
But really, how interesting is saturating a 1.5Mbit downstream (probably significantly less upstream) connection in 2006?
And yes, FreeBSD rocks
Traffic is going to be a lot more upstream because my server is sending content to peers, not the other way around. I believe my upsteam speed is 512Kbps at best.
I never said saturating anything downstream was interesting, not to mention it’s not downstream that gets saturated, it’s the upstream.
I just thought I’d write a post about how I was able to handle a lot of load on a crappy server running Wordpress.
w3rd, slashd0t.0rg
Brandon, iThankyou for this informative post which actually was what made me want to try WordPress with WP-Cache. I’m not a computer scientist, but I find the whole idea of not caching dynamic content utterly bizarre, and I would never have stuck with a CMS that doesn’t have any caching abilities to host my site on my home server. It’s a fairly fast machine, an Athlon XP 1900+ running Linux, but trying out WordPress without caching made me feel unsafe every time I generated some reloads in my browser while watching top’s real-time output…
However, to whom it may concern, I run into a slight problem while trying to use WP-Cache 2.0 on WordPress 2, the feared blank/empty page error. I googled a bit and found to my problem, which turned out to be PHP5-related.
A very interesting article, I have followed a couple of your suggestions and are currently benchmarking the improvments. Not likely that I will ever be dugg but it’s always nice to know that ones server is ticking over the best it can.
Continuing the Discussion