Surviving A Slashdotting With a Celeron 466: My Slashdot Experience
On Sunday, March 6th of 2005 I was messing around on my workstation retagging some Mp3’s so I could add them into my iPod playlists when I got a GMail notification of a new comment on my blog. I went to check it out and it was fairly basic comment “prepare to be slashdotted. I went to check-out Slashdot and didn’t see anything there, so I assumed it was likely just a spam comment. It wasn’t long after that when a link to an entry I’d wrote a couple of months previous appeared at the top of their front page.
I was actually quite shocked—being linked to from Slashdot is a very big deal in the geek community. For those not familiar with Slashdot, the best comparison I can make is if you were an amateur author and had your work published in Newsweek or something like that. Slashdot does not have nearly the readership of Newsweek, but it’s certainly one of the most trafficked international geek news sites that I know of offhand– I’m a regular reader myself.
Once I knew for sure I was getting Slashdotted, I actually decided to get-up from the computer and go do something else while I thought about how I could prepare for a load like that rather than freaking out and breaking things. My server at the time was a temporary server built out of spare parts (I host my site at home) which was over 6 years old, so I had no idea what I’d be able to do with it. After remembering articles several years ago about something to the effect of “A Pentium 90 can easily put out enough bandwidth to saturate a T1�?, I kept that in mind and started to think of ways to eliminate bottlenecks. I eventually came back, saw that my server was non-responsive via HTTP. I then blocked port 80 at the firewall, and went to work.
I was not a bit surprised the server was non-responsive; my site at the time was running on a Celeron 466 with 192 MB of RAM. I also of course used Apache HTTPD Server, as well as PHP5 and MySQL to power my Wordpress blog in which the entry was in that was being linked to.
My first course of action was to figure-out how to make the page static so that it wouldn’t have to hit MySQL each time someone requested that page. I assumed there was probably a Wordpress plug-in to do this with, so I googled for one and found one which Matt Mullenweg himself was using, Staticize Reloaded. (Note: I’ve since switched to an upgraded version of the same plugin.) This was a major help, but not quite enough given that I hadn’t really touched this particular FreeBSD install much as it was temporary until my new server came in.
My second major course of action was to change the settings on Apache to keep it from dying. It’s been quite some time since all of this happened so I don’t remember the exact settings I used, but basically I upped the number of min and max servers, set a server timeout of 15 seconds, and made sure new server processes were started every 500 requests or so to keep runaway processes from killing the server.
One last thing I did was kill all unnecessary processes such as SMTP (I knew my mail would queue on external SMTP servers trying to reach mine, and I get more spam than legitimate mail anyhow), made sure nothing in cron was intensive, etc. Had bandwidth also been a problem after these changes, I planned on moving my site to the default Wordpress template which did not have any graphics—just XHTML & CSS.
From there I decided to reboot the box just to clean it up and open port 80 back-up again to accept HTTP traffic again. Surprisingly FreeBSD never once crashed the whole time even after the initial Slashdotting before I started blocking traffic.. it was just very bogged down with requests (mostly MySQL was dying). I rebooted it after changing the configuration as a precautionary measure. This was a very stock install of FreeBSD– I didn’t even make a custom kernel for it or anything.
Once I put the box back in, it took a bit more Apache tweaking to get it perfect, but eventually my spare parts Celeron 466 was handling a Slashdotting at only 30% CPU utilization! Here is an example of the number of connections it was getting (which is crazy for me to see on a personal machine) as the Slashdot effect was winding down:
Active Internet connections Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 0 791 grimmace.http puddle189.drizzl.3055 ESTABLISHED tcp4 0 33580 grimmace.http c-67-166-235-247.4364 ESTABLISHED tcp4 0 0 grimmace.http 213.58.2.244.1227 ESTABLISHED tcp4 0 32120 grimmace.http puddle189.drizzl.3054 ESTABLISHED tcp4 0 26018 grimmace.http c-67-166-235-247.4363 ESTABLISHED tcp4 0 0 grimmace.http planetlab2.georg.55584 TIME_WAIT tcp4 0 0 grimmace.http c-67-170-48-141..61898 ESTABLISHED tcp4 0 0 grimmace.http tide505.microsof.39548 ESTABLISHED tcp4 0 0 grimmace.http tide532.microsof.54754 ESTABLISHED tcp4 0 0 grimmace.http unassigned.purec.1361 ESTABLISHED tcp4 0 0 grimmace.http c-67-170-48-141..61897 ESTABLISHED tcp4 0 0 grimmace.http tide508.microsof.30208 ESTABLISHED tcp4 0 0 grimmace.http tide70.microsoft.53537 ESTABLISHED tcp4 0 0 grimmace.http c-24-19-42-108.c.4436 ESTABLISHED tcp4 0 0 grimmace.http tide85.microsoft.40545 ESTABLISHED tcp4 0 0 grimmace.http S01060050baa51fa.33235 ESTABLISHED tcp4 0 0 grimmace.http c-24-19-42-108.c.4434 ESTABLISHED tcp4 0 0 grimmace.http S01060050baa51fa.33234 ESTABLISHED tcp4 0 0 grimmace.http tide85.microsoft.40500 ESTABLISHED tcp4 0 0 grimmace.http tide531.microsof.36377 FIN_WAIT_2 tcp4 0 0 grimmace.http tide517.microsof.12997 FIN_WAIT_2 tcp4 0 0 grimmace.http tide518.microsof.18692 FIN_WAIT_2 tcp4 0 0 grimmace.http tide510.microsof.52133 ESTABLISHED tcp4 0 0 grimmace.http c-67-182-8-198.c.60914 FIN_WAIT_2 tcp4 0 0 grimmace.http 68-113-20-232.wa.37925 ESTABLISHED tcp4 0 0 grimmace.http c-67-182-8-198.c.60913 FIN_WAIT_2 tcp4 0 0 grimmace.http 68-113-20-232.wa.37924 ESTABLISHED tcp4 0 0 grimmace.http msnbot.msn.com.37805 TIME_WAIT tcp4 0 12427 grimmace.http dial-148-240-184.3124 ESTABLISHED tcp4 0 0 grimmace.http pool-68-163-42-6.4368 TIME_WAIT tcp4 0 0 grimmace.http unassigned.purec.1359 ESTABLISHED tcp4 0 32327 grimmace.http dial-148-240-184.3122 ESTABLISHED tcp4 0 0 grimmace.http relax15.idiap.ch.40978 TIME_WAIT tcp4 0 0 grimmace.http relax15.idiap.ch.40977 TIME_WAIT tcp4 0 0 grimmace.http pool-68-163-42-6.4366 TIME_WAIT tcp4 0 0 grimmace.http relax15.idiap.ch.40975 TIME_WAIT tcp4 0 0 grimmace.http 24.125.148.80.1176 TIME_WAIT tcp4 0 0 grimmace.http 213.58.2.244.1208 TIME_WAIT tcp4 0 0 grimmace.http CPE0050da6ab5da-.3371 TIME_WAIT tcp4 0 0 grimmace.http 24.125.148.80.1175 TIME_WAIT tcp4 0 0 grimmace.http adsl-70-240-237-.1266 FIN_WAIT_2 tcp4 0 0 grimmace.http CPE0050da6ab5da-.3370 TIME_WAIT tcp4 0 0 grimmace.http bbox.ne.client2..47283 TIME_WAIT tcp4 0 0 grimmace.http msnbot.msn.com.36671 TIME_WAIT tcp4 0 0 grimmace.http host-148-244-150.46757 TIME_WAIT tcp4 0 0 grimmace.http hannibal.lr-s.tu.50584 TIME_WAIT tcp4 0 0 grimmace.http rdu25-17-055.nc..54974 TIME_WAIT tcp4 0 0 grimmace.http 163.80-202-31.ne.57984 TIME_WAIT tcp4 0 0 grimmace.http bbox.ne.client2..47282 TIME_WAIT tcp4 0 0 grimmace.http rdu25-17-055.nc..54973 TIME_WAIT tcp4 0 0 grimmace.http hannibal.lr-s.tu.50578 TIME_WAIT tcp4 0 0 grimmace.http c-67-174-111-123.3682 TIME_WAIT tcp4 0 0 grimmace.http c-67-174-111-123.3681 TIME_WAIT tcp4 0 0 grimmace.http c-67-174-111-123.3680 TIME_WAIT tcp4 0 0 grimmace.http relax15.idiap.ch.40952 TIME_WAIT tcp4 0 0 grimmace.http relax15.idiap.ch.40953 TIME_WAIT tcp4 0 0 grimmace.http c-67-174-111-123.3679 TIME_WAIT tcp4 0 33304 grimmace.http relax15.idiap.ch.40943 LAST_ACK tcp4 0 0 grimmace.http 163.80-202-31.ne.57983 TIME_WAIT tcp4 0 33304 grimmace.http relax15.idiap.ch.40931 CLOSE_WAIT tcp4 0 0 grimmace.http ool-4570bffb.dyn.54562 TIME_WAIT tcp4 0 0 grimmace.http 208-59-174-198.c.1771 TIME_WAIT tcp4 0 0 grimmace.http xdsl-81-173-248-.3833 TIME_WAIT tcp4 0 0 grimmace.http 24-117-147-243.c.60104 TIME_WAIT tcp4 0 0 grimmace.http pcp658486pcs.cap.29623 TIME_WAIT tcp4 0 0 grimmace.http pcp658486pcs.cap.29622 TIME_WAIT tcp4 0 0 grimmace.http ool-4570bffb.dyn.54558 TIME_WAIT tcp4 0 0 grimmace.http 208-59-174-198.c.1770 TIME_WAIT tcp4 0 0 grimmace.http 24-117-147-243.c.60103 TIME_WAIT tcp4 0 0 grimmace.http xdsl-81-173-248-.3825 TIME_WAIT tcp4 0 0 grimmace.http pool-68-238-49-1.50253 TIME_WAIT tcp4 0 0 grimmace.http pool-68-238-49-1.50252 TIME_WAIT tcp4 0 0 grimmace.http pool-68-238-49-1.50251 TIME_WAIT tcp4 0 0 grimmace.http pool-68-238-49-1.50250 TIME_WAIT tcp4 0 24300 grimmace.http ip160-108.om.apa.14426 CLOSING tcp4 0 30640 grimmace.http ip160-108.om.apa.14403 FIN_WAIT_1 tcp4 0 33304 grimmace.http 209.237.230.104.21469 ESTABLISHED tcp4 0 33580 grimmace.http S010600067b078af.1423 ESTABLISHED tcp4 0 29326 grimmace.http bidwell.textdriv.55107 FIN_WAIT_1 tcp4 0 0 grimmace.http c68.115.108.235..3517 FIN_WAIT_1 tcp4 0 0 grimmace.http c68.115.108.235..3516 FIN_WAIT_1 tcp4 0 0 grimmace.http cust1392.vic01.d.20836 FIN_WAIT_1 tcp4 0 0 grimmace.http node-423a2a62.df.62359 FIN_WAIT_1 tcp4 0 0 grimmace.http cust1392.vic01.d.20835 FIN_WAIT_1 tcp4 0 0 grimmace.http node-423a2a62.df.62358 FIN_WAIT_1 tcp4 0 0 grimmace.http 66.89.66.69.ptr..57168 FIN_WAIT_1 tcp4 0 0 grimmace.http 81.104.95.80.ip..13863 FIN_WAIT_2 tcp4 0 0 grimmace.http 66.89.66.69.ptr..57166 FIN_WAIT_1 tcp4 0 0 grimmace.http 81.104.95.80.ip..13854 FIN_WAIT_1 tcp4 0 0 grimmace.http lata228-05-c136..29431 FIN_WAIT_2 tcp4 0 0 grimmace.http lata228-05-c136..29430 FIN_WAIT_2 tcp4 0 24300 grimmace.http ip160-108.om.apa.14400 CLOSING tcp4 0 0 grimmace.http 207-171-180-101..44370 FIN_WAIT_1 tcp4 0 0 grimmace.http 207-171-180-101..44302 FIN_WAIT_1 tcp4 0 0 grimmace.http lata228-05-c136..29429 FIN_WAIT_2 tcp4 0 0 grimmace.http lata228-05-c136..29428 FIN_WAIT_2 tcp4 0 0 grimmace.http ausisapc102-dmz..22690 FIN_WAIT_2 tcp4 0 0 grimmace.http h000625d8511c.ne.4303 FIN_WAIT_2 tcp4 0 0 grimmace.http 209.237.230.104.28587 FIN_WAIT_2 tcp4 0 0 grimmace.http 66.193.93.3.47306 FIN_WAIT_2 tcp4 0 0 grimmace.http 66.193.93.3.47302 FIN_WAIT_2 tcp4 0 0 grimmace.http adsl-6-96-227.ms.46767 FIN_WAIT_2 tcp4 0 33580 grimmace.http c-24-6-68-25.cli.1107 ESTABLISHED
Unfortunately I’d estimate I was offline for around 2 hours while I fixed the machine, etc, so there is no telling how much traffic I would of seen while my site was the top link on Slashdot, but here are a few graphical examples of what type of traffic I saw once I got the server working (I think it would have been significantly more had it not been on a Sunday afternoon.):
Usually I do around 2-2.5 GB of web traffic per month, but in one day I did over 4 GB! Granted this isn’t impressive compared to a real production environment, but not bad for an old Celeron on a DSL line. I’ve also had one of my entries end up on del.ici.ous/popular which upped my traffic to about 4 GB+ of bandwidth in a month, but it was much more steady rather than one big rush of requests all at once.
I had other stuff to do on Sunday other than sit around on the computer, but I did spend a considerable amount of time responding to comments both on Slashdot as well as my own site. Surprisingly I didn’t get any comments which I was forced to delete. I did however have some other strange experiences which I recently wrote about.
I’d say my main highlights to pass on from this experience are:
- Content based sites are easy to scale (having previous work experience with this on a platform that ran about 200 mall websites using a single software platform on just a handful of servers, it was just another data point for said observation)
- FreeBSD is amazing, and should get more hype though it’s unfortunately shadowed by Linux (but I already knew this as well)
- Wordpress is a great, extensible blogging platform, and can stand-up to load without breaking a sweat
- Though it’s not my language of choice to work in when doing web scripting, the combination of PHP and MySQL is amazingly fast while requiring the least amount of money and resources of other web scripting languages
- You may not always be able to do the “academically sound” process when faced with a conflict, but you can certainly hack your way through an unexpected situation to get you by in a pinch (though hacking is toxic in the long term)
All in all it was a very interesting experience, and hopefully it will happen again sometime. The server which was on order at the time is currently serving the site, and should be able to handle much more load without me needing to touch it if it were to happen again, but we’ll see…
Tags: apache, caching, celeron, load, performance, slashdot, slashdotting, survive, survive-slashdotting, tuning
July 7th, 2005 at 1:20 am
Brandon…
Great to meet you at CFUnited… hope to see you at Max.
I only hope I get to be slashdotted one day. Perhaps eventually I shall be worthy.
Laterz!
July 7th, 2005 at 7:59 am
Brandon, also good to meet you at CFUNITED. Thanks for posting this stuff, interesting…
July 7th, 2005 at 9:22 am
Good to meet both of you as well. I actually seriously doubt I’ll be at Max, but who knows I guess.
After talking to Pete at CFUNITED about this and other various things I remembered I never got around to blogging about what happened and so forth, so this was one item on my post-conference to-do list.
July 7th, 2005 at 5:36 pm
Traffic-proofing servers
Traffic-proofing servers: Brandon Harper writes of what he did when finding that material on his little Celeron 466 server was suddenly linked to a hot topic on Slashdot. First step was to turn dynamic content (database query) into a static…
August 2nd, 2005 at 8:25 am
[...] lashdotting With a Celeron 466 and Wordpress Categories - LinkyLoo — Mark Surviving A Slashdotting With a Celeron 466 [...]
August 2nd, 2005 at 1:05 pm
[...] e faire slashdotter un jour ! Toi non plus tu n’y crois pas ? Sâches que ça arrive à des mecs bien de se faire slas [...]
August 2nd, 2005 at 2:15 pm
I’ll add one to the list of highlights: Pay attention when you are warned of slashdotting!
Great read.
August 2nd, 2005 at 3:29 pm
Well it’s certainly not a Slashdotting, but you’ve been dugg… or is it digged? Take yer pick
August 2nd, 2005 at 5:24 pm
[...] tutorial para los que tienen su blog hosteado en su casa. Corriendo WP en un Celeron 466, Devnulled sobrevivió [...]
August 3rd, 2005 at 10:46 am
[...] r Post Wedding Crashers: My Curt Review Django: The Python Alternative to Ruby on Rails [...]
August 3rd, 2005 at 5:19 pm
[...] nce many a site has died while trying… Thankfully, there’s finally someone who let’s us know exactly how to do it [...]
August 4th, 2005 at 2:11 pm
[...] running off a DSL line on old hardware? Brandon Harper [...]
August 5th, 2005 at 5:56 am
[...] ut
Surviving A Slashdotting With a Celeron 466
Surviving A Slashdotting With [...]
August 6th, 2005 at 9:34 pm
[...] deal with exactly these problems back in March of this year. He details his experience of Surviving A Slashdotting With a Celeron 466 [...]
October 6th, 2006 at 3:49 am
Shame you didnt survive the Digg effect…
oh well, a Slashdotting (?) is still pretty impressive
October 6th, 2006 at 4:30 am
Are you using a Dynamic or Static IP system for your DSL line?
Is your DSL line 1.5Mbps up, down or both?
October 6th, 2006 at 5:54 am
[...] A cool article that just shows that you dont need to have a ub3r server to survive the digg/slashdot effect, this guy survived it on T1 with a 486!read more | digg story [...]
October 6th, 2006 at 7:09 am
It is pretty ironic that this page didn’t survive the Digg effect.
Great story, though!
October 6th, 2006 at 7:19 am
HMTKSteve:
I have a single static IP, and my DSL line is only 1.5 Mbps down.
October 6th, 2006 at 7:53 am
[...] My Slashdotting entry is being dugg/digged again. More to follow, but so far today I’ve served up 8K users, and have a bunch of active connections: [...]
October 6th, 2006 at 8:08 am
[...] Another account of how one can counter Slashdotting/The Digg Effect http://devnulled.com/content/2005/07/surviving-a-slashdotting-with-a-celeron-466-my-slashdot-experience/ [...]
October 6th, 2006 at 8:52 am
Good post, and I have to second your wp-cache-2 recommentation. It’s a great plugin. My lowly VPS survives lots of spikes in traffic because of it!
October 6th, 2006 at 8:56 am
Tom & junger:
I was seeding Torrents and was sleeping when this hit the front page at Digg. I killed the torrents and it seems to be okay now (being that I’m hitting it from work and all).
October 6th, 2006 at 9:06 am
Yeah, FreeBSD is pretty awesome and blows Linux away even now. http://ftp.freebsd.org use to move 700gb a day on one server back in the mid-90’s. I can’t imagine what it moves now.
October 6th, 2006 at 11:57 am
prepare to be Dugg!
October 6th, 2006 at 2:46 pm
So, you published something interesting and it appeared on Slashdot.
Congratulations, you can be proud.
But really, how interesting is saturating a 1.5Mbit downstream (probably significantly less upstream) connection in 2006?
And yes, FreeBSD rocks
October 6th, 2006 at 2:55 pm
Traffic is going to be a lot more upstream because my server is sending content to peers, not the other way around. I believe my upsteam speed is 512Kbps at best.
I never said saturating anything downstream was interesting, not to mention it’s not downstream that gets saturated, it’s the upstream.
I just thought I’d write a post about how I was able to handle a lot of load on a crappy server running Wordpress.
October 9th, 2006 at 10:19 pm
[...] The last time it was taken down, it was to move to another location in the building. The worst problem I had with it other than trying to get Java 1.5 on it is explaining to the Security Department what cvsup is and why I needed firewall ports opened up so I could update the system. I’ve had the same experience many times as I’ve been using FreeBSD to host sites at home since 1999 or so, in fact I still have an infamous old 466 Celeron running FreeBSD sitting around on a shelf, unplugged. [...]
October 10th, 2006 at 1:38 am
[...] Sterker nog, je kunt ook op een gewone, simpele Celeron@466Mhz een webserver maken die grote aantallen Slashdot-bezoekers kan overleven. Je moet alleen de webserver en ondersteunende software goed afstellen… [...]
October 10th, 2006 at 5:34 pm
[...] Surviving A Slashdotting With a Celeron 466: My Slashdot Experience (tags: apache bsd config digg freebsd howto LAMP http load slashdot server php wordpress web performance) [...]
October 16th, 2006 at 10:39 am
w3rd, slashd0t.0rg
November 8th, 2006 at 7:52 pm
Brandon, iThankyou for this informative post which actually was what made me want to try WordPress with WP-Cache. I’m not a computer scientist, but I find the whole idea of not caching dynamic content utterly bizarre, and I would never have stuck with a CMS that doesn’t have any caching abilities to host my site on my home server. It’s a fairly fast machine, an Athlon XP 1900+ running Linux, but trying out WordPress without caching made me feel unsafe every time I generated some reloads in my browser while watching top’s real-time output…
However, to whom it may concern, I run into a slight problem while trying to use WP-Cache 2.0 on WordPress 2, the feared blank/empty page error. I googled a bit and found to my problem, which turned out to be PHP5-related.
April 18th, 2007 at 5:29 pm
A very interesting article, I have followed a couple of your suggestions and are currently benchmarking the improvments. Not likely that I will ever be dugg but it’s always nice to know that ones server is ticking over the best it can.
April 18th, 2007 at 6:26 pm
[...] I never before thought about installing somthing like this, the idea that the content which my visitors would be viewing isn’t live troubled me slightly as I update my blog several times a day and like to think that as soon as I press publish it’s live. A silly notion I’m sure, but it did worry me none the less. After reading Brandon Harper’s article on Surviving A Slashdotting With a Celeron 466 and reading about wordpress optimisation on openswitch.org I was inspired to give this plugin a go because other than reducing the number of content based plugins I use there is very little else I can do to minimise the amount that apahce2 hits the mysql database. The benifit of this plugin is that it entirely removes the mysql interaction with the page for as long as you wish the page to be cached for. I chose a safe time of ten minutes however if one where to get dugg then a cache timeout of an hour or so would more likely be advisable as then the advantage would be felt for longer instead of ten minutes of low load and then a minute of high load as the page is cached again with no change. It would be nice to see how it handles new post and comments, and wether it updates the cache when they are created. [...]
July 21st, 2007 at 12:16 pm
[...] I also take for granted that WHS has some quirks making it barely usable for Unixy people, and by the way, what kind of an operating system requires a minimum of 512 megabytes of RAM and a Pentium III processor to sit on a network to serve a web interface and files? Again, one of my favorite examples of what real software can do, is this story by a guy whose Celeron 466 MHz box running FreeBSD almost survived the owner’s WordPress blog being featured on Slashdot. [...]
July 31st, 2007 at 10:25 am
[...] Interesting article how a Celeron 466 handled a slashdot with little sweat. He did make some config changes for this to work though, but there is some good tips.read more | digg story [...]
August 13th, 2007 at 9:18 am
[...] read more | digg story Posted by seethepost Filed in news [...]