devnulled: A stream for software developers and engineers
Surviving A Slashdotting With a Celeron 466: My Slashdot Experience
06 Jul
Posted by: Brandon Harper in: A Day In The Life Of, ColdFusion, FreeBSD, PHP, Tips, Hacks, & Tricks, Uncategorized
On Sunday, March 6th of 2005 I was messing around on my workstation retagging some Mp3’s so I could add them into my iPod playlists when I got a GMail notification of a new comment on my blog. I went to check it out and it was fairly basic comment “prepare to be slashdotted. I went to check-out Slashdot and didn’t see anything there, so I assumed it was likely just a spam comment. It wasn’t long after that when a link to an entry I’d wrote a couple of months previous appeared at the top of their front page.
I was actually quite shocked—being linked to from Slashdot is a very big deal in the geek community. For those not familiar with Slashdot, the best comparison I can make is if you were an amateur author and had your work published in Newsweek or something like that. Slashdot does not have nearly the readership of Newsweek, but it’s certainly one of the most trafficked international geek news sites that I know of offhand– I’m a regular reader myself.
Once I knew for sure I was getting Slashdotted, I actually decided to get-up from the computer and go do something else while I thought about how I could prepare for a load like that rather than freaking out and breaking things. My server at the time was a temporary server built out of spare parts (I host my site at home) which was over 6 years old, so I had no idea what I’d be able to do with it. After remembering articles several years ago about something to the effect of “A Pentium 90 can easily put out enough bandwidth to saturate a T1�?, I kept that in mind and started to think of ways to eliminate bottlenecks. I eventually came back, saw that my server was non-responsive via HTTP. I then blocked port 80 at the firewall, and went to work.
I was not a bit surprised the server was non-responsive; my site at the time was running on a Celeron 466 with 192 MB of RAM. I also of course used Apache HTTPD Server, as well as PHP5 and MySQL to power my Wordpress blog in which the entry was in that was being linked to.
My first course of action was to figure-out how to make the page static so that it wouldn’t have to hit MySQL each time someone requested that page. I assumed there was probably a Wordpress plug-in to do this with, so I googled for one and found one which Matt Mullenweg himself was using, Staticize Reloaded. (Note: I’ve since switched to an upgraded version of the same plugin.) This was a major help, but not quite enough given that I hadn’t really touched this particular FreeBSD install much as it was temporary until my new server came in.
My second major course of action was to change the settings on Apache to keep it from dying. It’s been quite some time since all of this happened so I don’t remember the exact settings I used, but basically I upped the number of min and max servers, set a server timeout of 15 seconds, and made sure new server processes were started every 500 requests or so to keep runaway processes from killing the server.
One last thing I did was kill all unnecessary processes such as SMTP (I knew my mail would queue on external SMTP servers trying to reach mine, and I get more spam than legitimate mail anyhow), made sure nothing in cron was intensive, etc. Had bandwidth also been a problem after these changes, I planned on moving my site to the default Wordpress template which did not have any graphics—just XHTML & CSS.
From there I decided to reboot the box just to clean it up and open port 80 back-up again to accept HTTP traffic again. Surprisingly FreeBSD never once crashed the whole time even after the initial Slashdotting before I started blocking traffic.. it was just very bogged down with requests (mostly MySQL was dying). I rebooted it after changing the configuration as a precautionary measure. This was a very stock install of FreeBSD– I didn’t even make a custom kernel for it or anything.
Once I put the box back in, it took a bit more Apache tweaking to get it perfect, but eventually my spare parts Celeron 466 was handling a Slashdotting at only 30% CPU utilization! Here is an example of the number of connections it was getting (which is crazy for me to see on a personal machine) as the Slashdot effect was winding down:
Active Internet connections Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 0 791 grimmace.http puddle189.drizzl.3055 ESTABLISHED tcp4 0 33580 grimmace.http c-67-166-235-247.4364 ESTABLISHED tcp4 0 0 grimmace.http 213.58.2.244.1227 ESTABLISHED tcp4 0 32120 grimmace.http puddle189.drizzl.3054 ESTABLISHED tcp4 0 26018 grimmace.http c-67-166-235-247.4363 ESTABLISHED tcp4 0 0 grimmace.http planetlab2.georg.55584 TIME_WAIT tcp4 0 0 grimmace.http c-67-170-48-141..61898 ESTABLISHED tcp4 0 0 grimmace.http tide505.microsof.39548 ESTABLISHED tcp4 0 0 grimmace.http tide532.microsof.54754 ESTABLISHED tcp4 0 0 grimmace.http unassigned.purec.1361 ESTABLISHED tcp4 0 0 grimmace.http c-67-170-48-141..61897 ESTABLISHED tcp4 0 0 grimmace.http tide508.microsof.30208 ESTABLISHED tcp4 0 0 grimmace.http tide70.microsoft.53537 ESTABLISHED tcp4 0 0 grimmace.http c-24-19-42-108.c.4436 ESTABLISHED tcp4 0 0 grimmace.http tide85.microsoft.40545 ESTABLISHED tcp4 0 0 grimmace.http S01060050baa51fa.33235 ESTABLISHED tcp4 0 0 grimmace.http c-24-19-42-108.c.4434 ESTABLISHED tcp4 0 0 grimmace.http S01060050baa51fa.33234 ESTABLISHED tcp4 0 0 grimmace.http tide85.microsoft.40500 ESTABLISHED tcp4 0 0 grimmace.http tide531.microsof.36377 FIN_WAIT_2 tcp4 0 0 grimmace.http tide517.microsof.12997 FIN_WAIT_2 tcp4 0 0 grimmace.http tide518.microsof.18692 FIN_WAIT_2 tcp4 0 0 grimmace.http tide510.microsof.52133 ESTABLISHED tcp4 0 0 grimmace.http c-67-182-8-198.c.60914 FIN_WAIT_2 tcp4 0 0 grimmace.http 68-113-20-232.wa.37925 ESTABLISHED tcp4 0 0 grimmace.http c-67-182-8-198.c.60913 FIN_WAIT_2 tcp4 0 0 grimmace.http 68-113-20-232.wa.37924 ESTABLISHED tcp4 0 0 grimmace.http msnbot.msn.com.37805 TIME_WAIT tcp4 0 12427 grimmace.http dial-148-240-184.3124 ESTABLISHED tcp4 0 0 grimmace.http pool-68-163-42-6.4368 TIME_WAIT tcp4 0 0 grimmace.http unassigned.purec.1359 ESTABLISHED tcp4 0 32327 grimmace.http dial-148-240-184.3122 ESTABLISHED tcp4 0 0 grimmace.http relax15.idiap.ch.40978 TIME_WAIT tcp4 0 0 grimmace.http relax15.idiap.ch.40977 TIME_WAIT tcp4 0 0 grimmace.http pool-68-163-42-6.4366 TIME_WAIT tcp4 0 0 grimmace.http relax15.idiap.ch.40975 TIME_WAIT tcp4 0 0 grimmace.http 24.125.148.80.1176 TIME_WAIT tcp4 0 0 grimmace.http 213.58.2.244.1208 TIME_WAIT tcp4 0 0 grimmace.http CPE0050da6ab5da-.3371 TIME_WAIT tcp4 0 0 grimmace.http 24.125.148.80.1175 TIME_WAIT tcp4 0 0 grimmace.http adsl-70-240-237-.1266 FIN_WAIT_2 tcp4 0 0 grimmace.http CPE0050da6ab5da-.3370 TIME_WAIT tcp4 0 0 grimmace.http bbox.ne.client2..47283 TIME_WAIT tcp4 0 0 grimmace.http msnbot.msn.com.36671 TIME_WAIT tcp4 0 0 grimmace.http host-148-244-150.46757 TIME_WAIT tcp4 0 0 grimmace.http hannibal.lr-s.tu.50584 TIME_WAIT tcp4 0 0 grimmace.http rdu25-17-055.nc..54974 TIME_WAIT tcp4 0 0 grimmace.http 163.80-202-31.ne.57984 TIME_WAIT tcp4 0 0 grimmace.http bbox.ne.client2..47282 TIME_WAIT tcp4 0 0 grimmace.http rdu25-17-055.nc..54973 TIME_WAIT tcp4 0 0 grimmace.http hannibal.lr-s.tu.50578 TIME_WAIT tcp4 0 0 grimmace.http c-67-174-111-123.3682 TIME_WAIT tcp4 0 0 grimmace.http c-67-174-111-123.3681 TIME_WAIT tcp4 0 0 grimmace.http c-67-174-111-123.3680 TIME_WAIT tcp4 0 0 grimmace.http relax15.idiap.ch.40952 TIME_WAIT tcp4 0 0 grimmace.http relax15.idiap.ch.40953 TIME_WAIT tcp4 0 0 grimmace.http c-67-174-111-123.3679 TIME_WAIT tcp4 0 33304 grimmace.http relax15.idiap.ch.40943 LAST_ACK tcp4 0 0 grimmace.http 163.80-202-31.ne.57983 TIME_WAIT tcp4 0 33304 grimmace.http relax15.idiap.ch.40931 CLOSE_WAIT tcp4 0 0 grimmace.http ool-4570bffb.dyn.54562 TIME_WAIT tcp4 0 0 grimmace.http 208-59-174-198.c.1771 TIME_WAIT tcp4 0 0 grimmace.http xdsl-81-173-248-.3833 TIME_WAIT tcp4 0 0 grimmace.http 24-117-147-243.c.60104 TIME_WAIT tcp4 0 0 grimmace.http pcp658486pcs.cap.29623 TIME_WAIT tcp4 0 0 grimmace.http pcp658486pcs.cap.29622 TIME_WAIT tcp4 0 0 grimmace.http ool-4570bffb.dyn.54558 TIME_WAIT tcp4 0 0 grimmace.http 208-59-174-198.c.1770 TIME_WAIT tcp4 0 0 grimmace.http 24-117-147-243.c.60103 TIME_WAIT tcp4 0 0 grimmace.http xdsl-81-173-248-.3825 TIME_WAIT tcp4 0 0 grimmace.http pool-68-238-49-1.50253 TIME_WAIT tcp4 0 0 grimmace.http pool-68-238-49-1.50252 TIME_WAIT tcp4 0 0 grimmace.http pool-68-238-49-1.50251 TIME_WAIT tcp4 0 0 grimmace.http pool-68-238-49-1.50250 TIME_WAIT tcp4 0 24300 grimmace.http ip160-108.om.apa.14426 CLOSING tcp4 0 30640 grimmace.http ip160-108.om.apa.14403 FIN_WAIT_1 tcp4 0 33304 grimmace.http 209.237.230.104.21469 ESTABLISHED tcp4 0 33580 grimmace.http S010600067b078af.1423 ESTABLISHED tcp4 0 29326 grimmace.http bidwell.textdriv.55107 FIN_WAIT_1 tcp4 0 0 grimmace.http c68.115.108.235..3517 FIN_WAIT_1 tcp4 0 0 grimmace.http c68.115.108.235..3516 FIN_WAIT_1 tcp4 0 0 grimmace.http cust1392.vic01.d.20836 FIN_WAIT_1 tcp4 0 0 grimmace.http node-423a2a62.df.62359 FIN_WAIT_1 tcp4 0 0 grimmace.http cust1392.vic01.d.20835 FIN_WAIT_1 tcp4 0 0 grimmace.http node-423a2a62.df.62358 FIN_WAIT_1 tcp4 0 0 grimmace.http 66.89.66.69.ptr..57168 FIN_WAIT_1 tcp4 0 0 grimmace.http 81.104.95.80.ip..13863 FIN_WAIT_2 tcp4 0 0 grimmace.http 66.89.66.69.ptr..57166 FIN_WAIT_1 tcp4 0 0 grimmace.http 81.104.95.80.ip..13854 FIN_WAIT_1 tcp4 0 0 grimmace.http lata228-05-c136..29431 FIN_WAIT_2 tcp4 0 0 grimmace.http lata228-05-c136..29430 FIN_WAIT_2 tcp4 0 24300 grimmace.http ip160-108.om.apa.14400 CLOSING tcp4 0 0 grimmace.http 207-171-180-101..44370 FIN_WAIT_1 tcp4 0 0 grimmace.http 207-171-180-101..44302 FIN_WAIT_1 tcp4 0 0 grimmace.http lata228-05-c136..29429 FIN_WAIT_2 tcp4 0 0 grimmace.http lata228-05-c136..29428 FIN_WAIT_2 tcp4 0 0 grimmace.http ausisapc102-dmz..22690 FIN_WAIT_2 tcp4 0 0 grimmace.http h000625d8511c.ne.4303 FIN_WAIT_2 tcp4 0 0 grimmace.http 209.237.230.104.28587 FIN_WAIT_2 tcp4 0 0 grimmace.http 66.193.93.3.47306 FIN_WAIT_2 tcp4 0 0 grimmace.http 66.193.93.3.47302 FIN_WAIT_2 tcp4 0 0 grimmace.http adsl-6-96-227.ms.46767 FIN_WAIT_2 tcp4 0 33580 grimmace.http c-24-6-68-25.cli.1107 ESTABLISHED
Unfortunately I’d estimate I was offline for around 2 hours while I fixed the machine, etc, so there is no telling how much traffic I would of seen while my site was the top link on Slashdot, but here are a few graphical examples of what type of traffic I saw once I got the server working (I think it would have been significantly more had it not been on a Sunday afternoon.):
Usually I do around 2-2.5 GB of web traffic per month, but in one day I did over 4 GB! Granted this isn’t impressive compared to a real production environment, but not bad for an old Celeron on a DSL line. I’ve also had one of my entries end up on del.ici.ous/popular which upped my traffic to about 4 GB+ of bandwidth in a month, but it was much more steady rather than one big rush of requests all at once.
I had other stuff to do on Sunday other than sit around on the computer, but I did spend a considerable amount of time responding to comments both on Slashdot as well as my own site. Surprisingly I didn’t get any comments which I was forced to delete. I did however have some other strange experiences which I recently wrote about.
I’d say my main highlights to pass on from this experience are:
- Content based sites are easy to scale (having previous work experience with this on a platform that ran about 200 mall websites using a single software platform on just a handful of servers, it was just another data point for said observation)
- FreeBSD is amazing, and should get more hype though it’s unfortunately shadowed by Linux (but I already knew this as well)
- Wordpress is a great, extensible blogging platform, and can stand-up to load without breaking a sweat
- Though it’s not my language of choice to work in when doing web scripting, the combination of PHP and MySQL is amazingly fast while requiring the least amount of money and resources of other web scripting languages
- You may not always be able to do the “academically sound” process when faced with a conflict, but you can certainly hack your way through an unexpected situation to get you by in a pinch (though hacking is toxic in the long term)
All in all it was a very interesting experience, and hopefully it will happen again sometime. The server which was on order at the time is currently serving the site, and should be able to handle much more load without me needing to touch it if it were to happen again, but we’ll see…
38 Responses
Jared Rypka-Hauer
07|Jul|2005 1Brandon…
Great to meet you at CFUnited… hope to see you at Max.
I only hope I get to be slashdotted one day. Perhaps eventually I shall be worthy.
Laterz!
Pete Freitag
07|Jul|2005 2Brandon, also good to meet you at CFUNITED. Thanks for posting this stuff, interesting…
Brandon Harper
07|Jul|2005 3Good to meet both of you as well. I actually seriously doubt I’ll be at Max, but who knows I guess.
After talking to Pete at CFUNITED about this and other various things I remembered I never got around to blogging about what happened and so forth, so this was one item on my post-conference to-do list.
JD on MX
07|Jul|2005 4Traffic-proofing servers
Traffic-proofing servers: Brandon Harper writes of what he did when finding that material on his little Celeron 466 server was suddenly linked to a hot topic on Slashdot. First step was to turn dynamic content (database query) into a static…
Weblog Tools Collection » Surviving A Slashdotting With a Celeron 466 and Wordpress
02|Aug|2005 5[...] lashdotting With a Celeron 466 and Wordpress Categories - LinkyLoo — Mark Surviving A Slashdotting With a Celeron 466 [...]
Deep-Ocean blog machine » Blog Archive » Imagine, je pourrais me faire slashdotter un jour !
02|Aug|2005 6[...] e faire slashdotter un jour ! Toi non plus tu n’y crois pas ? Sâches que ça arrive à des mecs bien de se faire slas [...]
jalenack
02|Aug|2005 7I’ll add one to the list of highlights: Pay attention when you are warned of slashdotting!
Great read.
SteveK
02|Aug|2005 8Well it’s certainly not a Slashdotting, but you’ve been dugg… or is it digged? Take yer pick
LinkLog » Sobreviviendo al efecto Slashdot con WP
02|Aug|2005 9[...] tutorial para los que tienen su blog hosteado en su casa. Corriendo WP en un Celeron 466, Devnulled sobrevivió [...]
devnulled: A blog by Brandon Harper » It’s Gaining Legs…
03|Aug|2005 10[...] r Post Wedding Crashers: My Curt Review Django: The Python Alternative to Ruby on Rails [...]
Incoherent Babble » Blog Archive » Ever Wonder how to Survive a Slashdotting?
03|Aug|2005 11[...] nce many a site has died while trying… Thankfully, there’s finally someone who let’s us know exactly how to do it [...]
The /. (slashdot) effect
04|Aug|2005 12[...] running off a DSL line on old hardware? Brandon Harper [...]
HouseOfMIke » Blog Archive » Surviving A Slashdotting With a Celeron 466
05|Aug|2005 13[...] ut
Surviving A Slashdotting With a Celeron 466
Surviving A Slashdotting With [...]
Russell’s Blog » Surviving A Slashdotting
06|Aug|2005 14[...] deal with exactly these problems back in March of this year. He details his experience of Surviving A Slashdotting With a Celeron 466 [...]
Tom
06|Oct|2006 15Shame you didnt survive the Digg effect…
oh well, a Slashdotting (?) is still pretty impressive
HMTKSteve
06|Oct|2006 16Are you using a Dynamic or Static IP system for your DSL line?
Is your DSL line 1.5Mbps up, down or both?
EveryDigg » Blog Archive » Surviving A Slashdotting With a Celeron 466: My Slashdot Experience
06|Oct|2006 17[...] A cool article that just shows that you dont need to have a ub3r server to survive the digg/slashdot effect, this guy survived it on T1 with a 486!read more | digg story [...]
junger
06|Oct|2006 18It is pretty ironic that this page didn’t survive the Digg effect.
Great story, though!
Brandon Harper
06|Oct|2006 19HMTKSteve:
I have a single static IP, and my DSL line is only 1.5 Mbps down.
» The Digg-Effect Again : devnulled: a blog by brandon harper
06|Oct|2006 20[...] My Slashdotting entry is being dugg/digged again. More to follow, but so far today I’ve served up 8K users, and have a bunch of active connections: [...]
bongobelly.com » Blog Archive » Surviving A Slashdotting With a Celeron 466: My Slashdot Experience
06|Oct|2006 21[...] Another account of how one can counter Slashdotting/The Digg Effect http://devnulled.com/content/2.....xperience/ [...]
Donncha O Caoimh
06|Oct|2006 22Good post, and I have to second your wp-cache-2 recommentation. It’s a great plugin. My lowly VPS survives lots of spikes in traffic because of it!
Brandon Harper
06|Oct|2006 23Tom & junger:
I was seeding Torrents and was sleeping when this hit the front page at Digg. I killed the torrents and it seems to be okay now (being that I’m hitting it from work and all).
Bob
06|Oct|2006 24Yeah, FreeBSD is pretty awesome and blows Linux away even now. http://ftp.freebsd.org use to move 700gb a day on one server back in the mid-90’s. I can’t imagine what it moves now.
metric
06|Oct|2006 25prepare to be Dugg!
Arthur
06|Oct|2006 26So, you published something interesting and it appeared on Slashdot.
Congratulations, you can be proud.
But really, how interesting is saturating a 1.5Mbit downstream (probably significantly less upstream) connection in 2006?
And yes, FreeBSD rocks
Brandon Harper
06|Oct|2006 27Traffic is going to be a lot more upstream because my server is sending content to peers, not the other way around. I believe my upsteam speed is 512Kbps at best.
I never said saturating anything downstream was interesting, not to mention it’s not downstream that gets saturated, it’s the upstream.
I just thought I’d write a post about how I was able to handle a lot of load on a crappy server running Wordpress.
» The Difference Between Retiring Unix and Windows Servers : devnulled: a blog by brandon harper
09|Oct|2006 28[...] The last time it was taken down, it was to move to another location in the building. The worst problem I had with it other than trying to get Java 1.5 on it is explaining to the Security Department what cvsup is and why I needed firewall ports opened up so I could update the system. I’ve had the same experience many times as I’ve been using FreeBSD to host sites at home since 1999 or so, in fact I still have an infamous old 466 Celeron running FreeBSD sitting around on a shelf, unplugged. [...]
RealOpen IT - Open Source, Open Mind » Archives » High Availability Linux Apache cluster voor websites
10|Oct|2006 29[...] Sterker nog, je kunt ook op een gewone, simpele Celeron@466Mhz een webserver maken die grote aantallen Slashdot-bezoekers kan overleven. Je moet alleen de webserver en ondersteunende software goed afstellen… [...]
links for 2006-10-11 « Where Is All This Leading To?
10|Oct|2006 30[...] Surviving A Slashdotting With a Celeron 466: My Slashdot Experience (tags: apache bsd config digg freebsd howto LAMP http load slashdot server php wordpress web performance) [...]
duel
16|Oct|2006 31w3rd, slashd0t.0rg
Thomas
08|Nov|2006 32Brandon, iThankyou for this informative post which actually was what made me want to try WordPress with WP-Cache. I’m not a computer scientist, but I find the whole idea of not caching dynamic content utterly bizarre, and I would never have stuck with a CMS that doesn’t have any caching abilities to host my site on my home server. It’s a fairly fast machine, an Athlon XP 1900+ running Linux, but trying out WordPress without caching made me feel unsafe every time I generated some reloads in my browser while watching top’s real-time output…
However, to whom it may concern, I run into a slight problem while trying to use WP-Cache 2.0 on WordPress 2, the feared blank/empty page error. I googled a bit and found to my problem, which turned out to be PHP5-related.
Simon
18|Apr|2007 33A very interesting article, I have followed a couple of your suggestions and are currently benchmarking the improvments. Not likely that I will ever be dugg but it’s always nice to know that ones server is ticking over the best it can.
Photogabble » Wordpress Processor Load
18|Apr|2007 34[...] I never before thought about installing somthing like this, the idea that the content which my visitors would be viewing isn’t live troubled me slightly as I update my blog several times a day and like to think that as soon as I press publish it’s live. A silly notion I’m sure, but it did worry me none the less. After reading Brandon Harper’s article on Surviving A Slashdotting With a Celeron 466 and reading about wordpress optimisation on openswitch.org I was inspired to give this plugin a go because other than reducing the number of content based plugins I use there is very little else I can do to minimise the amount that apahce2 hits the mysql database. The benifit of this plugin is that it entirely removes the mysql interaction with the page for as long as you wish the page to be cached for. I chose a safe time of ten minutes however if one where to get dugg then a cache timeout of an hour or so would more likely be advisable as then the advantage would be felt for longer instead of ten minutes of low load and then a minute of high load as the page is cached again with no change. It would be nice to see how it handles new post and comments, and wether it updates the cache when they are created. [...]
Windows Home Server and a new kind of home server products at Thomas Nybergh’s pages
21|Jul|2007 35[...] I also take for granted that WHS has some quirks making it barely usable for Unixy people, and by the way, what kind of an operating system requires a minimum of 512 megabytes of RAM and a Pentium III processor to sit on a network to serve a web interface and files? Again, one of my favorite examples of what real software can do, is this story by a guy whose Celeron 466 MHz box running FreeBSD almost survived the owner’s WordPress blog being featured on Slashdot. [...]
Actual Technology News Blog » Surviving A Slashdotting With a Celeron 466: My Slashdot Experience
31|Jul|2007 36[...] Interesting article how a Celeron 466 handled a slashdot with little sweat. He did make some config changes for this to work though, but there is some good tips.read more | digg story [...]
Surviving A Slashdotting With a Celeron 466: My Slashdot Experience « See the Post
13|Aug|2007 37[...] read more | digg story Posted by seethepost Filed in news [...]
Ever Wonder how to Survive a Slashdotting? - Habari
30|Aug|2008 38[...] of us, since many a site has died while trying… Thankfully, there’s finally someone who let’s us know exactly how to do it, using none other than WordPress (uhh, yes, that is what you’re reading [...]
Leave a reply
Search
Categories
Archives
Links
Calendar
A design creation of Design Disease
devnulled is a personal blog and reflects the ideas and opinions of brandon harper and not his employer, dog, shoes, or stapler.
All content copyright © 2002-9002 brandon harper. All rights reserved.
devnulled is powered by FreeBSD, WordPress and the combustion of dead dinosaurs. It has also been made magically delicious by the pfm module.