Yes, the processing time for this site is still taking forever, but I’ve finally found out what the problem is, and how to fix it.

When I decided to install GeekLog, I went ahead and upgraded all of the essential ports which are required by GL to the latest versions from the FreeBSD ports tree. When I did the cvsup to update the ports, I wasn’t paying close attention to the “supfile” I was using, and I believe it started to update “src-all”, which is not only all of the ports build files, but the full system source as well. I started to update it and then walked away.. when I came back I saw that it was downloading a bunch of files to /usr/src/sys, so I canceled it and started over with the correct supfile.

Once I got GeekLog installed, configured, and started adding stories to it, it seemed very strange to me that the pages were taking so long to process. My first reaction was that I needed to tune MySQL. That did help a bit, but not enough to get it to something more realistic such as 100-250 milliseconds.

I then started playing with the settings in Apache such as commenting out unneeded modules and connectors, etc., and none of that made a difference. By this time my machine would start freezing up when I load tested it and it would take forever for me to even get an SSH session back into it. Once I did I’d only get a bunch of “too many files open” error every time I would run a command, and the only command which worked was /sbin/reboot. This indicated to me something was not releasing file handles properly, but I still had no idea what.

I played more with the Apache configuration file, and found something which is very useful when trying to debug resource leak within Apache or the modules which it loads at start up the parameter MaxRequestsPerChild. Essentially if you set this to a value other than 0, it will kill Apache / Apache child processes once they have hit that limit and then spawn them again, thereby cleaning up any dirty deeds they might of done while they were alive.

I set this to a low number, and guess what? That’s right, my server stopped crashing every few hours. I wouldn’t say that it fixed the problem, but it does make the server self-healing.

My best guess is that when either Apache or PHP compiled, they used some dependencies out of the new FreeBSD source tree which was downloaded during the upgrade process. Since the downloaded source tree and the current world which is running on my box conflict, its probably why Apache or PHP keeps crashing.

The next step to try and resolve this problem is to do an all new buildworld. I’ve tried this several times in the past and something has always trashed the box in the process. I’ll wait until I can get someone to cover my mail and so forth for me and give it a go.