|I started to write this post many weeks ago and finally publish it even if it’s not totally finish. It is just a little feedback about tuning a full LAMP server with some user traffic and services load. Important thing to notice is that all stuff in this post is NOT THE SOLUTION. You will probably have to tune little more for adapt all this to your personal server usage, server load, development & architecture. So, use those tips as a kind of inspiration instead of an “how to”. Don’t forget that when you do such tuning, take care to keep a backup of your previous configuration files.
We will try to tune the following server :
We were runing a heavy website with DotClear and the heavy PhpADS with all its stuff (geoip, all counters, etc.)
In fact, services weren’t so loaded but the box was crashing a lot and swapping often without using too much CPU.
First things that I do was to change the Linux Kernel from a 2.4.32 to a 2.6.18. Lot of things were improves in 2.6. I convey you to take a look at this post on IBM :
After this update, I take the time to update all version software for using a MySQL 5.0.27, PHP 5.2 etc. Without looking at the changelogs, bugfixes will still help us 🙂
Our HTTPD is using some modules as url rewriting, server info, php5, GeoIP and other basic modules. We could optimize much more by using an Apache 2.2.3 Worker and only useful modules or even more delivering static pages and using proxy for dynamic pages. All this depend on your developments and your server usage. Here we will only focus on the Apache Prefork.
As I don’t have lot of memory available on the server I ‘m constraint to decrease drastically the number of running servers from 150 to 60. As I have an apache using approximatly 13Mo of memory (withdraw 3Mo of shared memory), I need approximately 600 Mo of available memory when all the apache child process are running. It’s really important in our case to dedicate memory for avoid to swap too much and lost the box in a freeze. you can follow your memory usage by using TOP and looking for your apache/httpd process. (Do a quick “man top” for know more). If you have little more free memory you can take a look to the apache documentation for further tuning.
Our server is often overload, with lot of traffic. When I need to restart the apache, or in case of any crashes the apache server start with only 5 Child server process and will add new one 1 second later, 2 new child 2 second later, 4 new at the third second, etc. It’s really too long when you are in a peak ! So, I configured StartServers for let us start directly with 30 child Server process. That will help us to deliver quickly the clients and minimize the impact of the server restart.
MinSpareServers and MaxSpareServers is used in same way as StartServer. When your apache server isn’t load, there is idle child waiting for connection. It’s not usefull to have all your child still open but, In case of a new peak the best way to minimize its impact on your server is to deliver web pages as quick as possible. So keeping some idle Child Process still waiting for client isn’t so stupid. Furthermore in case of our touchy server we consider to be able to allocate 600Mo of RAM. So, We can use it even if it’s for idle Child Process as we dedicate this RAM for apache. For avoid any module Memory Leak, and having fully available Child I set the MaxRequestPerChild to 1000, that mean that each 1000 request, the child will be kill and Apache Server will spare a new one. You’ll probably have to set this value to a higher number. It’s depend of the structure of your web page. You will have to monitor a little your server after those change for being sure to don’t have too much child kill/spare instead of delivering web pages.
Follow some security issue, we don’t display too much information about our server. As we don’t need the reverse lookup on the client ip, we keep the default value of HostnameLookups to Off and by this way we save some network traffic and server load.
For perform our page generation and save some cpu we use the php extension eaccelerator. Take a look at the documentation for install it.
As I don’t manage how has been coding many of running script, I decrease all the timeout MySQL connection for avoid congestion. Then I increase the number off simultaneous MySQL connection as we had lot of “Too many connection” error message.
Now we change the touchiest part of the MySQL configuration : The RAM usage. It’s touchy because a bad value can really decrease your server performance and result in a big server swap. After some test I decrease the table cache and the key buffer cache to 256Mo. In fact we don’t have so many available ram as we had 600Mo for our HTTPD and we have lot of other services running. I tried to set it up little higher, hopping that the swap won’t be to big, but in fact, due to our i/o load the swap were totaly not a good thing for MySQL 🙂
If you are using MYISAM tables I suggest you to use the “concurrent_insert=2” that will really increase your server performance in many case. MYISAM use table lock, with concurrent insert, the engine will sometime bypass the lock and allow INSERT and SELECT to run concurrently. We also disable all engine that is not used (innodb, bdb). Take a look at the MySQL documentation for better tuning.
Here is a touchy part of our tuning, we will try to perform the Linux Kernel behavior with our server load for save some memory and avoid too much swap. Furthermore, has we done a great stuff above this part, we have to manage more TCP connection and support correctly the peak. We will use the command “sysctl” for doing our update on values.
For our test we will create a test config file “/etc/sysctl.conf.testing” and we will load it by using the following command line :
When you will be glad of your change you could rename the file for “/etc/sysctl.conf”. All the sysctl variable are documented with the Kernel Sources. I suggest you to download the documentation corresponding to your kernel version and read it carefully if you decide to change some values.
As we had some swap troubles, important thing to do is to change the value of vm.swappiness where the default value is 60. This variable control how much the kernel should favor swapping out applications, its value can be 0 to 100. I set it to 10 for minimize the swap.
We upgrade the max backlog for support more TCP traffic and we change the congestion control algorithm to BIC. The Linux Kernel support lot of congestion algorithm like Reno (default one), htcp, vegas, westwood, etc.
For avoid to have a big TCP queue and so memory usage for not really active connection I decrease some TCP timeout and force the kernel to recycle quickly tcp connection. We don’t cache the value of ssthresh (Slow Start Threshold) for avoid to impact a given host to have a reduced ssthresh for all is next connections.
It’s critical to use the optimal SEND and RECEIVE socket buffer size for the link you are using. In our case we have a 100Mbits link connection. So for a better TCP connection and congestion control we had to increase the TCP Buffer. You can read more about this here.
Now, this server support twice more traffic load. Technical aspect was our traffic growth bottleneck. Lot of other tuning could be done for better performance (on i/o and disk access, other kernel options, compile a new kernel, using apache worker, etc.). This post was just some clues about how to tune your servers. One important thing to don’t forget is whatever you tune on your server, that will never be enough if you have a bloody developed programs running on it !