This post is quite succinct as I consider you already know Ganglia, Python and Java.

To monitor Java JMX values with Ganglia, I used JPype. JPype is an effort to allow python programs full access to java class libraries. This allowed me to port easily the JMX Munin plugin, you can get this plugin on Munin Exchange. The port, is really basic as I just converted few method to make sure I can publicly access them from the Python module. You can take a look at the jmxquery.patch or download jmxquery.jar.

diff -urBN jmxquery.a/src/jmxquery/src/org/munin/JMXQuery.java jmxquery.b/src/jmxquery/src/org/munin/JMXQuery.java
--- jmxquery.a/src/jmxquery/src/org/munin/JMXQuery.java	2007-05-14 01:07:44.000000000 -0700
+++ jmxquery.b/src/jmxquery/src/org/munin/JMXQuery.java	2010-01-15 15:40:00.000000000 -0800
@@ -61,7 +61,7 @@
 		this.password = password;
 	}

-	private void connect() throws IOException
+	public void connect() throws IOException
 	{
 		Map environment = null;
 		if (username != null && password != null)
@@ -78,7 +78,18 @@
 		connection = connector.getMBeanServerConnection();
 	}

-	private void list() throws IOException, InstanceNotFoundException, IntrospectionException, ReflectionException
+	public void ping() throws IOException
+	{
+	  try {
+	    MBeanServerConnection testConnection = connector.getMBeanServerConnection();
+	    if (testConnection == null)
+	       this.connect();
+	  } catch (IOException e) {
+	    this.connect();
+	  }
+	}
+
+	public void list() throws IOException, InstanceNotFoundException, IntrospectionException, ReflectionException
 	{
 		if (config == null)
 		{
@@ -90,7 +101,7 @@
 		}
 	}

-	private void listConfig()
+	public void listConfig()
 	{
 		for (FieldProperties field : config.getFields())
 		{
@@ -107,7 +118,22 @@
 		}
 	}

-	private void output(String name, Object attr, String key)
+	public String output(String name, String JmxObjectName, String JmxAttributeName, String Key)
+	{
+    try
+    {
+      Object value = connection.getAttribute(new ObjectName(JmxObjectName), JmxAttributeName);
+      return output(name, value, Key);
+    }
+    catch (Exception e)
+    {
+      System.err.println("Fail to output " + name);
+      e.printStackTrace();
+    }
+    return null;
+	}
+
+	public String output(String name, Object attr, String key)
 	{
 		if (attr instanceof CompositeDataSupport)
 		{
@@ -116,15 +142,17 @@
 			{
 				throw new IllegalArgumentException("Key is null for composed data " + name);
 			}
-			System.out.println(name + ".value " + format(cds.get(key)));
+			//System.out.println(name + ".value " + format(cds.get(key)));
+			return format(cds.get(key));
 		}
 		else
 		{
-			System.out.println(name + ".value " + format(attr));
+			//System.out.println(name + ".value " + format(attr));
+		  return format(attr);
 		}
 	}

-	private void output(String name, Object attr)
+	public void output(String name, Object attr)
 	{
 		if (attr instanceof CompositeDataSupport)
 		{
@@ -142,7 +170,7 @@
 	}

 	@SuppressWarnings("unchecked")
-	private void listAll() throws IOException, InstanceNotFoundException, IntrospectionException, ReflectionException
+	public void listAll() throws IOException, InstanceNotFoundException, IntrospectionException, ReflectionException
 	{
 		Set

Once JPype is installed, make sure your Ganglia is configured with the python module.

# The modules section describes the module
#  that should be loaded.
#   name - module name
#   path - load path of the .so
#   params - path to the directory where mod_python
#             should look for python metric modules
modules {
  module {
    name = "python_module"
    path = "modpython.so"
    params = "/opt/ganglia/python_modules"
  }
}

include ('/opt/ganglia/etc/conf.d/*.pyconf')

Then you can use this basic jmx python module:

"""
Ganglia Module to graph JMX values using JPype
Author: Nicolas Brousse
"""
from jpype import *

import os, platform, sys, time

debug = 0

_jmx = None
_jmx_cur_status = {}
_jmx_pre_status = {}
_jmx_pre_time = {}
_eta = 0
_jmx_params = {'User': '',
              'Password': '',
              'Host': '127.0.0.1',
              'Port': '8989',
              'JMXWrapper': '/opt/ganglia/jmxquery.jar'}

descriptors = []

gauge_metrics = {}

def Metric_Handler(name):
    global _jmx

    if _jmx is None:
        _jmx = initJVM()

    pre = None
    cur = 0
    eta = 0

    if debug:
        print "[ DEBUG ] _jmx_cur_status length: %d" % len(_jmx_cur_status)
        print "[ DEBUG ] _jmx_pre_status length: %d" % len(_jmx_pre_status)

    if _jmx is None:
        raise TypeError("No valid JMX Object !")

    if name in _jmx_cur_status:
        _jmx_pre_status[name] = _jmx_cur_status[name]

    for d in descriptors:
        if d['name'] == name:
            break

    try:
        if "jmxKeyName" in d:
            key = d['jmxKeyName']
        else:
            key = None

        if debug:
            print "[ DEBUG ] _jmx.output(%s, %s, %s, %s)" % \
                (d['name'], d['jmxObjectName'], d['jmxAttributeName'], key)

        _jmx.ping()
        result = _jmx.output(d['name'], d['jmxObjectName'], d['jmxAttributeName'], key)
    except:
        print "Error: ", sys.exc_info()[1]
        raise

    if result.isdigit():
        if debug: print "[ DEBUG ] %s: %s" % (name, result)
        _jmx_cur_status[name] = result

        cur_time = time.time()
        if name in _jmx_pre_time:
            eta = cur_time - _jmx_pre_time[name]
            if debug:
                print '[ DEBUG ] ETA: %d, Last time: %d, Cur time: %d' % (eta, _jmx_pre_time[name], cur_time)

        _jmx_pre_time[name] = cur_time

    if name in _jmx_cur_status:
        cur = _jmx_cur_status[name]
    else:
        cur = 0
    if len(_jmx_pre_status) > 0 and name in _jmx_pre_status:
        pre = _jmx_pre_status[name]
    else:
        pre = 0
    eta = int(eta)

    if name in gauge_metrics:
        if pre is not None and eta > 0:
            ret = (long(cur) - long(pre)) / eta
        else:
            ret = 0
    else:
        ret = cur

    if debug:
        print "[ DEBUG ] Metric_Handler(%s): %s (eta: %s, cur: %s, pre: %s)" % (name, ret, eta, cur, pre)

    return long(ret)

def Init_Metric (name, jmxObjectName, jmxAttributeName, jmxKeyName, tmax, type, slope, units, fmt, handler):
    '''Create a metric definition dictionary object.'''
    d = {'name': name.lower(),
        'jmxObjectName': jmxObjectName,
        'jmxAttributeName': jmxAttributeName,
        'jmxKeyName': jmxKeyName,
        'call_back': handler,
        'time_max': tmax,
        'value_type': type,
        'units': units,
        'slope': slope,
        'format': fmt,
        'description': 'JMX '+name.replace("_"," "),
	    'groups': 'jmx'}
    return d

def metric_init(params):
    global _jmx, _jmx_params

    if debug: print "[ DEBUG ] metric_init()"

    if 'User' in params:
        _jmx_params['User'] = params['User']

    if 'Password' in params:
        _jmx_params['Password'] = params['Password']

    if 'Host' in params:
        _jmx_params['Host'] = params['Host']

    if 'Port' in params:
        _jmx_params['Port'] = int(params['Port'])

    if 'JMXWrapper' in params:
        _jmx_params['JMXWrapper'] = params['JMXWrapper']

    return descriptors

def initJVM():
    if debug: print "[ DEBUG ] initJVM()"

    if platform.architecture()[0] == "32bit":
        arch="i386"
    else:
        arch="amd64"

    jvm="/usr/java/jdk/jre/lib/%s/client/libjvm.so" % arch
    if not os.path.isfile(jvm):
        jvm="/usr/java/jdk/jre/lib/%s/server/libjvm.so" % arch
        if not os.path.isfile(jvm):
            raise IOError("Can't find the libjvm.so (%s)" % jvm)

    if not os.path.isfile(_jmx_params['JMXWrapper']):
        raise IOError("Can't find the JMX Wrapper at'%s'" % \
            _jmx_params['JMXWrapper'])

    try:
        startJVM(jvm, "-Djava.class.path=%s" % _jmx_params['JMXWrapper'])

        url = "service:jmx:rmi:///jndi/rmi://%s:%d/jmxrmi" % \
               (_jmx_params['Host'], int(_jmx_params['Port']))

        JMXQuery = JClass("org.munin.JMXQuery")

        if _jmx_params['Password'] is not '':
            _jmx = JMXQuery(url, username, password)
        else:
            _jmx = JMXQuery(url)

        _jmx.connect()
    except:
        print "Error: ", sys.exc_info()[1]
        raise

    return _jmx

def metric_cleanup():
    '''Clean up the metric module.'''
    #shutdownJVM()

def Build_Conf():
    print "modules {\n module {\n  name = \"jmx\"\n  language = \"python\"\n }\n}\n"
    print "collection_group {\n collect_every = 30\n time_threshold = 60"
    for d in descriptors:
        print " metric {\n  name = \""+d['name']+"\"\n  title = \""+d['description']+"\"\n  value_threshold = 1.0\n }"
    print "}\n"

descriptors.append(
    Init_Metric("java_cpu_time", "java.lang:type=Threading", "CurrentThreadCpuTime", "",
        int(300), 'uint', 'both', '', '%u', Metric_Handler))
descriptors.append(
    Init_Metric("java_cpu_user_time", "java.lang:type=Threading", "CurrentThreadUserTime", "",
        int(300), 'uint', 'both', '', '%u', Metric_Handler))
descriptors.append(
    Init_Metric("java_memory_nonheap_committed", "java.lang:type=Memory", "NonHeapMemoryUsage", "committed",
        int(300), 'uint', 'both', '', '%u', Metric_Handler))
descriptors.append(
    Init_Metric("java_memory_nonheap_max", "java.lang:type=Memory", "NonHeapMemoryUsage", "max",
        int(300), 'uint', 'both', '', '%u', Metric_Handler))
descriptors.append(
    Init_Metric("java_memory_nonheap_used", "java.lang:type=Memory", "NonHeapMemoryUsage", "used",
        int(300), 'uint', 'both', '', '%u', Metric_Handler))
descriptors.append(
    Init_Metric("java_memory_heap_committed", "java.lang:type=Memory", "HeapMemoryUsage", "committed",
        int(300), 'uint', 'both', '', '%u', Metric_Handler))
descriptors.append(
    Init_Metric("java_memory_heap_used", "java.lang:type=Memory", "HeapMemoryUsage", "used",
        int(300), 'uint', 'both', '', '%u', Metric_Handler))
descriptors.append(
    Init_Metric("java_memory_heap_max", "java.lang:type=Memory", "HeapMemoryUsage", "max",
        int(300), 'uint', 'both', '', '%u', Metric_Handler))
descriptors.append(
    Init_Metric("os_memory_physical", "java.lang:type=OperatingSystem", "FreePhysicalMemorySize", "",
        int(300), 'uint', 'both', '', '%u', Metric_Handler))
descriptors.append(
    Init_Metric("os_memory_vm", "java.lang:type=OperatingSystem", "CommittedVirtualMemorySize", "",
        int(300), 'uint', 'both', '', '%u', Metric_Handler))
descriptors.append(
    Init_Metric("java_thread_count", "java.lang:type=Threading", "ThreadCount", "",
        int(300), 'uint', 'both', '', '%u', Metric_Handler))
descriptors.append(
    Init_Metric("java_thread_count_peak", "java.lang:type=Threading", "PeakThreadCount", "",
        int(300), 'uint', 'both', '', '%u', Metric_Handler))

#This code is for debugging and unit testing
if __name__ == '__main__':
    try:
        if len(sys.argv) <= 1:
            debug = 1

        metric_init(_jmx_params)

        if len(sys.argv) <= 1:
            while True:
                for d in descriptors:
                    v = d['call_back'](d['name'])
                time.sleep(5)
        elif sys.argv[1] == "config":
            Build_Conf()
            metric_cleanup()

    except KeyboardInterrupt:
        print "Process interrupted."
        if _JMX_WorkerThread.running and not _JMX_WorkerThread.shuttingdown:
            _JMX_WorkerThread.shutdown()
        time.sleep(0.2)
        sys.exit(1)

You can simply run “python jmx.py config” to generate the jmx.pyconf file. Also, you’d probably need to edit the plugin to check for your own JMX values. A lot of improvement could be done to this plugin, but all the basics are there and it can give you some awesome graph to monitor your Java application.

                          

Steps will be :

1. Create an image with dd
2. Build a new device using the image with an encrypt algorythm by using losetup
3. Format the device using mkfs.ext3
4. Mount the device and start using it !

Of course, when you have mounted the device, your data are readable to anyone who have access to the mounted directory.

Create an image with dd

root@vm-ubuntu-lamp:~# dd if=/dev/zero of=encrypted.img bs=4k count=1000 seek=4001
1000+0 records in
1000+0 records out
4096000 bytes (4,1 MB) copied, 0,10063 seconds, 40,7 MB/s
We now have a raw image file using 4MB.


root@vm-ubuntu-lamp:~# ls -l encrypted.img
-rw-r--r-- 1 root root 20484096 2008-07-12 13:38 encrypted.img
root@vm-ubuntu-lamp:~# du -hs encrypted.img
4,0M    encrypted.img

Create the encrypted device

root@vm-ubuntu-lamp:~# losetup -e aes /dev/loop0 encrypted.img
Password:
ioctl: LOOP_SET_STATUS: Invalid argument

Ooops.. Something wrong. Our losetup bin isn’t patched to use AES. On ubuntu/debian based OS, it’s is to deal.

apt-get install loop-aes-utils
root@vm-ubuntu-lamp:~# losetup -e aes /dev/loop0 encrypted.img
Password:
ioctl: LOOP_SET_STATUS: Invalid argument, requested cipher or key length (128 bits) not supported by kernel

Hmm.. Still not good, we need now to patch or change our kernel for support encryption. We have to check if the “aes” and “cryptoloop” modules are loaded, if not we will load them.

root@vm-ubuntu-lamp:~# lsmod | grep aes
root@vm-ubuntu-lamp:~# modprobe aes
root@vm-ubuntu-lamp:~# lsmod | grep aes
aes                    28608  0
root@vm-ubuntu-lamp:~# lsmod | grep cryptoloop
root@vm-ubuntu-lamp:~# modprobe cryptoloop
root@vm-ubuntu-lamp:~# lsmod | grep crypto
cryptoloop              4096  0
loop                   17928  1 cryptoloop

If you don’t have the module with your current kernel, you will have to build it by activate the some kernel options.

CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_CRYPTO_AES=m
CONFIG_CRYPTO_AES_586=m

Now we should be ok to load our encrypted image.
root@vm-ubuntu-lamp:~# losetup -e aes /dev/loop0 encrypted.img
Password:

Format the device with a proper filesystem

root@vm-ubuntu-lamp:~# mkfs.ext3 /dev/loop0
mke2fs 1.40-WIP (14-Nov-2006)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
5016 inodes, 20004 blocks
1000 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=20709376
3 block groups
8192 blocks per group, 8192 fragments per group
1672 inodes per group
Superblock backups stored on blocks:
8193

Writing inode tables: done
Creating journal (1400 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 39 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Mount the device

Easiest step, just have to use the mount command.

root@vm-ubuntu-lamp:~# mkdir /mnt/encrypted
root@vm-ubuntu-lamp:~# mount /dev/loop0 /mnt/encrypted

root@vm-ubuntu-lamp:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1              7850996   1346292   6105892  19% /
varrun                   63052        40     63012   1% /var/run
varlock                  63052         0     63052   0% /var/lock
procbususb               63052        68     62984   1% /proc/bus/usb
udev                     63052        68     62984   1% /dev
devshm                   63052         0     63052   0% /dev/shm
/dev/loop0               19366      1578     16788   9% /mnt/encrypted

root@vm-ubuntu-lamp:~# df -H
Filesystem             Size   Used  Avail Use% Mounted on
/dev/sda1              8,1G   1,4G   6,3G  19% /
varrun                  65M    41k    65M   1% /var/run
varlock                 65M      0    65M   0% /var/lock
procbususb              65M    70k    65M   1% /proc/bus/usb
udev                    65M    70k    65M   1% /dev
devshm                  65M      0    65M   0% /dev/shm
/dev/loop0              20M   1,7M    18M   9% /mnt/encrypted

If you want to go further on this subject : Encryption HOWTO

                          

1. Initial platform
2. New platform
3. How to do it ?

1. Initial platform

The initial platform is made of five production servers (Apache, PHP, MySQL, Postfix), one “monitoring server” running with cacti, nagios, POP3, IMAP, Subversion, Trac, Primary DNS, etc. and one “backup server” used as Secondary DNS, MySQL Slave and finally rsnapshot server. Of course I manage all this remotely with my old laptop. In this schema the Backup server can deal only 100 Go of backup.

Cons

• Backup server is a single point of failure (SPOF). If i lost my HD, I lost all my backup data
• Don’t use all my disk space on all my server
• When I need more space on my backup server, I have to rent and move to a new and more expensive server
• Backup server represent 15% of all my monthly cost (Really hudge for so small platform !)

2. New platform

I look at a quantity of open project for using distributed filesystem (Coda, AFS, LegionFS, MogileFS, DRDB, etc.), but I finally conclude that my needs wasn’t in the ability to distribute data to many clients with hudge load. I need a way to :

1. grow my available backup disk space easily
2. store my backup data in a redundant maner and most secure way (removing SPOF)
3. less expensive structure

I have lot of space unused on all my production servers. So, I choose to use Network Block Devices (NBD) with Raid 6 implementation. In this schema I currently have 4×100 Go (Raid device) that give me 200 Go of available space for my backups.

Why Raid 6 instead of Raid 5 ?

As I write earlier, my goal isn’t read/write performance, but security aspect. Due to the risk of lost a network member, I choose to use Raid 6 Implementation that let me the possibility to lost two node at the same time.

Pros

• Distributed backup data between all the array members
• No more SPOF if I crash the backup server (This structure let me the possibility to mount the array on another server or my laptop)
• Use all available space on my servers
• Less expensive platform and easy to upgrade

Cons

• If you crash more than two node of your RAID Array you will lost all your data, this mean that you must take care when you choose to reboot a server : RAID reconstruction is not fast !
• When writing to the RAID device that will does many network i/o and probably use some CPU on your nbd-server

3. How to do it ?

first of all, I install nbd-server on each server with unused disk space. I choose three server from my five production servers. I’m using Ubuntu Server distrib, so I just have to do an “apt-get install nbd-server“. Now I have to build an image disk on each server that will be used for the network block device. I use dd for build a 100 Go file image then I start nbd-server on each server (node).

dd if=/dev/zero of=/home/nbd/backup.img bs=1024 count=100000000

nbd-server 10130 /home/nbd/backup.img

I encourage you to look at the man page of nbd-server for know more about all the available option.

Next, we have to setup the main server for building the RAID Array. I choose RAID 6. We previously setup 3 servers for doing 3 Network Block Devices. I need one more server for implementing my RAID 6 Array. I choose to setup a file image on the local server where I build the RAID Array, I just need to use loop device with losetup. Then we “mount” the network block devices locally with nbd-client (look at the man page too).

dd if=/dev/zero of=/home/nbd/backup.img bs=1024 count=100000000

losetup /dev/loop0 /home/nbd/backup.img

modprobe nbd

nbd-client serv1.example.com 10130 /dev/nbd0 -persist

nbd-client serv2.example.com 10130 /dev/nbd1 -persist

nbd-client serv3.example.com 10130 /dev/nbd2 -persist

We have all our device up on our main server, we just need to use mdadm to build our array and mount our file system to use it like any other partition.

mdadm –create /dev/md0 –raid-devices=4 –level=6 –spare-devices=0 –chunk=256 /dev/nbd[012] /dev/loop0

mkfs.ext3 /dev/md0 -b 4096 -E stride=64

mount /dev/md0 /.snapshots

That’s all folks ! I can now run rsnapshot using my distributed RAID 6 implementation and stopping to lost my unused spaces. NB : I use a chunk size of 256k because of the RAID 6 implementation and the network constraint, performance looks better. Due to this chunk size, I use a block size of 4k and stride of 64 while building the file system (4k x 64 = 256k).

I need more space, how can I do ?

It’s easy ! You just need to install a new nbd-server on a server having enough space then load a new nbd-client on your server running the RAID Array. When it’s done, just use mdadm to grow your array.

mdadm –grow /dev/md0 –raid-disks=5

mdadm –add /dev/md0 /dev/nbd4

The server running the RAID Device crashed. How can I get my backup ??

It’s easy ! You just need to build a new device on another server or on your laptop.

mdadm –create /dev/md0 –assume-clean –raid-devices=4 –chunk=256 –level=6 /dev/nbd[012] missing

“missing” mean that our last device of our array (/dev/loop0) is missing, the RAID Array will start in degraded mode.

I have a big external HD at home for doing a second site backup. What I have to do for copying and read it ?

You will just need to copy all the file image (“backup.img”) to your external disk . Take care to stop your RAID Array before (except if you are sure to don’t have any write on your images while doing your copy). Then You will just have to load each images with losetup locally and building a RAID Array with mdadm.

To sum up

I think that this platform design is really useful for small sized (5/10 servers) platform because you get a distributed data backup solution at low cost. You don’t need to have dedicated server to manage your backup. However, you have to take care of the touchy aspect of any raid implementation, a wrong manipulation with mdadm or too many server crash could lead you to the lost of all your data. You have to keep in mind that RAID 6 need lot of time to rebuild large storage. If you plan to reboot lot of your servers, don’t forget to stop your RAID Array.

If you are interested in this kind of platform for build your SAN, I encourage you to look at the iSCSI or AoE Protocols.

                          

The server used for deploy and test the Zend Platform in real production condition is a Bi-Xeon 1,4Ghz with 2Go RAM and 2×36 Go SCSI. This server is part of a cluster made of 10 servers which deliver PHP scripts only. All statics media (gif, css, js, swf, etc.) are delivered thru other dedicated servers.

Statistics Without Zend Platform

• More than 3.000.000 Hits/Day
• 30 Hits/sec
• 60% Average CPU Load

Statistics With Zend Platform

• More than 10.000.000 Hits/Day
• 100 Hits/sec
• 20% Average CPU Load

Download the PDF Version

Download the PNG Version

To sum up

• Manage 3.5 times more traffic
• Having 3 times less load
• Use same quantity of process
• All the advanced futures for debug, profile, analyze your code
• Centralize error message easily from multiple front (With stats, alerts, vhost, etc.)
• HTML Caching solution

Of course, all this stuff could be done with xDebug, xCache and many other Open Source solution, this will depend of your free time for deploying it. The Zend Platform is an “All in One” package with some advanced features and a good interaction between the Platform and your IDE (Zend Studio/Neon).

                          

We will try to tune the following server :

• Current OS : Debian GNU Linux Kernel 2.4.32 ipv4 + GRSEC
• 1Go RAM DDR
• Intel(R) Celeron(R) CPU 2.66GHz
• SWAP 512Mo
• 3Go on / and 226Go on /home
• Running services are Qmail, Bind9, mrtg, Apache 2.2.2, PHP 5.1.4, MySQL 5.0.21

The best way for tuning a server is to have dedicated services on one server and so, having multiple server especially for MySQL and Apache.

We were runing a heavy website with DotClear and the heavy PhpADS with all its stuff (geoip, all counters, etc.)
The server up to a load of 114 in some peak with a swap totally used ! And so.. a big freeze of services… 70k mails/day , 110k pv/day, 12k v/day, 47 sql queries/sec

In fact, services weren’t so loaded but the box was crashing a lot and swapping often without using too much CPU.

First things that I do was to change the Linux Kernel from a 2.4.32 to a 2.6.18. Lot of things were improves in 2.6. I convey you to take a look at those posts :
http://www-128.ibm.com/developerworks/linux/library/l-web26/
http://developer.osdl.org/craiger/hackbench/
http://kerneltrap.org/node/903

After this update, I take the time to update all version software for using a MySQL 5.0.27, PHP 5.2 etc. Without looking at the changelogs, bugfixes will still help us :-)
After this, we will tune our software configuration that still use default values (this is really bad ! :) then we will tune a little the kernel without recompile a new one.

• Apache 2.2.2 Prefork

Our HTTPD is using some modules as url rewriting, server info, php5, GeoIP and other basic modules. We could optimize much more by using an Apache 2.2.3 Worker and only useful modules or even more delivering static pages and using proxy for dynamic pages. All this depend on your developments and your server usage. Here we will only focus on the Apache Prefork.
Nowadays, it’s important to keep active the KEEPALIVE functionality. This will increase the speed of delivring pages for lot of modern browsers (it’s supported by ie, firefox, safari, opera, etc.). The only thing is to touch a little to the default value. In fact, if your keepalive time out is too big, you will keep an entire apache slot open for a user that is probably gone ! A 4 seconds timeout is enough for delivering a full web page and take care of any network congestion. MaxKeepAliveRequests is used to define the maximum number of request manage by an apache slot during a keepalive session. Except if you have lot of pictures to load on your web pages you don’t really need to have a big value at this state.

KeepAlive On
KeepAliveTimeout 4
MaxKeepAliveRequests 500

As I don’t have lot of memory available on the server I ‘m constraint to decrease drastically the number of running servers from 150 to 60. As I have an apache using approximatly 13Mo of memory (withdraw 3Mo of shared memory), I need approximately 600 Mo of available memory when all the apache child process are running. We have to consider, for our further tuning, that this memory is used. It’s really important in our case to dedicate memory for avoid to swap too much and lost the box in a freeze. you can follow your memory usage by using TOP and looking for your apache/httpd process. (Do a quick “man top” for know more). If you have little more free memory you can take a look to the apache documentation for further tuning.

ServerLimit 60
MaxClients 60

Our server is often overload, with lot of traffic. When I need to restart the apache, or in case of any crashes the apache server start with only 5 Child server process and will add new one 1 second later, 2 new child 2 second later, 4 new at the third second, etc. It’s really too long when you are in a peak ! So, I configured StartServers for let us start directly with 30 child Server process. That will help us to deliver quickly the clients and minimize the impact of the server restart.

MinSpareServers and MaxSpareServers is used in same way as StartServer. When your apache server isn’t load, there is idle child waiting for connection. It’s not usefull to have all your child still open but, In case of a new peak the best way to minimize its impact on your server is to deliver web pages as quick as possible. So keeping some idle Child Process still waiting for client isn’t so stupid. Furthermore in case of our touchy server we consider to be able to allocate 600Mo of RAM. So, We can use it even if it’s for idle Child Process as we dedicate this RAM for apache. For avoid any module Memory Leak, and having fully available Child I set the MaxRequestPerChild to 1000, that mean that each 1000 request, the child will be kill and Apache Server will spare a new one. You’ll probably have to set this value to a higher number. It’s depend of the structure of your web page. You will have to monitor a little your server after those change for being sure to don’t have too much child kill/spare instead of delivering web pages.

StartServers 30
MinSpareServers 30
MaxSpareServers 30
MaxRequestsPerChild 1000

Follow some security issue, we don’t display too much information about our server. As we don’t need the reverse lookup on the client ip, we keep the default value of HostnameLookups to Off and by this way we save some network traffic and server load.

ServerTokens Prod
ServerSignature Off
HostnameLookups Off

• PHP 5.1.4

For perform our page generation and save some cpu we use the php extension eaccelerator. Take a look at the documentation for install it.
We dedicate 32Mo of our RAM for eaccelerator (shm_size) and will use it with shared memory and file cache (“shm_and_disk” value for keys, sessions and content variable). (Memory is really useful in our case, because of all the mails, apache log and MySQL disk access that generate too much i/o and slow down considerably all the server). As we don’t change often the php script on the server we don’t need to use the check_mtime functionality. When set to “1″, that will do a stat on the php script for checking of last modification date We don’t need this because we want to save disk access and we don’t have so many updates on the running scripts. We just have to clean the cache directory after an update.

eaccelerator.shm_size=”32″
eaccelerator.cache_dir=”/www/tmp/eaccelerator”
eaccelerator.enable=”1″
eaccelerator.optimizer=”1″
eaccelerator.check_mtime=”0″
eaccelerator.debug=”0″
eaccelerator.filter=”"
eaccelerator.shm_max=”0″
eaccelerator.shm_ttl=”3600″
eaccelerator.shm_prune_period=”1″
eaccelerator.shm_only=”0″
eaccelerator.compress=”1″
eaccelerator.compress_level=”9″
eaccelerator.keys = “shm_and_disk”
eaccelerator.sessions = “shm_and_disk”
eaccelerator.content = “shm_and_disk”

• MySQL 5.0.24

As I don’t manage how has been coding many of running script, I decrease all the timeout MySQL connection for avoid congestion. Then I increase the number off simultaneous MySQL connection as we had lot of “Too many connection” error message.

wait_timeout=6
connect_timeout=5
interactive_timeout=120
max_connections = 500
max_user_connections = 500

Now we change the touchiest part of the MySQL configuration : The RAM usage. It’s touchy because a bad value can really decrease your server performance and result in a big server swap. After some test I decrease the table cache and the key buffer cache to 256Mo. In fact we don’t have so many available ram as we had 600Mo for our HTTPD and we have lot of other services running. I tried to set it up little higher, hopping that the swap won’t be to big, but in fact, due to our i/o load the swap were totaly not a good thing for MySQL :-)

If you are using MYISAM tables I suggest you to use the “concurrent_insert=2” that will really increase your server performance in many case. MYISAM use table lock, with concurrent insert, the engine will sometime bypass the lock and allow INSERT and SELECT to run concurrently. We also disable all engine that is not used (innodb, bdb). Take a look at the MySQL documentation for better tuning.

join_buffer_size=1M
sort_buffer_size=1M
read_buffer_size=1M
read_rnd_buffer_size=1M
table_cache=256M
max_allowed_packet=4M

key_buffer=256M
key_buffer_size=256M
thread_cache=256M
thread_concurrency=2
thread_cache_size=40
thread_stack=128K

concurrent_insert=2

query_cache_limit=1M
query_cache_size=256M
query_cache_type=1
skip-bdb
skip-innodb

• Linux Kernel 2.6.18

Here is a touchy part of our tuning, we will try to perform the Linux Kernel behavior with our server load for save some memory and avoid too much swap. Furthermore, has we done a great stuff above this part, we have to manage more TCP connection and support correctly the peak. We will use the command “sysctl” for doing our update on values.

# display value of a variable or group of variable
sysctl [-n] [-e] variable …
# set a new value toe the specified variable
sysctl [-n] [-e] [-q] -w variable=value …
# display all the variable
sysctl [-n] [-e] -a
# load a sysctl config file
sysctl [-n] [-e] [-q] -p (default /etc/sysctl.conf)

For our test we will create a test config file “/etc/sysctl.conf.testing” and we will load it by using the following command line :

sysctl -p /etc/sysctl.conf.testing

When you will be glad of your change you could rename the file for “/etc/sysctl.conf”. All the sysctl variable are documented with the Kernel Sources. I suggest you to download the documentation corresponding to your kernel version and read it carefully if you decide to change some values.
A really good article on Security Focus give us some key for minimize the impact of a SYN ATTACK / SYN SPOOFING. In this goal we activate the syncookies and the route validation

net.ipv4.conf.default.rp_filter=1
net.ipv4.tcp_syncookies=1
net.ipv4.tcp_synack_retries=3
net.ipv4.tcp_syn_retries=3

As we had some swap troubles, important thing to do is to change the value of vm.swappiness where the default value is 60. This variable control how much the kernel should favor swapping out applications, its value can be 0 to 100. I set it to 10 for minimize the swap.

vm.swappiness=10

We upgrade the max backlog for support more TCP traffic and we change the congestion control algorithm to BIC. The Linux Kernel support lot of congestion algorithm like Reno (default one), htcp, vegas, westwood, etc.

net.core.netdev_max_backlog=2500 # Interface buffering
net.ipv4.tcp_max_syn_backlog=4096
net.core.somaxconn=1024 # Limit of socket listen() backlog. Default is 128.
net.ipv4.tcp_congestion_control=bic

For avoid to have a big TCP queue and so memory usage for not really active connection I decrease some TCP timeout and force the kernel to recycle quickly tcp connection. We don’t cache the value of ssthresh (Slow Start Threshold) for avoid to impact a given host to have a reduced ssthresh for all is next connections.

net.ipv4.tcp_keepalive_time=900
net.ipv4.tcp_fin_timeout=30
net.ipv4.tcp_max_orphans=16384
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_rfc1337=1
net.ipv4.tcp_no_metrics_save=1

It’s critical to use the optimal SEND and RECEIVE socket buffer size for the link you are using. In our case we have a 100Mbits link connection. So for a better TCP connection and congestion control we had to increase the TCP Buffer. You can read more about this here.

net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_rmem=4096 87380 16777216
net.ipv4.tcp_wmem=4096 65536 16777216

• That’s all folks !

Now, this server support twice more traffic load. Technical aspect was our traffic growth bottleneck. Lot of other tuning could be done for better performance (on i/o and disk access, other kernel options, compile a new kernel, using apache worker, etc.). This post was just some clues about how to tune your servers. One important thing to don’t forget is whatever you tune on your server, that will never be enough if you have a bloody developed programs running on it !