Recently I had a need to write a socket server in C. In the past I’ve done lots of these in Java, and some in C. Being a huge fan and avid user of memcached, and knowing that memcached uses libevent at its core, I decided to go the libevent route this time. So I looked for some examples. I found cliserver and echoserver, both of which were helpful (I apologize for not providing links, as the links I had before are now dead). So, I went about implementing my server using these two as examples for how to put libevent to work dispatching events and doing non-blocking I/O for me. So far, so good.
Libevent is a nice library for handling and dispatching events, as well as doing nonblocking I/O. This is fine, except that it is basically single-threaded — which means that if you have multiple CPUs or a CPU with hyperthreading, you’re really under-utilizing the CPU resources available to your server application because your event pump is running in a single thread and therefore can only use one CPU core at a time.
The solution is to create one libevent event queue (AKA event_base) per active connection, each with its own event pump thread. This project does exactly that, giving you everything you need to write high-performance, multi-threaded, libevent-based socket servers.
There are mentionings of running libevent in a multithreaded implementation, however it is very difficult (if not impossible) to find working implementations. So, I went about developing a working implementation of a multi-threaded, libevent-based socket server.
The overall architecture is a standard, multi-threaded socket server. The main thread listens on a socket and accepts new connections, then farms the actual handling of those connections out to a pool of worker threads. Each connection has its own isolated event queue and runs on a single worker thread.
You can download the code from the GitHub project page here: https://github.com/roncemer/libevent-thread.
The server itself simply echoes whatever you send to it. Start it up, then telnet to it:
telnet localhost 5555
Everything you type should be echoed back to you.
One advantage of using libevent with a handful of event pump threads, is that in many cases you don’t need hundereds or thousands of threads to achieve good performance under load. In theory, for maximum performance, the number of worker threads should be set to the number of CPU cores available. Feel free to experiment with this. If your request processing code tends to block while waiting for database or other blocking operations to complete, you’ll need hundereds or thousands of threads to achieve good performance at speed, because when a thread blocks, it isn’t doing any work. But if your request processing is more of a compute-intensive task which does not block while waiting for external operations or resources, matching the number of threads to the number of CPU cores should theoretically give you optimum performance.
Also note that the server includes a multithreaded work queue implementation, which can be re-used for other purposes.
Since the code is BSD licensed, you are free to use the source code however you wish, either in whole or in part.
Some inspiration and coding ideas came from echoserver and cliserver, both of which are single-threaded, libevent-based servers.
Multi-Threaded Libevent Server Example: https://github.com/roncemer/libevent-thread