The Reliable, High Performance TCP/HTTP Load Balancer
Mirror Sites: Master
Quick linksQuick News
They use it!
10GbE load-balancing (updated)
HATop: Ncurses Interface
Herald: load feedback agent
Thanks for your support !
December 25th, 2016 : 1.6.11 and 1.5.19
HAProxy is a free, very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications. It is particularly suited for very high traffic web sites and powers quite a number of the world's most visited ones. Over the years it has become the de-facto standard opensource load balancer, is now shipped with most mainstream Linux distributions, and is often deployed by default in cloud platforms. Since it does not advertise itself, we only know it's used when the admins report it :-)
Its mode of operation makes its integration into existing architectures very easy
and riskless, while still offering the possibility not to expose fragile web servers
to the net, such as below :
We always support at least two active versions in parallel and an extra old one in critical fixes mode only. The currently supported versions are :
Each version brought its set of features on top of the previous one. Upwards compatibility is a very important aspect of HAProxy, and even version 1.5 is able to run with configurations made for version 1.0 13 years before. Version 1.6 dropped a few long-deprecated keywords and suggests alternatives. The most differenciating features of each version are listed below :
Version 1.2 has been in production use since 2006 and provided an improved performance level on top of 1.1. It is not maintained anymore, as most of its users have switched to 1.3 a long time ago. Version 1.1, which has been maintaining critical sites online since 2002, is not maintained anymore either. Users should upgrade to 1.4 or 1.5.
HAProxy is known to reliably run on the following OS/Platforms :
Highest performance is achieved with modern operating systems supporting scalable polling mechanisms such as epoll on Linux 2.6/3.x or kqueue on FreeBSD and OpenBSD. This requires haproxy version newer than 1.2.5. Fast data transfers are made possible on Linux 3.x using TCP splicing and haproxy 1.4 or 1.5. Forwarding rates of up to 40 Gbps have already been achieved on such platforms after a very careful tuning. While Solaris and AIX are supported, they should not be used if extreme performance is required.
Current typical 1U servers equipped with a dual-core Opteron or Xeon generally achieve between 15000 and 40000 hits/s and have no trouble saturating 2 Gbps under Linux.
Well, since a user's testimony is better than a long demonstration, please take a look at Chris Knight's experience with haproxy saturating a gigabit fiber in 2007 on a video download site. Since then, the performance has significantly increased and the hardware has become much more capable, as my experiments with Myricom's 10-Gig NICs have shown two years later. Now as of 2014, 10-Gig NICs are too limited and are hardly suited for 1U servers since they do rarely provide enough port density to reach speeds above 40-60 Gbps in a 1U server. 100-Gig NICs are coming and I expect to run new series of tests when they are available.
HAProxy involves several techniques commonly found in Operating Systems architectures to achieve the absolute maximal performance :
All these micro-optimizations result in very low CPU usage even on moderate loads. And even at very high loads, when the CPU is saturated, it is quite common to note figures like 5% user and 95% system, which means that the HAProxy process consumes about 20 times less than its system counterpart. This explains why the tuning of the Operating System is very important. This is the reason why we ended up building our own appliances, in order to save that complex and critical task from the end-user.
In production, HAProxy has been installed several times as an emergency solution when very expensive, high-end hardware load balancers suddenly failed on Layer 7 processing. Some hardware load balancers still do not use proxies and process requests at the packet level and have a great difficulty at supporting requests across multiple packets and high response times because they do no buffering at all. On the other side, software load balancers use TCP buffering and are insensible to long requests and high response times. A nice side effect of HTTP buffering is that it increases the server's connection acceptance by reducing the session duration, which leaves room for new requests.
There are 3 important factors used to measure a load balancer's performance :
A load balancer's performance related to these factors is generally announced for the best case (eg: empty objects for session rate, large objects for data rate). This is not because of lack of honnesty from the vendors, but because it is not possible to tell exactly how it will behave in every combination. So when those 3 limits are known, the customer should be aware that it will generally perform below all of them. A good rule of thumb on software load balancers is to consider an average practical performance of half of maximal session and data rates for average sized objects.
You might be interested in checking the 10-Gigabit/s page.
Being obsessed with reliability, I tried to do my best to ensure a total continuity of service by design. It's more difficult to design something reliable from the ground up in the short term, but in the long term it reveals easier to maintain than broken code which tries to hide its own bugs behind respawning processes and tricks like this.
In single-process programs, you have no right to fail : the smallest bug will either crash your program, make it spin like mad or freeze. There has not been any such bug found in stable versions for the last 13 years, though it happened a few times with development code running in production.
HAProxy has been installed on Linux 2.4 systems serving millions of pages every day, and which have only known one reboot in 3 years for a complete OS upgrade. Obviously, they were not directly exposed to the Internet because they did not receive any patch at all. The kernel was a heavily patched 2.4 with Robert Love's jiffies64 patches to support time wrap-around at 497 days (which happened twice). On such systems, the software cannot fail without being immediately noticed !
Right now, it's being used in many Fortune 500 companies around the world to reliably serve billions of pages per day or relay huge amounts of money. Some people even trust it so much that they use it as the default solution to solve simple problems (and I often tell them that they do it the dirty way). Such people sometimes still use versions 1.1 or 1.2 which sees very limited evolutions and which targets mission-critical usages. HAProxy is really suited for such environments because the indicators it returns provide a lot of valuable information about the application's health, behaviour and defects, which are used to make it even more reliable. Version 1.3 has now received far more testing than 1.1 and 1.2 combined, so users are strongly encouraged to migrate to a stable 1.3 or 1.4 for mission-critical usages.
As previously explained, most of the work is executed by the Operating System. For this reason, a large part of the reliability involves the OS itself. Latest versions of Linux 2.4 have been known for offering the highest level of stability ever. However, it requires a bunch of patches to achieve a high level of performance, and this kernel is really outdated now so running it on recent hardware will often be difficult (though some people still do). Linux 2.6 and 3.x include the features needed to achieve this level of performance, but old LTS versions only should be considered for really stable operations without upgrading more than once a year. Some people prefer to run it on Solaris (or do not have the choice). Solaris 8 and 9 are known to be really stable right now, offering a level of performance comparable to legacy Linux 2.4 (without the epoll patch). Solaris 10 might show performances closer to early Linux 2.6. FreeBSD shows good performance but pf (the firewall) eats half of it and needs to be disabled to come close to Linux. OpenBSD sometimes shows socket allocation failures due to sockets staying in FIN_WAIT2 state when client suddenly disappears. Also, I've noticed that hot reconfiguration does not work under OpenBSD.
The reliability can significantly decrease when the system is pushed to its limits. This is why finely tuning the sysctls is important. There is no general rule, every system and every application will be specific. However, it is important to ensure that the system will never run out of memory and that it will never swap. A correctly tuned system must be able to run for years at full load without slowing down nor crashing.
Security is an important concern when deploying a software load balancer. It is possible to harden the OS, to limit the number of open ports and accessible services, but the load balancer itself stays exposed. For this reason, I have been very careful about programming style. Vulnerabilities are very rarely encountered on haproxy, and its architecture significantly limits their impact and often allows easy workarounds. Its remotely unpredictable even processing makes it very hard to reliably exploit any bug, and if the process ever crashes, the bug is discovered. All of them were discovered by reverse-analysis of an accidental crash BTW.
Anyway, much care is taken when writing code to manipulate headers. Impossible state combinations are checked and returned, and errors are processed from the creation to the death of a session. A few people around the world have reviewed the code and suggested cleanups for better clarity to ease auditing. By the way, I'm used to refuse patches that introduce suspect processing or in which not enough care is taken for abnormal conditions.
I generally suggest starting HAProxy as root because it can then jail itself in a chroot and drop all of its privileges before starting the instances. This is not possible if it is not started as root because only root can execute chroot(), contrary to what some admins believe.
Logs provide a lot of information to help maintain a satisfying security level. They are commonly sent over UDP because once chrooted, the /dev/log UNIX socket is unreachable, and it must not be possible to write to a file. The following information are particularly useful :
HAProxy also provides regex-based header control. Parts of the request, as well as request and response headers can be denied, allowed, removed, rewritten, or added. This is commonly used to block dangerous requests or encodings (eg: the Apache Chunk exploit), and to prevent accidental information leak from the server to the client. Other features such as Cache-control checking ensure that no sensible information gets accidentely cached by an upstream proxy consecutively to a bug in the application server for example.
The source code is covered by GPL v2. Source code and pre-compiled binaries for Linux/x86 and Solaris/Sparc can be downloaded right here for some old versions :
There are three types of documentation now : the Reference Manual which explains how to configure HAProxy but which is outdated, the Architecture Guide which will guide you through various typical setups, and the new Configuration Manual which replaces the Reference Manual with more a explicit configuration language explanation. The official documentation is the pure-text one provided with the sources. However, Cyril Bonté's automated conversion to HTML is much easier to use and constantly up to date, so it is the preferred one when available.
In addition to Cyril's HTML converter above, an automated format converter is being developed by Pavel Lang. At the time of writing these lines, it is able to produce a PDF from the documentation, and some heavy work is ongoing to support other output formats. Please consult the project's page for more information. Here's an example of what it is able to do on version 1.5 configuration manual.
If you think you don't have the time and skills to setup and maintain a free load balancer, or if you're seeking for commercial support to satisfy your customers or your boss, you have the following options :
Some happy users have contributed code which may or may not be included. Others spent a long time analysing the code, and there are some who maintain ports up to date. The most difficult internal changes have been contributed in the form of paid time by some big customers who can afford to pay a developer for several months working on an opensource project. Unfortunately some of them do not want to be listed, which is the case for the largest of them.
Some contributions were developped and not merged, most often by lack of sign of interest from the users or simply because they overlap with some pending changes in a way that could make it harder to maintain future compatibility.
Quite some time ago now, Cyril Bonté contacted me about a very interesting feature he has developped, initially for 1.4, and which now supports both 1.4 and 1.5. This feature is Geolocation, which many users have been asking for for a long time, and this one does not require to split the IP files by country codes. In fact it's extremely easy and convenient to configure.
The feature was not merged yet because it does for a specific purpose (GeoIP) what we wanted to have for a more general use (map converters, session variables, and use of variables in the redirect URLs), which will allow the same features to be implemented with more flexibility (eg: extract the IP from a header, or pass the country code and/or AS number to a backend server, etc...). Cyril was very receptive to these arguments and accepted to maintain his patchset out of tree waiting for the features to be implemented (Update: 1.5-dev20 with maps now make this possible). Cyril's code is well maintained and used in production so there is no risk in using it on 1.4, except the fact that the configuration statements will change a bit once you upgrade to 1.5.
The code and documentation are available here : https://github.com/cbonte/haproxy-patches/wiki/Geolocation
Neil Mckee posted a patch to the list in early 2013, and unfortunately this patch did not receive any sign of interest nor feedback, which is sad considering the amount of work that was done. I personally am clueless about sFlow and expressed my skepticism to Neil about the benefits of sampling some HTTP traffic when you can get much more detailed informations for free with existing logs.
Neil kindly responded with the following elements :
The value that sFlow brings is that the measurements are standard, and are designed to integrate seamlessly with sFlow feeds from switches, routers, servers and applications to provide a comprehensive end to end picture of the performance of large scale multi-tier systems. So the purpose is not so much to troubleshoot haproxy in isolation, but to analyze the performance of the whole system that haproxy is part of.
Perhaps the best illustration of this is the 1-in-N sampling feature. If you configure sampling.http to be, say, 1-in-400 then you might only see a handful of sFlow records per second from an haproxy instance, but that is enough to tell you a great deal about what is going on -- in real time. And the data will not bury you even if you have a bank of load-balancers, hundreds of web-servers, a huge memcache-cluster and a fast network interconnect all contributing their own sFlow feeds to the same analyzer.
Even after that explanation, no discussion emerged on the subject on the list, so I guess there is little interest among users for now. I suspect that sFlow is probably more deployed among network equipments than application layer equipments, which could explain this situation. The code is large (not huge though) and I am not convinced about the benefits of merging it and maintaining it if nobody shows even a little bit of interest. Thus for now I prefer to leave it out of tree. Neil has posted it on GitHub here : https://github.com/sflow/haproxy.
Please, if you do use this patch, report your feedback to the mailing list, and invest some time helping with the code review and testing.
This table enumerates all known significant contributions that led to version 1.4, as well as proposed fundings and features yet to be developped but waiting for spare time. It is not more up to date though.
Some older code contributions which possibly do not appear in the table above are still listed here.
If you don't need all of HAProxy's features and are looking for a simpler solution, you may find what you need here :
Feel free to contact us for any questions or comments :
Some people regularly ask if it is possible to send donations, so I have set up a Paypal account for this. Click here if you want to donate.
An IRC channel for haproxy has been opened on FreeNode (but don't seek me there, I'm not) :
Here are some links to possibly useful external contents I gathered on the net. I have found most of them due to their link to haproxy's site ;-)