HAProxy

The Reliable, High Performance TCP/HTTP Load Balancer


Quick links

Quick News
Description
Main features
Supported Platforms
Performance
Reliability
Security
Download
Documentation
Live demo
They use it!
Commercial Support
Add-on features
Other Solutions
Contacts
External links
Mailing list archives
10GbE load-balancing (updated)
Contributions
Coding style
Known bugs



visitors online

Thanks for your support !




News

November, 25th, 2014 : 1.5.9

October, 31th, 2014 : 1.5.8

    Dmitry Sivachenko reported that an occurrence of "-ldl" accidently slipped into the makefile in 1.5.7, complicating builds on systems like FreeBSD with no libdl. Godbach fixed a bug which appears only when users force tune.maxrewrite to 0 (which is impractical in production), and which causes buffer insertions to write at the wrong location and to crash the process. There's no security impact here given that such configurations cannot be used in production. I preferred to issue a new version so that everyone can upgrade without trouble. If you already run with 1.5.7, then you don't need to upgrade to 1.5.8. Code and changelog are available here as usual.

October, 30th, 2014 : 1.5.7

    A nasty bug reported by Dmitry Sivachenko can cause haproxy to die in some rare cases when a monitoring system issues a lot of "show sess" commands on the CLI and aborts them in the middle of a transfer. The probability to hit it is so low that it has existed since v1.4 and was only noticed now. Cyril Bonté fixed a bug causing wrong flags to be sometimes reported in the logs for keep-alive requests. A bug where the PROXY protocol is used with a banner protocol causes an extra 200ms delay for the request to leave, slowing down connection establishment to SMTP or FTP servers. Christian Ruppert found and fixed a bug in the way regex are compiled when HAProxy is built with support for PCRE_JIT but the libpcre is built without. The way original connection addresses are detected on a system where connections are NAT'd by Netfilter was fixed so that we wouldn't report IPv4 destination addresses for v6-mapped v4 addresses. This used to cause the PROXY protocol to emit "UNKNOWN" as the address families differred for the source and destination! John Leach reported an interesting bug in the way SSL certificates were loaded : if a certificate with an invalid subject (no parsable CN) is loaded as the first in the list, its context will not be updated with the bind line arguments, resulting in such a certificate to accept SSLv3 despite the "no-sslv3" keyword. That was diagnosed and fixed by Emeric, who also implemented the global "ssl-default-bind-options" and "ssl-default-server-options" keywords, and implemented "ssl_c_der" and "ssl_f_der" to pass the full raw certificate to the server if needed. That's all for this version. Nothing critical again, but we're just trying to keep a fast pace to eliminate each and every bug. Code and changelog are available here as usual.

October, 18th, 2014 : 1.5.6

    Very few fixes this time, 1.5.6 fixes the annoying bug reported this week about disabled proxies, an issue in the URI hash (the question mark of a query string was accidently hashed when present), an off-by-one when checking the stick-counter number in "track-sc" rules, resulting in the "track-sc3" action being accepted and reported as valid but ignored, and slightly improves the systemd wrapper. Code and changelog are available here.

October, 8th, 2014 : 1.5.5

    This time, nothing really important, just a few issues in the config parser (eg: stop trying to propagate process binding to dynamic backends, stop sending warnings for stats attached to multi-process frontends if it relies on single-process bind lines), a fix for an annoying issue causing the HTTP mode (close/keep-alive) to be ignored in backends, a fix for TCP checks not properly detecting connection failures is there was no tcp-check rule, and better support for supervisord in systemd-wrapper. Since there was no rush here, it can be the good timing to upgrade to a reasonably stable version after testing it calmly :-) As usual code and changelog are available here.

September, 2nd, 2014 : 1.5.4

    A critical bug was fixed in 1.5.4. This bug was introduced in 1.5-dev23, so all users of any version between 1.5-dev23 and 1.5.3 must upgrade. This bug can cause haproxy to crash if a number of conditions are met together. Basically, we need a client which can upload multiples of 2GB of POST data much faster than the server can read, and the server must accept all these data slowly enough. If all of this happens, it is possible during the roll-over at every 2GB that the chunk parser tries to parse a chunk length out of the input buffer, causing haproxy to crash. In practice, it can essentially be exploited when the attacker controls both the client, the server, and the timing. This cannot be used to modify data nor execute code though, it's only a denial of service. CVE-2014-6269 was assigned to this bug. Another bug was a possible busy loop in tcp-request content track-sc rules. Other bugs are less important and can be found in the changelog available with the code here.

July 25th, 2014 : 1.5.3

    Version 1.5.3 fixes a few issues on top of 1.5.2. Essentialy, a possible memory leak un SSL DHE exchanges, and a possible memory corruption when building the proxy protocol v2 header. For sure few people will feel impacted, but better release a new version while everything else is calm. The source code and changelog are available here.

July 12th, 2014 : 1.5.2

    Two extra important issues were discovered since 1.5.1 which were fixed in 1.5.2. The first one can cause some sample fetch combinations to fail together in a same expression, and one artificial case (but totally useless) may even crash the process. The second one is an incomplete fix in 1.5-dev23 for the request body forwarding. Hash-based balancing algorithms and http-send-name-header may fail if a request contains a body which starts to be forwarded before the contents are used. A few other bugs were fixed, and the max syslog line length is now configurable per logger. As usual, the source code and changelog are available here.

June 24th, 2014 : 1.5.1

    Version 1.5.1 fixes a few bugs from 1.5.0 among which a really annoying one which can cause some file descriptor leak when dealing with clients which disappear from the net, resulting in the impossibility to accept new connections after some time. This bug was introduced in 1.5-dev25, so affected users are strongly encouraged to upgrade. For more information, please consult the source code and changelog here.

    Also today I was pleased to receive a bottle of Champagne sent by our friends at Loadbalancer.org! Thank you guys!

June 19th, 2014 : HAProxy 1.5.0 released!

    After 4 years of hard work, HAProxy 1.5.0 is finally released!

    For people who don't follow the development versions, 1.5 expands 1.4 with many new features and performance improvements, including native SSL support on both sides with SNI/NPN/ALPN and OCSP stapling, IPv6 and UNIX sockets are supported everywhere, full HTTP keep-alive for better support of NTLM and improved efficiency in static farms, HTTP/1.1 compression (deflate, gzip) to save bandwidth, PROXY protocol versions 1 and 2 on both sides, data sampling on everything in request or response, including payload, ACLs can use any matching method with any input sample maps and dynamic ACLs updatable from the CLI stick-tables support counters to track activity on any input sample custom format for logs, unique-id, header rewriting, and redirects, improved health checks (SSL, scripted TCP, check agent, ...), much more scalable configuration supports hundreds of thousands of backends and certificates without sweating.

    Since dev26, a few bugs were fixed, and some low-importance things were integrated. Basic OCSP stapling support from Dirkjan and Emeric was finally merged. Sasha's header replace actions were merged as well. I've added a few more info in the stats page (avg response times) and CSV output (health check status), added support for PROXY v2 on the accept side, and added the "capture" action on tcp-request in order to log contents such as SNI or payload. Rémi's dh-param was finally integrated.

    People love numbers, so here are a few. From 1.4.0 to 1.5.0, we had :

    • 1574 calendar days (4 yr 3 mon)
    • 26 development versions (one every 2 months on average)
    • 540 bugs fixed (387 added during 1.5, 153 affecting 1.4 as well)
    • 2549 commits
    • 683 unique commit dates (at least this many days worked)
    • up to 24 commits per day
    • 69712 lines removed, 122279 lines added
    • many extremely useful bug reports (too many to list)
    • 73 code/doc contributors : Adrian Bridgett, Alex Davies, Aman Gupta, Andreas Kohn, Apollon Oikonomopoulos, Arnaud Cornet, Baptiste Assmann, Bertrand Jacquin, Bhaskar Maddala, Conrad Hoffmann, Cyril Bonté, Daniel Schultze, David BERARD, David Cournapeau, Dave McCowan, David du Colombier, Delta Yeh, Dirkjan Bussink, Dmitry Sivachenko, Emeric Brun, Emmanuel Hocdet, Evan Broder, Finn Arne Gangstad, Gabor Lekeny, Geoff Bucar, Wei Zhao, Guillaume Castagnino, Guillaume de Lafond, Hervé COMMOWICK, Hiroaki Nakamura, James Voth, Jamie Gloudon, Jarno Huuskonen, Joe Williams, Joshua M. Clulow, Julien Vehent, Justin Karneges, Kevin Hester, Kevin Musker, Kristoffer Grönlund, Krzysztof Piotr Oledzki, Lukas Tribus, Marc-Antoine Perennou, Mark Lamourine, Mathieu Trudel, Michael Scherer, Neil Prockter, Nenad Merdanovic, Nick Chalk, Olivier Burgard, Oskar Stolc, Patrick Mézard, Pieter Baauw, Prach Pongpanich, Rauf Kuliyev, Remi Gacogne, Sagi Bashari, Sasha Pachev, Sean Carey, Sergiy Prykhodko, Simon Horman, Simone Gotti, Stathis Voukelatos, Tait Clarridge, Thierry Fournier, Todd Lyons, Vincent Bernat, William Lallemand, William Turner, Willy Tarreau, Yuxans Yao, Yves Lafon.

    Additionally, we are very thankful to a few organisations who have sponsored the development of certain advanced features which required to dedicate a person or a team for a significant amount of time (I hope I have not missed any) :

    Don't forget to offer a beer to your distro packagers who make your life easier. It's hard to list them all, but if you don't build from sources, you're likely running a package made and maintained by one of these people :

    • debian: Vincent Bernat, Apollon Oikonomopoulos, Prach Pongpanich
    • Fedora: Ryan O'hara
    • OpenSuSE: Marcus Rückert
    • Other?: contact me to mention you

    And last, I'd like to assign a special mention to our most active mailing list supporters during that period who make the project a reality by off- loading the support task from developers and kindly help our 800 permanent subscribers on a daily basis, BIG THANKS to you guys :

    • Baptiste Assmann
    • Lukas Tribus
    • Cyril Bonté
    • Jonathan Matthews
    • Thomas Heil

    For the HAProxy development team here in France, it will be time to do some errands and buy some Champagne to celebrate the event :-)

June 7th, 2014 : HTTP/1.1 reloaded

    Six years ago, Daniel Stenberg notified on the libcurl mailing list that Mark Nottingham, chair of the IETF HTTP Working Group, initiated a review of existing HTTP implementations. Since HAProxy was mentionned in this list, Aleks Lazic relayed this mail to the haproxy mailing list and I attempted to fill in the form with the information I could provide on HAProxy. By then HAProxy 1.3.15 had just been released without keep-alive nor HTTP/1.1 support. Being confronted to a lot of interoperability issues between clients and servers through HAProxy, I felt that it was a great opportunity to join the Working Group to try to improve the situation from my angle of view. I was impressed to see that participants work professionally and calmly. Noone really tries to defend his own implementation, what matters is interoperability, even if that sometimes needs changes in their code. Of course, the concerns are different for various types of components (eg: security, performance, complexity, etc), often resulting in long debates. And like many other products, HAProxy experienced quite a number of code changes consecutive to clarifications made to the spec. The latest example of this happened after Google Chrome users were seeing HAProxy's 408 error page on a number of web sites.

    And 7 years after this positive work started (OK, 6 years for me, I started late), the group's efforts have finally resulted in 11 new RFCs! The 6 first (RFC 7230 to 7235) replace the aging RFC2616 :

    1. RFC7230 : Message Syntax and Routing
    2. RFC7231 : Semantics and Content
    3. RFC7232 : Conditional Requests
    4. RFC7233 : Range Requests
    5. RFC7234 : Caching
    6. RFC7235 : Authentication

    The next ones are about authentication scheme and method registrations (7236 and 7237), and extensions whose discussion was initiated in the working group but which were not carried by the working group (status code 308, "Forwarded" and "Prefer" header).

    If you're implementing an HTTP agent (client, server, proxy, gateway, or whatever), please read them. They cover a lot of corner cases that are encountered in field and which are the result of the fact that the protocol evolved faster in field than in the docs. The first two ones are already a very good start to demystify the protocol and get rid of your wrong beliefs.

    What next ? HTTP/2.0 is on the rails and has been for about 2 years now. It's time to focus on it!

May 28th, 2014 : 1.5-dev26 : hopefully last!

    This version tries to be the last one in the development cycle. There were very few changes. The agent-check was completed. A potential CPU busy loop when reloading with the systemd wrapper was fixed. A possible buffer overflow with regex in inlikely configs was fixed. The "str" match is now used by default for string samples. No major new feature is considered missing for the final release, so only fixes and minor updates may be merged now. Source code and changelog are available as usual here.

May 10th, 2014 : 1.5-dev25

    Four important issues were fixed since dev24. One could cause crashes on out-of-memory. Another one concerns FreeBSD where the shared session cache could have been used without locking, causing random crashes as well. The recent fixes for HTTP request body forwarding randomly caused pauses when using balance url_param. Last, arguments "-i" and "-n" were ignored on ACLs since dev23. Some pending changes were completed as well. Half-closed timeouts are now supported. Unix sockets are supported on the server side, as well as abstract namespace sockets on Linux. This allows backends and frontend to connect together without consuming TCP ports. The old unmaintained BSD and OSX Makefiles were removed. Per-listener process binding is finally possible using the "process" keyword on "bind" lines, which makes it possible to have one stats socket per process. Version 2 of the PROXY protocol was implemented on the server side. A few other minor improvements were made. Source code and changelog are available here.

Apr 26th, 2014 : 1.5-dev24

    This version fixes 3 major regressions from dev23, one causing transfers to be interrupted after the configured timeout, another one where redirects could sometimes cause a crash, and one slowing down SSL. Other minor issues were fixed. The stats page now supports chunked mode, keep-alive and compression, and may be declared in any section with no warning. Health checks can be started within a smaller delay. http-request/response now support set-map/del-map/add-acl/del-acl to add/remove pattern entries to maps and ACLs on the fly based on data extracted from the traffic. Heartbleed attacks (CVE-2014-0160) are detected and blocked even on vulnerable OpenSSL implementations. Source code and changelog are available here.

Apr 23th, 2014 : 1.5-dev23 : Broken

    This new version addresses half of the remaining changes before -final : use_backend supports log-format expressions. Maps and ACLs now share the same pattern lists which are dynamically updatable from the CLI. SSL web sites now load faster thanks to dynamic record size adjustments. Compression of chunked HTTP responses was fixed and enabled again. About 35 bugs were fixed. We're getting close to 1.5-final. Source code and changelog are available here.

Feb 3rd, 2014 : 1.5-dev22

    The whole polling system was finally reworked to replace the speculative I/O model designed 7 years ago with a fresh new event cache. This was needed because the OpenSSL API is not totally compatible with a polled I/O model since it stores data into its internal buffers. One of the benefits is that despite the complexity of the operation, the code is now less tricky and much safer. HTTP keep-alive is now enabled by default when no other option is mentionned, as it is what users expect by default when they don't specify anything and end up with tunnel mode. A new option "tunnel" was thus added for users who would find a benefit in using tunnel mode. After an HTTP 401 or 407 response, we automatically stick to the same server so there's normally no need for "option prefer-last-server" anymore. SSL buffer size are automatically adjusted to save round trips as suggested by Ilya Grigorik, reducing the handshake time by up to 3. The CLI reports more info in "show info" and now supports "show pools" to check memory usage. SSL supports "maxsslrate" to protect the SSL stack against connection rushes. The "tcp-check" directive now supports a "connect" action, enabling multi-port checking. Very few changes are pending before 1.5-final : ACL/map merge (being reviewed), HTTP body analyzer fixes (in progress), agent check update (started), per-listener process binding (experimentations completed). Source code is available here.

Dec 17th, 2013 : 1.5-dev21

    Several early testers reported a few annoying bugs yesterday, so I preferred to fix them quickly and issue another release than wasting everyone's time on these bugs. Use this version instead of 1.5-dev20 to be safe. Please refer to the changelog for more information. Source code is available here.

Dec 16th, 2013 : 1.5-dev20 : keep-alive, finally!

    This version took 6 months to be released given the difficulty of some changes, and it collected a number of new features. The most awaited of them is server-side keep-alive. The first commit for this feature was initially attempted almost 4 years ago. There are still some limitations, idle server connections are not accounted for maxconn and not reported in the stats for example. And option http-keep-alive still needs to be specified to benefit from the feature. The memory usage has significantly dropped, 640 bytes saved per idle connection on 64-bit systems. All sample fetch expressions (including ACLs) now support a list of converters applied to the sample. The new map feature allows input samples to be converted to other ones. The most common usage of this is geolocation, but it may also be used for massive redirect tables. Maps are updatable live from the CLI. Redirects support the log-format syntax and can embed some elements collected from the request. Hash algorithms can now be selected per backend. Health checks have been improved with the agent-check and tcp-check to build send/expect rules. Please refer to the changelog for more information. Source code is available here.

Jun 17th, 2013 : 1.4.24 and 1.5-dev19 : security fix

    A new vulnerability was discovered in all versions from 1.4.4 and above. It was fixed today with 1.4.24 and 1.5-dev19 (CVE-2013-2175). This vulnerability can induce a crash when configs involving tail-relative header offset such as hdr_ip(xff,-1) are used. Please see the advisory for more details. All 1.4 and 1.5 users must upgrade.

    Additionally, a few other important bugs were fixed. One of them was a regression introduced in 1.5-dev12, which could randomly crash haproxy on rare circumstances when using pipelined requests with a slow client. Last, some endless loops were possible since 1.4 when using consistent hashing algorithms with flapping servers.

    On the positive side, many small new features were finally merged, such as http-response rule sets, ability to change task priority, DSCP field, Netfilter mark and log-level based on L7 ACLs, ability to selectively accept the PROXY protocol header using ACLs, support for environment variables in ACLs and fetches, addition of a 3rd stick-counter, filtering on the stats page, transparent proxy for FreeBSD/OpenBSD, and a few other things. Last but not least, the doc on ACLs/fetches got a major lift-up to deduplicate keywords. A few important issues still need to be addressed, and server-side keep-alive is expected as well before final 1.5 can be released, hopefull by the end of this summer.

    Please refer to the 1.4 changelog and 1.5 changelog for more information. Source code is available here for 1.4 and here for 1.5.

Apr 3rd, 2013 : 1.4.23 and 1.5-dev18 : security fix

    A vulnerability in all 1.4 and 1.5 releases was fixed in 1.4.23 and 1.5-dev18 (CVE-2013-1912), affecting HTTP fetches in TCP request inspection rules. All 1.4 and 1.5 users must upgrade.

    In addition to this, 1.5-dev18 brings a significant number of improvements, among which use of environment variables in all addresses (bind, log, source, server, ...), agent-based health check system, support for systemd, TLS ALPN, and a few other nice features.

    Please refer to the 1.4 changelog and 1.5 changelog for more information. Source code is available here for 1.4 and here for 1.5.

Apr 1st, 2013 : April Fool's

Recent news...

Dec 28th, 2012 : Development 1.5-dev17

    I broke a few things in dev16 which have been repaired in 1.5-dev17. This is a very small version with few changes. All 1.5-dev users are then encouraged to upgrade to dev17.

Dec 24th, 2012 : Development 1.5-dev16

    Here comes 1.5-dev16. Thanks to the amazing work Sander Klein and John Rood have done at Picturae ICT we could finally spot the freeze bug after one week of restless digging ! This bug was amazingly hard to reproduce in general and would only affect POST requests under certain circumstances that I never could reproduce despite many efforts. It is likely that other users were affected too but did not notice it because end users did not complain (I'm thinking about webmail and file sharing environments for example). During this week of code review and testing, around 10 other minor to medium bugs related to the polling changes could be fixed.

    Another nasty bug was fixed on SSL. It happens that OpenSSL maintains a global error stack that must constantly be flushed (surely they never heard how errno works). The result is that some SSL errors could cause another SSL session to break as a side effect of this error. This issue was reported by J. Maurice (wiz technologies) who first encountered it when playing with the tests on ssllabs.com.

    Another bug present since 1.4 concerns the premature close of the response when the server responds before the end of a POST upload. This happens when the server responds with a redirect or with a 401, sometimes the client would not get the response. This has been fixed.

    Krzysztof Rutecki reported some issues on client certificate checks, because the check for the presence of the certificate applies to the connection and not just to the session. So this does not match upon session resumption. Thus another ssl_c_used ACL was added to check for such sessions.

    Among the other nice additions, it is now possible to log the result of any sample fetch method using "%[]". This allows to log SSL certificates for example. And similarly, passing such information to HTTP headers was implemented too, as "http-request add-header" and "http-request set-header", using the same format as the logs. This also becomes useful for combining headers !

    Some people have been asking for logging the amount of uploaded data from the client to the server, so this is now available as the "%U" log-format tag. Some other log-format tags were deprecated and replaced with easier to remind ones. The old ones still work but emit a warning suggesting the replacement.

    And last, the stats HTML version was improved to present detailed information using hover tips instead of title attributes, allowing multi-line details on the page. The result is nicer, more readable and more complete, as can be seen on the demo page.

    All 1.5-dev users are then encouraged to upgrade to dev16. Update: minor last-minute regression on the stats page, please use the latest snapshot instead.

Dec 12th, 2012 : Development 1.5-dev15

    This is an incremental fixup on top of dev14 to address the few remaining bugs that were reported since its release, and particularly the high CPU usage that a few users have reported. Some SSL issues were also fixed and its cache was improved to use 4 times less memory. The conditions to enable compression were tightened. The strange server errors that were logged and counted for years were in fact client errors, and that was fixed. SSL handshake errors are now logged. Tracking layer 7 information is now possible ; it was limited to "src" till now. It will allow people behind proxies to benefit from some scraping or DoS protection.

    All 1.5-dev users are then encouraged to upgrade to dev15.

Nov 26th, 2012 : Development 1.5-dev14

    This is a quick fixup for all the bugs that were reported in dev13. All users are encouraged to upgrade to dev14 and to drop both dev12 and dev13 !

Nov 22th, 2012 : Development 1.5-dev13 with Compression!

    This is the largest development version ever issued, 295 patches in 2 months! We managed to keep the Exceliance team busy all the time, which means that the code is becoming more modular with less cross-dependences, I really like this ! First, we got an amazing amount of feedback from early adopters of dev12. It seems like SSL was expected for too long a time. We really want to thank all those who contributed patches, feedback, configs, cores (yes there were) and even live gdb access, you know who you are and you deserve a big thanks for this! Git log says there were 55 bugs fixed since dev12 (a few of them might have been introduced in between). Still, this means that dev12 should be avoided as much as possible, which is why I redirected many of you to more recent snapshots. These bugs aside, I'm proud to say that the whole team did a really great job which could be summarized like this :

    1. SSL:
      • many more features ; client and server certificates supported on both sides with CA and CRL checks. Most of the information available in SSL can be used in ACLs for access control. Some information such as protocol and ciphers can be reported in the logs. These information are still not added to HTTP logs though, a lot of config work is still needed.
      • cache life time and maximum concurrent SSL connections can be set. Unfortunately OpenSSL happily dereferences NULL malloc returns and causes the process to die if memory becomes scarce. So we can only limit its maximum amount of connections if we want to limit the memory it uses.
      • TLS NPN was implemented with the help from Simone Bordet from Jetty, and can be used to offload SSL/TLS for SPDY and to direct to a different server depending on the protocol chosen by the client.
      • Ivan Ristic from ssllabs and Andy Humphreys from Robinson-way provided very valuable help in diagnosing and fixing some nasty issues with aborts on failed handshakes and improve from an E-grade to an A-grade.
    2. HTTP Compression
      • HTTP payload compression was implemented at Exceliance to achieve bandwidth usage reduction and reduce page load time on congested or small links. Compression is extremely CPU and memory intensive, so we spent most of the time developing dynamic adaptations. It is possible to limit the maximum RAM dedicated to compression, the CPU usage threshold and bandwidth thresholds above which compression is disabled. It is even possible to adjust some of these settings from the stats socket and to monitor bandwidth savings in real time. Proceeding like this ensures a high reliability at low cost and with little added latency. I've put it on the haproxy web site with nice bandwidth savings (72% avg on compressible objects, 50% on average, considering that most downloads are compressed sources). I must say I'm very happy of this new feature which will reduce bandwidth costs in hosted infrastructures ! And it goes back to the origins of haproxy in zprox 14 years ago :-)
    3. Health checks
      • SSL is now usable with health checks. By default it is enabled if the server has the "ssl" keyword and no "port" nor "addr" setting. It can be forced using "check-ssl" otherwise. So now running an HTTPS health check simply consists in using "option httpchk" with "ssl" on the server.
      • send-proxy is also usable with health checks, with the same rules as above, and the "check-send-proxy" directive to force it. The checks also respect the updated spec which suggests sending real addresses with health checks instead of sending unknown addresses. This makes it compatible with some products such as postfix 2.10 for example.
    4. Polling
      • speculative polling was generalized to all pollers, and sepoll disappeared as it was superseded by epoll. The main reason for this important change is the way OpenSSL works and the fact that it can easily get stuck with some data in buffers with no I/O event to unblock them. So we needed to engage into this difficult change. I'd have preferred to delay it to 1.6 if I was offered the choice ! But in the end this is good because it's done and it improves both performance and reliability. Even select() and poll() are now fast.
      • the maxaccept setting was too low on some platforms to achieve the highest possible performance, so it was doubled to 64 and is now per listener so that it automatically adjusts to the number of processes the listener is bound to. This ensures both best performance in single process mode, and quite good fairness in multi-process mode.
    5. Platform improvements
      • Linux 3.6 TCP Fast Open is supported on listeners ("tfo" bind keyword). This is used to allow compatible clients to re-establish a TCP connection in a single packet and save one round-trip. The kernel code for this is still young, I'd be interested in any feedback.
      • use of accept4() on Linux >= 2.6.28 saves one system call.
    6. Process management
      • stats socket can now be bound to specific processes. This is useful to monitor a specific process only.
      • "bind-process" now supports ranges instead of silently ignoring them.
      • "cpu-map" establishes a mapping between process numbers and CPU cores. This is important when running SSL offloaders on dedicated processes because you don't want them to pollute the low-latency L7 core.
    7. Misc : "redirect scheme" makes it easier to redirect between http and https, config error reporting was improved for "bind" and "server" lines by enumerating the list of supported options dynamically.

    I must say I'm much more confident in dev13 than I was with dev12 and I have already upgraded the main web site which has been upgraded every few days with recent snapshots. I've build and run it on Linux i586/x86_64/armv5/v7, OpenBSD/amd64 and Solaris/sparc without any issue anymore.

    To all those running SSL tests on dev12, please drop it for dev13. I don't think we introduced regressions (but that's still possible), but I know for sure that we fixed a lot! The usual changelog and source are available at the usual place.

Sept 10th, 2012 : Development 1.5-dev12 with SSL!!!

    The main, long-awaited, feature this time is native SSL support on both sides, with SNI and multi-process session sharing. The work took several months to be done at Exceliance because it required a major rewrite of the lower connection layers in order to support multiple data layers. This was a very painful task, but doing so allowed us to shrink the SSL patch from several thousands of lines of hardly maintainable code to a few hundreds of SSL-specific code. The code supports the Server Name Indication TLS extension (SNI), which consists in presenting the certificate which matches the host name requested by the client. This also works with wildcard certificates, of course. The certificates can be loaded from a directory, which makes it more convenient to load hundreds or thousands at a time. And since they are loaded into a binary tree, there is no lookup overhead even if there are hundreds of thousands, which is very convenient for massive hosting providers.

    In current state, the code does not yet support checking certificates, which also means that connecting to an SSL server is only useful if the LAN is safe (in short, it's only useful if the server absolutely wants to get the connection to port 443). But the Exceliance team is actively working on this.

    We took care of correctly arranging connection and data layers. Right now it's perfectly possible to chain multiple layers of haproxy servers to offload more SSL, using SSL-ID affinity and the PROXY protocol in order not to lose the client's source address. Doing this with off-the shelf hardware can result in quite a cheap SSL offloader even for huge loads. We measured 4000 TPS on SSLv3 on an Atom D510 and have not yet run the tests on larger hardware.

    Among the other features in this version, we can list IPv6 transparent mode, "base" pattern/acl to match a concatenation of the Host header and the URI, "urlp_val" ACL to match a URL parameter's value, support for the "nice" keyword on "bind" lines to change the priority of sessions using this bind line (useful to limit SSL CPU impact), the ability to clear/feed stick-table entries on the stats CLI (which got lost forgotten in a dead branch), and the usual set of halog features and optims.

    The changelog is available for more information, though there are a lot of commits to transform the connection layers. Users who need SSL should really give it a try. While we got a number of useful reports on the mailing list and could fix some issues, it is very likely that some bugs remain, so if you observe abnormal behaviours, please report your experiences there.

    On the stable branch side, 1.4.22 was silently released one month ago with a number of small fixes and a number of minor feature improvements, such as the ability for putting a server in soft-stop mode from the stats web page in admin mode, and support for the "httponly" and "secure" flags on cookies.

June 4th, 2012 : Development 1.5-dev11

    A large number of bugs were fixed again since 1.5-dev10, some of them being regressions from 1.5-dev8 and later versions. See the changelog for more information, but nobody should be running on dev9 nor dev10. Minor harmless features were added in dev11, such as new actions on the stats page, a few new cookie options, and some minor improvements on URI hashing and server recovery mode. Users should really upgrade, as I don't want to waste time trying to spot stupid bugs in configs that are notoriously broken.

May 21st, 2012 : Stable 1.4.21

    A number of old bugs were reported recently. Some of them are quite problematic because they can lead to crashes while parsing configuration or when starting up, which is even worse considering that startup scripts will generally not notice it. Among the bugs that 1.4.21 fixes, we can list : risk of crash if using reqrep/rsprep and having tune.bufsize manually configured larger than what was compiled in, risk of crash when using header captures on a TCP frontend (uncaught invalid configuration), risk of crash when some servers are declared with checks in a farm which does not use an LB algorithm (eg: "option transparent" or "dispatch"), "balance source" did not correctly hash IPv6 addresses resulting in IPv4 connections to IPv6 listeners always having the same hash. Some other minor fixes and improvements were merged. While it's very likely that almost nobody is affected by the bugs above, troubleshooting them is annoying enough to justify an upgrade.

May 8th, 2012 : Development 1.5-dev9

    Many new features were added since 1.5-dev7 (I forgot to announce dev8 here). Let's summarize this shortly : new logging subsystem with customizable log formats, a unique-ID generator, full rework of the buffers and HTTP message storage, merge of the ACL and pattern fetch code, ACL support for IPv6 addresses, cookies, URL parameters and arbitrary payload, support for specifying a precise occurrence in fetch functions, much better error reporting for ACL parsing errors, the long-awaited "use-server" directive, minor improvements to the error capture reports, and a significant number of bugfixes. Please give it a test.

March 10th, 2012 : Stable 1.4.20

    A few bugs were reported since 1.4.19 was released, and some were found in 1.5 during development. Servers tracking disabled servers would still be used while disabled. Zero-weight servers could still dequeue requests pending in the backend's queue. The build was broken on FreeBSD since 1.4.19. Since the introduction of client keep-alive, a server would not pick a pending requests after releasing a connection if it keeps exactly maxconn-1 connections, which is problematic with low maxconn values. POST requests smaller than the buffer would experience an undesirable additional delay of 200ms due to a flag being left unconditionally enabled on the buffer. Sometimes when sending data wrapping across the buffer, haproxy would fail to merge TCP segments into a single one, which results in a few PUSH packets that can sometimes be observed during chunked-encoded transfers (this was just a missed optimization). 1.4.20 was released with all these changes. Some of them are important enough to justify an upgrade, eventhough they've been here for a very long time.

January 8th, 2012 : Stable 1.4.19

    A few bugs were fixed since 1.4.18, and they impacted users so I wanted to release something now eventhough none of them is critical. First, Sagi Bashari fixed the usage of alternative header name for the "forwardfor" option. An incompatibility between server tracking and slowstart, was diagnosed by Ludovic Levesque : the weight would remain at the lowest level forever. Daniel Rankov reported that option nolinger did not work in backends. It looks like it has been the case for a very long time now. An issue in the string indexing in ebtrees was diagnosed by Julien Thomas. It is used in ACLs could theorically affect the ACL code though it has no visible effect since all patterns in the same ACL are interchangeable. Timothy Garnett reported an issue where Ruby clients were experiencing an extra delay in response time. After analyzing some network traces, it appeared that Ruby likes to send POST requests in multiple incomplete packets, waiting for the first one to be ACKed before pushing the rest, which is incompatible with the delayed ACK. Since we get the incomplete request, we can notice that it's missing data and re-enable quick ACKs to make the client send the rest ASAP. Obvously the client should be fixed as its behaviour makes it very sensible to network latency. Brian Lagoni reported that TProxy broke after Linux 2.6.34 kernel, because the address family was previously assumed to be AF_INET and was not set in HAProxy. Last bug, I was fed up with HAProxy blocking invalid server responses which were sent without headers. I finally understood that it was because some requests were sent with a "\0" in the URI which HAProxy did not block, and Apache considered the request line truncated and ignored the HTTP version, resulting in HTTP/0.9. So the request parser was modified to reject control characters in the URI (the standard forbids other characters but we can't change too much in a stable version without risking breaking some setups). One minor feature was merged. Mark Lamourine worked on a solution to send a server's name in a header when connections are established to a server. I know this can be useful in some silo-like setups and the code does not present any risk of regression so I accepted to include it in 1.4. So 1.4.19 was released with all these changes. If you have no problem with current version, there is no need to upgrade.

September 16th, 2011 : Stable 1.4.18

    The fix for the space parsing in the headers that I made of 1.4.17 was not complete, because it results in negative header lengths being returned for headers that are exclusively composed of spaces. I have checked the whole code to see if this can have any nasty effect, and I couldn't find one, since everytime, we check the length before the contents (we're saved by an optimization). Still, I don't like having dangerous code lie around, especially in stable versions. I know for instance that some people apply custom patches on top of it and may get trapped. So i have issued 1.4.18 with that fix. I also included the recent patch from Finn Arne Gangstad to split domain names on ':' too, as I agree that whenever a port is specified, the host name cannot easily be checked. And I added a match for header length so that it's now a lot easier to check for an empty header. The rest are just usual doc and halog updates. I don't think there is any specific reason to rush on this new version, but if you're in the process of upgrading an older one, please avoid 1.4.17 and use 1.4.18 instead.

September 10th, 2011 : Development 1.5-dev7

    Five months have elapsed since 1.5-dev6. A massive amount of changes was merged since then. Most of them were cleanups and optimizations. A number of changes were dedicated to making listeners more autonomous. The immediate effect is a more robust handling of resource saturation, and the second effect is the removal of the 10-years old maintain_proxies() function which was harming performance and hard to get over. Halog was improved too (faster with more filters). A significant number of external contributions were merged, among them the stats socket updates to clear session-table keys by values. There are too many changes to list, but nothing too dangerous, so I'd say it's the 1.5-dev version I trust the most today. Please give it a test.

September 5th, 2011 : Stable 1.4.17

    Last week an issue was discovered with an application emitting spaces after the content-length value, which caused haproxy to report an error when parsing it. After some checks, it appeared that haproxy ought to ignore these spaces, so this was addressed. It was an opportunity to improve invalid request and responses captures, so that any message rejected for its malformation can be captured. A new minor feature making the X-Forwarded-For header addition conditional was added because users had to resort to complex tricks to do that. Last, halog was updated to latest version. Due to the issue with the header above, I released 1.4.17. Quite frankly most users don't need to upgrade. However it's better to use this one for new deployments.

August 6th, 2011 : Stable 1.3.26 and status updates

    Previous 1.3 version was released 14 months ago, the same day as 1.4.8. Since then, a number of fixes went into 1.4 and a part of them were queued for 1.3. These fixes are not *that much* important but are still worth a release. Thus, both 1.3.26 and 1.3.15.13 were released and are available as source and precompiled binaries for Linux/x86 and Solaris/sparc.

    I realized that I don't use 1.3 anymore myself, and for this reason I'm afraid of the risk of introducing regressions with future backports. So I decided that it was time to turn 1.3 into a "critical fixes only" status after 2.5 years of stable releases and 5 years of existence, meaning that minor fixes will probably never get there anymore, and that future releases, if any, will be focused on important bugs. That does not mean it's not supported anymore, but that fixes will come at a very slow pace and that new deployments are encouraged to use 1.4 if they expect a responsive support.

    I'm also switching the 1.3.14 and 1.2 branches to the "unmaintained" status since nobody appears to be using them anymore (last fixes were more than 2 and 3 years ago respectively).

August 5th, 2011 : Stable 1.4.16

    Since 1.4.15 was released 2 months ago, very few minor bugs were detected. They were so minor that it was worth waiting for other ones to be found, but after some time, there wasn't any point making users wait any more, so I released 1.4.16. A few minor improvements were also made based on feedback from users. Among the changes, MySQL checks now support Mysqld versions after 5.5, health checks support for multi-packet response has been fixed, the HTTP 200 status can be configured for monitor responses, a new http-no-delay option has been added to work around buggy HTTP implementations that assume packet-based transfers, chunked-encoded transfers have been optimised a bit, the stats interface now support URL-encoded forms, and halog correctly handles truncated files. There is no real emergency to upgrade.

June 7th, 2011 : Country IP Blocks needs help

    Quite a few HAProxy users rely on geolocation lists freely provided by Country IP blocks, either to add a request header indicating the origin country, or to select the datacenter closest to the client. Now this nice service needs some money to continue operations otherwise they're forced to close. They're asking for donations. Their service has been offered for free with a high quality to many HAProxy users for some time now, I think it would really be fair that these users in turn help their nice provider. They need $2000 before next week, this certainly is achievable if all big site using their lists donate $100 each to keep them alive. Never forget that for any free software or service on the Net, there are always people working hard to keep the service alive and who have to pay bills at the end of the month.

May 31st, 2011 : HAProxy participates to IPv6 day

    About two weeks ago I registered to participate to the World IPv6 day. The concept is very simple : on June 8th, many web sites will be available both in IPv4 and IPv6. Why only that day ? Because there exists some places where IPv6 can be resolved but not reached, causing the dual-stack sites to be unreachable from these places. By having many sites running IPv6 on the same day, network admins will notice the problem comes from their site and not from the outside since many sites will be unreachable at the same time.

    HAProxy was running dual-stack a few years ago but I had to revert this due to many problem reports. Still some visitors might have noticed the little green image indicating to them if their browser can connect to IPv6. Since I noticed on the participants list that some sites were already running with dual-stack enabled, I decided to enable it here again in advance, so that I'll be able to revert it in case some visitors report any issue. If no issue is reported until June 8th, I'll probably leave that enabled.

    Unfortunately, the Dedibox serving as a cache for the web site is in a network that is not yet IPv6-enabled. That's really a shame, considering that we upgraded it from an old box that was on an IPv6-capable network. I really don't understand what's happening at Free for taking that long a time to enable IPv6 on all their network segments, it does not seem to be on their top priority list. But since the site is running at home behind my Nerim internet access which has been IPv6-enabled for something like 10 years now, I could announce the ADSL endpoint address in the DNS.

    Enabling IPv6 on your web site really is trivial with HAProxy. You just have to add "bind :::80" to your frontend and announce the IPv6 address as an AAAA record in your DNS zone, and that's all. No readdressing, no routing changes, nothing fancy. And you can even get IPv4/IPv6 statistics like here. BTW, I know for sure that some of the World IPv6 Day participants have done exactly that with their HAProxy config too :-)

Apr 8th, 2011 : stable 1.4.15 & 1.5-dev6

    Two annoying bugs were detected on 1.4 at Exosec, one week apart. The first one limits the usable content-length to 32-bit on 32-bit platforms, despite the efforts made in the code to support 64-bit quantities everywhere. It was then fixed in 1.4.14. Yesterday, while working on the backport of 1.4 fixes to 1.3, I spotted that the patch to fix the issues with spaces in cookies that was merged in 1.4.9 introduced a regression due to a typo. In some circumstances, a malformed header sent by the server can crash haproxy when cookie-based persistence is enabled. Thus 1.4.15 was released as an emergency update to address this. The bug has never been reported because it's extremely unlikely to appear, unless a server tries to provoke it on purpose.

    In the mean time, 1.5-dev4 was released with a huge amount of fixes and architectural reorganizations (too many to list here), which were needed to continue the work towards server-side keep-alive. 1.5-dev5 enabled server-side IPv6 support and fixed a number of remaining bugs. 1.5-dev6 was finally released to address the last regressions reported on the list yesterday as well as the important bug above.

    Now, everyone should have understood that all users of 1.4 >= 1.4.9 or 1.5 > 1.5-dev3 must upgrade.

    Please consult the 1.4 CHANGELOG and the 1.5 CHANGELOG for more information.

Mar 9th, 2011 : stable 1.4.13

    Many annoying bugs were discovered both when working on 1.5-dev and by users. Some headers were not correctly processed after removal of the last header (issue reported to the list by Stefan Behte), disabling a disabled proxy from the CLI could result in a segfault (reported by Bryan Talbot, fixed by Cyril Bonté), "balance url_param" was completely broken on POST requests (reported by Bryan Talbot too), it is theorically possible to get HTTP chunk size wrong if only the CR is sent as the last byte of the buffer, waiting for the LF to wrap around in a subsequent packet, ACLs loaded from a file did not correctly close the file descriptor upon success (reported by Bertrand Jacquin), the recently added srv_id ACL could segfault if the server is not known (reported by Hervé Commowick), rlimits were not correctly updated for listening sockets (reported by the loadbalancer.org team), the stats page in admin mode did not support multi-packet requests (fixed by Cyril).

    1.4.12 was released with all those fixes, and Hank A. Paulson reported a crash with pattern files with empty lines caused by a regression introduced into 1.4.11 by a fix for correctly handling empty lines. So 1.4.13 was released a few hours later to avoid any issue.

    I'd like to thank all of these contributors, because well-detailed bug reports are equally important as code contributions. Once again, all users of 1.4 are encouraged to upgrade in order to avoid boring troubleshooting of stupid bugs. This time I have added Sparc builds too, as there are still requests for those. As usual, please check Changelog, with sources and Linux binaries at the usual places.

Feb 10th, 2011 : stable 1.4.11

    While working on keep-alive on 1.5, several issues were discovered, some of which were found to also affect 1.4. For this reason, I preferred to delay the next 1.4 release to until 1.5-dev4 was complete but development has recently stalled, so I preferred to release 1.4.11 anyway. One bug is tagged as critical because it can cause a session to remain indefinitely upon certain conditions that are hard but not impossible to trigger when a server dies. A bug in the ebtree code could cause stick tables to not always match arbitrary length keys. Cyril Bonté has definitely fixed the http-pretend-keepalive option to correctly handle the situations where it was combined with httpclose or in tunnel mode. Until now, it was common to see the client wait for the server to close the connection before returning, causing very bad performance. Since all combinations were extensively tested, I think we should be OK now. Some error conditions were fixed to report correct flags in the logs (eg: client aborts in the middle of HTTP trunking used to report chunking errors). There was an issue with server connection error processing which prevented pending connections from being processed when maxconn was set to 1, because the current connection was counted in them. The error capture from the stats socket is now also able to report incorrectly chunked data. This helps troubleshooting faulty applications. Also the error captures now include an error counter to ease processing with external monitoring scripts. Joe Williams added a global "log-send-hostname" statement which makes it possible to pass the hostname field in the emitted syslog messages. Other various minor improvements on the config parser were merged too.

    With all that, all users of 1.4 are strongly encouraged to upgrade. As usual, please check Changelog, with sources and Linux binaries at the usual places.

Nov 11th, 2010 : devel 1.5-dev3

    Haproxy 1.5-dev3 was released with everything that went into 1.4.9, plus some added bonus that were mainly developped at Exceliance, among which support for binding to UNIX socket on the accept side so that Haproxy can now receive connections over a UNIX socket. This is particularly useful when combined with stunnel, for which a patch is also available. The new PROXY protocol was implemented in order to permit stunnel to forward transport-level information to haproxy, such as the protocol, source and destinations of an incoming connection, so that haproxy can make use of that everywhere internally (acls, logs, transparent, ...) instead of stunnel's address. The main advantage over the x-forwarded-for patch is that it now supports keep-alive and is not limited to HTTP anymore. When combined with the UNIX socket, it can make haproxy and stunnel integrate seamlessly and reliably, provided that this patch is applied to stunnel. Stick tables can now learn from responses, which enables SSL-ID stickiness. And more importantly, stick-tables can now be synchronized in a multi-master fashion between multiple haproxy instances. Also, during soft-restarts, the new process learns the table from the old one so that restarts do not lose that precious information anymore. This tough task was the second half of the large work co-sponsored by Exceliance and LoadBalancer.org.

Oct 29th, 2010 : stable 1.4.9

    Four months after 1.4.8 was released, some minor issues have accumulated and a new release was necessary. It was also an opportunity to add some long-awaited minor feature improvements.

    Among the issues that were fixed, a listener could be left in an unrecoverable state in case of memory shortage during an accept(). POST requests that were followed by a CRLF (forbidden) in a late packet could cause some TCP resets to be emitted on Linux due to those two unread bytes (diagnosed with Dietrich Hasselhorn). Servers that were disabled while processing requests could still drain new requests from the global queue. HTTP header handling for ACLs did not correctly consider quotes and used to consider commas within quotes as a list delimitor. A server with address 0.0.0.0 used to rely on the system to connect to this address (which is always itself). Now it forwards the connection the same way as in the transparent mode. Various error reports and logs were fixed or improved, and many doc typos were fixed.

    Now concerning the improvements, Krzysztof Oledzki improved his netsnmp-perl plugin to support listening sockets, and Mathieu Trudel's Cacti templates were merged. Judd Montgomery and Cyril Bonté's work to support setting servers up/down from the stats interface has been merged too. Gabor Lekeny added LDAPv3 health checks. Hervé Commowick improved the MySQL check to support a complete login sequence with a real username. When option "abortonclose" is set and a client disconnects while waiting for the server, now we forward the close notification to the server. That way the server can decide whether to continue or close. This is important for servers dealing with long polling requests. The Explicit Content Validation (ECV) check code was finally merged after 18 months of reviewd and fixes by various people. That was one major cause for delaying this release. Health checks can now rely on a string that is looked up in server responses. Persistence cookies now support inactivity timeouts and time to live. This is needed with some new terminals such as iPhones where the browser is never closed and the terminal sticks to the same server forever (which is particularly undesired during a partial outage). Also, we now have a new "preserve" option for cookies in "insert" mode, which indicate that if the server sets the cookie, then we let it pass unaffected. This allows servers to terminate persistence upon logout. Last, the "halog" utility was improved to support per-url and per-termination code statistics. This means that it's now trivial to know what URLs take the most processing time.

    Version 1.4.9 was released with all that, with sources and Linux binaries at the usual places. Some of these fixes will slip into 1.3 too.

Oct 23th, 2010 : new httperf results : 572000 reqs/s

    This morning I came across this interesting post from Kristian Lyngstol about the performance tests he ran on the Varnish cache. What struck me was the number of requests per second Kristian managed to reach : 275000, not less. I'm not surprized at all that Varnish can withstand such high rates, it's known for being very fast. My surprize came from the fact that Kristian managed to find fast enough tools to run this test. My old injector is limited to around 100k requests per second on my machines, as it does not support keep-alive, and Apache's ab to around 150k with keep-alive enabled. And when I managed to reach 2 millions requests per second, I was feeding a constant stream of pipelined requests with netcat, which is particularly inconvenient to use.

    Kristian said he used httperf. I tried it in the past but did not manage to get good numbers out of it. He said he found some "httperf secrets", so that made me want to try again. First tests were limited to approximately 50000 requests per second with httperf at 100% CPU. Something close to my memories. But reading the man, I found that httperf can work in a session-based mode with the "--wsess" parameter, where it also support HTTP pipelining. Hmmm nice, we'll be less sensible to packet round-trips :-) So I tried again with haproxy simply doing redirects. Performance was still limited to 50000 requests per second.

    In fact, there appears to be a default limit of 50000 requests per second when "--rate" is not specified. I set it to 1 million and ran the test again. Result: about 158000 requests per second at 100% CPU and with haproxy at 44%. Since my machine is a Core2 Quad at 3 GHz, I fired 3 httperf against one haproxy process. The load reached a max of 572000 requests/s with an average around 450000 requests per second. This time, haproxy and all 3 httperf were using 100% CPU. What an improvement!

    These tests mean nothing at all for real world uses of course, because when you have many clients, they won't send you massive amounts of pipelined requests. However it's very nice to be able to stress-test the HTTP engine for regression testing. And this will be an invaluable measurement tool to test the end-to-end keep-alive when it's finished. I still have to figure out the meaning of some options and how to make the process less verbose. Right now it fills a screen with many zeroes, making it hard to extract the useful numbers. I'm grateful to Kristian to have made me revisit httperf !

Aug 26th, 2010 : devel 1.5-dev1 with many goodies

    Three months ago I was approached by the Stack Overflow Team team who needed to get some improvements in HAProxy. Overall, their needs would have been addressed by the final release of version 1.5 scheduled around the end of the year, but having to wait that long was not practical due to some architectural constraints imposed by an intermediate solution. They proposed that we find an agreement on which we could work together. Since we were already having productive exchanges for some time, and I knew they were good guys (after all they already donated to the project last year), I accepted the deal.

    Also, I must say that as a software engineer, it's always a lot better to have someone explain their needs with high expectations than having to guess how a feature will be used.

    Geoff Dalgas and Jeff Atwood described to me in great details what they needed to do : perform request throttling per IP address, possibly based on various criteria, in order to limit risks of service abuse. That was very interesting, because that feature was being thought about for about 4 years without enough time to completely develop it, and also the new stickiness framework that was contributed by Exceliance and Loadbalancer.org was making that really possible, although an important design rework had to be operated first within the code.

    During the tests with Geoff and Kyle Brandt, it appeared that some more changes had to be operated to be able to store any criteria in the tables (eg: bandwidth per IP address), and to be able to consult and change the table contents at runtime, leading to a more and more generic code. Kyle has been very patient and comprehensive, I think I have changed the mechanisms and configuration syntax at least 5 or 6 times during the tests, but he always took the time to understand the changes and adapt his configurations. If I had been at his place, I would have got bored earlier, so I owe him a big thanks for his patience !

    Now the code has been running fine in production overthere, so it's time to release it and merge it into the master branch. I won't extend further on how it works, since Kyle has put an excellent explanation on his blog that is a lot more clear than the doc (that reminds me that the architecture guide really needs some lifting).

    I'll add quick status on the current code. Some core changes that I wanted to do earlier will now start. But that means that 1.5-dev1 should be as stable as 1.4.8. I'm not saying that I would suggest to anyone to push it into production, but it can clearly be used to mitigate DDoS attacks as well as stop service abuses. I could get it to stop connection floods slightly above 200000 connections per second (yes, two hundreds thousands) and let the good traffic pass through. So for this reason, I think that people who are regularly exposed to such trouble may find it useful to keep it handy.

    Now what's next ? Right now the data in the tables is local to one process, so it is not shared if you start multiple processes, nor it is across reloads. The second step of the stickiness extensions developped by Exceliance and Loadbalancer.org will include stickiness table synchronization between multiple hosts. Some work will still be needed to be able to share counters, but since this development is done in a flexible way, it should not be too hard to adapt it later. BTW, I also owe a big kudos to the Git versionning system, which has made it very easy to rework my patches after every change and bugfix until they were looking good, through massive abuse of branching and rebasing.

    Too much talk. The code is available here, the ChangeLog is here and the doc is here. Since this is a development version, no binary is provided.

    The last words naturally go to the really cool guys at Stack Overflow. It's very nice to see some sites and companies involve time and money and take risks to make Open Source products better. Of course they benefit from this work, but at no point during the whole development did they try to reduce the focus to their specific needs, quite the opposite. From the very first exchanges, their goal clearly was to make the product better, and that must be outlined. That's now achieved and I really appreciate their involvement. Thank you guys !

June 16th, 2010 : stable 1.4.8 and 1.3.25

    Today, Ben Congleton, Morten Gade Sřrensen and Hervé Commowick reported and diagnosed two bugs in 1.4.7. One was a regression introduced in 1.4.7, breaking the stick-table feature due to a side effect of the fix for a memory leak in it. The second one is quite old, it dates 1.3.16 (2 years old) ! It causes crashes soon after a connection has matched a monitor-net in a tcp-only instance. I wanted to quickly fix both issues so that users who have not yet upgraded to 1.4.7 don't waste their time, and I'm very grateful to the 3 guys above who were extremely reactive and allowed the bugs to be fixed in a few hours. Finally, Patrick Mézard's doc updates were merged too.

    Version 1.4.8 was released, with sources, Linux and Solaris binaries at the usual places. Version 1.3.25 was released too with a few other pending fixes, and sources and Linux/Solaris binaries at the usual places too.

    Users of 1.3 after 1.3.16, or 1.4 with TCP only frontends will likely want to upgrade, eventhough if they have not been hit yet, they will probably never be. Users of stick-tables will have to upgrade too. The fact that we're now spotting bugs that have been there for two years is an encouraging sign of stability.

June 10th, 2010 : stable 1.4.7

    Jeff Persch debugged the trouble affecting consistent hashing about two weeks ago. I wanted to issue 1.4.7 right after that fix got merged but I was lucky enough to spot a few minor bugs in the following days, so that finally got delayed. Most of these issues are really minor and were found while reviewing the code in preparation of 1.5-dev. Some stats were not accounted properly (failed req instead of denied req), some termination flags in the logs could be wrong, which is a shame because we use them for debugging, a TCP to HTTP transition was not properly handled, and the dispatch and http_proxy modes were broken since 1.4-dev, but apparently nobody uses them anymore.

    Some people will like the new halog, as it is able to report per-server stats on status codes, error ratios, average connection times and response times, which is very handy for quick checking of your prod's health.

    The complete ChangeLog for version 1.4.7 is here and the sources are here.

    On a side note, someone asked on the list how haproxy would perform on a Guruplug Server Plus. I ordered one a few months ago and finally received it. Well, it's a disaster, not only it is slower than the Geode, but it heats so much that metal parts hurt to the touch and power supplies fry after a few months. Definitely not something to buy. Check the full review on previous link if you're interested.

May 16th, 2010 : stable 1.4.6

    Version 1.4.5 triggered a conflict in a macro name with some Linux distros shipping glibc >= 2.10. Since this affects most recent distros, it was better to upgrade now. A few points about RDP cookies were clarified in the doc. Last, a new ACL match srv_is_up() has been added, to consider a specific server's state in ACLs. Those who had no problem building 1.4.5 don't really need to upgrade. mixing them with fixes.

May 13th, 2010 : stable 1.4.5

    No reliability issue was reported since 1.4.4 was released. This is a very good thing, because some people were asking for a few minor features, so it was a good opportunity to get them merged without mixing them with fixes. The ChangeLog for version 1.4.5 is here and the updates are here.

    First, Cyril Bonté provided the new ignore-persist directive. it allows haproxy to ignore the persistence cookie on some requests which validate an ACL-based condition. It is particularly suited to optimise the load balancing of static or stateless objects in the middle of a stateful farm.

    Second, it was planned 3 years ago to be able to feed ACLs with large data sets loaded from files, but it was still not implemented due to the lack of precise needs. Now, 3 years later, more and more people are reporting difficulties writing large configurations, and the last config I saw which was 104000 lines long convinced me that it was urgent to support this feature. But matching requests against very large datasets can be CPU intensive, so I have extended my Elastic Binary Trees to support new lookup methods and now it is possible to lookup a string or an IP address among tens of thousands in a few tens of nanoseconds. This means that it is now possible to use haproxy to perform geolocation. For instance, checking that a source address belongs to one of the 38400 european networks only consumes 2% CPU at 40000 requests per second.

    The rest are just minor improvements. Tt's now possible to stick on an IP address extracted from an HTTP header, and I improved a bit more the halog analyser, which is now possible to report request counts by status codes. It also gained some nice performance boost as it can now parse about 1.3 Gigabytes of logs per second on a 3 GHz Core2.

    I expect that this version will take some time to spread because it only contains new features and will likely not be backported to various distros. Still, some power users will probably interested in giving it a try.

April 7th, 2010 : stable 1.4.4

    Some people were experiencing optimisation issues with Tomcat and Jetty, with which it was not possible to perform client side keep-alive when the server received a "Connection: close" header. This is due to a strange design choice by which they decide the client is not interested in the response length if it intends to close after the transfer! Well technically that works... most often... Sometimes users may get truncated objects without being aware of that. Anyway, Cyril Bonté had a very smart idea for a workaround : pretend to the server we'll maintain its session alive while it's false. This fixed the problem, and is now available by adding option http-pretend-keepalive to option http-server-close.

    Jetty's HTTP implementation seems to be the flakiest though. It even manages to send "HTTP/1.1 100 Continue" intermediate responses when the client sends "Expect: 100-continue", but it closes the connection just after that message, resulting in a 502 error for the client.

    Cyril also fixed an issue with appsession where a cookie whose name begins like the appsession cookie could be mistaken for it. Those issues were enough to justify a new release.

    Very few other minor fixes were brought there, and a minor feature was added. It consists in being able to bind to a source address found in a header when connecting to a server. Normally this will be the X-Forwarded-For header. This requires use of the Linux kernel TPROXY patches, and makes it possible for backend servers to see the initial client's IP even when several layers of proxies have been passed through.

March 30th, 2010 : stable 1.4.3 and 1.3.24

    A few remaining issues and one regression were fixed in 1.4. In 1.3, I have backported all pending fixes since 1.3.23, most of which were discovered in 1.4. The most interested people will probably be users of FreeBSD where they could randomly get a SIGPIPE if the program was compiled on a very recent version (8, maybe 7.2 too). Anyway, due to other minor to medium fixes, 1.3 users should upgrade.
    It's worth noting that very few issues were reported on 1.4.2. The code has stabilized very quickly, and people in sensible environments will be able to start to think about evaluating it to plan an upgrade (from most reports, the new features such as client keep-alive and improved stats are very much demanded). If I could send them an advice, I'd say that we're going to release 1.4.4 soon with a few minor improvements and that if some people don't know what version they will start with, then let's start with 1.4.4 when it's available.

March 17th, 2010 : stable 1.4.2

    A new batch of annoying issues got fixed. Among these issues, we can find a risk of crash (broken pipe) on very recent versions of FreeBSD, a segfault when using CLF logs and capturing a non existant header, a very rare case of stuck client session when using keep-alive, some garbage which appeared on the stats page after 1000 client resets, or when the host name was too long, a url_param hash bug which could result in a dead server to be used in very rare situations, a risk of getting an empty result on the stats socket if the input closes before haproxy responds, and endless loop at config parsing time on the error-limit keyword, status codes 501 and 505 which could cause a server to be marked down if on-error was used, and a risk of getting truncated HTTP responses when chunk-encoding was used with chunk sizes that are exact divisors of the buffer size, and an issue with anonymous ACLs which were merged together into a single one. A few improvements were made to the health checks which now support multi-packet responses, and the stats socket can now display more debug information about a specific connection. The list is quite huge, it was important to release 1.4.2 without waiting for any possible new issue to come, eventhough things seem to calm down.

March 5th, 2010 : stable 1.4.1

    Some build issues on non-Linux platforms were preventing new 1.4 adopters from trying it. These issues are now fixed. Other issues concerned the appearance of more 502 errors in the logs than with 1.3. This was a bug that caused the status code to be changed to 502 even in case of connection abort during the data transfer. A few new error counters were added to the stats, and other minor issues were fixed. This new version now builds and works on FreeBSD, OpenBSD, OSX, Solaris, AIX and Linux, so let's not wait and release 1.4.1.

    Also, Solaris users will now be happy, I unpacked and replugged my Ultra5 so the Sparc binary is available again.

    On a side note, I have removed the link to the haproxy.org mirror because it has been outdated for the last 6 months and even remained 1 week on an expired DNS zone. I failed several times to contact Kevin Kuang there, so I don't even know who manages it now if any. If someone gets in touch with him, please ask him to contact me.

February 26th, 2010 : New stable branch 1.4 !

    After 11 months of active development and a lot of external contributions, version 1.4 is now released. It has been tested for the last 3 weeks by many people, and only very minor bugs were reported (and fixed), so it's now time to officially stamp it as stable. Version 1.4 brings a lot of new features, among which the long-awaited support for client-side HTTP keep-alive, the RDP protocol, and stickiness on anything, as well as many other nice usability improvements on the stats interface, checks and the CLI. It is also much more powerful than version 1.3 and will support addition of new protocols faster than before due to a better structured internal architecture. 1.3 will still be supported for some time, and the old 1.3.15.X branch is now entering a deep freeze where only critical bugs will be fixed. Please consult the ChangeLog for more information about all the changes. I particularly want to thank all the persons and companies who contributed to this version by code, testing or development funding ; without their efforts and participation, we would still be far away from a release !

January 28th, 2010 : stable 1.3.23

    Several minor bugs were fixed since 1.3.22, and the request-learn, force-persist and http-check send-state were backported because people need them for more transparent and reliable application updates. Since no new bug was in sight, and 3.5 months had elapsed since 1.3.22, it was the moment to release 1.3.23 with all that. As 1.4 approaches, 1.3.23 will probably be the last 1.3 which accepts new minor features. Future 1.3 versions will very likely be dedicated to bug fixes only.

January 25th, 2010 : 1.4-dev7 & 1.4-dev8

    While trying to work on end-to-end keep-alive, I encountered issues that needed to be fixed, so this has delayed dev7 quite a lot, and it does still not have this end-to-end keepalive. Think of it as a much cleaner dev6 instead since many bugs were fixed. The stickiness code sponsored by Exceliance and Loadbalancer.org got merged. Currently, it can almost only learn IP addresses, but it has been designed with an amazing flexibility so that it will be very easy to add stickiness on any request or response criteria. MySQL checks have been introduced and this code will evolve for slightly deeper and more reliable checks. A new "force-persist" statement allows admins to test their servers without opening them to the world, which is very convenient to ensure they're correctly installed and that their customers will not face a lot of crap. And as always, a bunch of bugs in many areas were fixed. Update: 1.4-dev5 to 1.4-dev7 had a nasty bug with keep-alive enabled, so please update to 1.4-dev8.

January 16th, 2010 : 4-hour network outage

    Some of you have noticed that the site was down from 11:45 to 15:45 local time, and it can be seen here. It was the longest outage I ever experienced in 8 years with my ISP (Nerim). The support told me the outage was at their ADSL provider, SFR. Well, 4 hours in 8 years is still 99.995% availability, I have nothing to complain about :-)

January 8th, 2010 : 1.4-dev6

    As could be expected, 1.4-dev5 did not work very well. The rule is pretty clear : if you don't like your code, it will fail. Just reread the last post and you'll see that it was destined to fail. With the nice help of Cyril Bonté and Hank A. Paulson, we could spot a lot of bugs and I finally got rid of those parts I found ugly. Now curiously, it works a lot better :-) Also, Krzysztof Oledzki contributed a nice feature he talked about some time ago : the default-server setting. This makes it possible to specify some common settings globally and not have to repeat them for all servers. This is useful for check intervals, maxconn, etc.. So it was time to release 1.4-dev6 so that all those who had a bad experience with 1.4-dev5 can try again. This is the version currently running on the site, so it looks fine :-)

January 2nd, 2010 : New year, new features

    After several weeks of work, I have committed the patch which introduces client-side HTTP keep-alive and pipelining support. The code is quite ugly and I'm not proud of it. This is because I got quite a bunch of last-minute surprises that I will have to workaround in a cleaner way. But since the code worked, I would have found it wasted to make you wait for it.
    In order to enable pipelining on the client side, just comment out any "option httpclose" statement in the defaults, frontend and backend sections and set "option http-server-close" in any of them. As the name implies it, the connection is still closed on the server side. This way we can still have low ressource usage on servers and correctly enforce maxconn while retaining keep-alive with the client.
    This code will be in 1.4-dev5 by the end of the week-end, but the impatient will be able to download a snapshot for their tests.
    The new code has been put in test on the Formilux server and already shows decent load time savings on this page. Stay tuned...

October 18th, 2009

    I have put online a matrix of all known bugs which affect stable versions 1.3 starting with 1.3.14. It took about 4 hours of work to put that in shape but I think it was worth it. Let's put it short : those of you running 1.3.15.2, 1.3.16 or 1.3.17 are doomed. Those running 1.3.15.X before 1.3.15.7, 1.3.19 or 1.3.21 are at risk. 1.3.14.14, 1.3.15.10 and 1.3.20 are pretty good, and 1.3.22 is the only one with no known bug yet.

October 14th, 2009 : 1.3.22

    A few hours after 1.3.21 was issued, John Lauro reported an important regression causing a crash when connecting to the stats socket. This was caused by a minor backport which should have been modified for 1.3 and that I didn't detect during the tests because I did not use this socket. 1.3.22 was released to fix this issue. Please don't use 1.3.21 and update to 1.3.22!

October 12th, 2009 : 1.4-dev4, 1.3.21

    A lot of changes have occured in only 3 weeks, so it was the right moment to release a new development version. It's worth noting that Krzysztof Oledzki has been very active, contributing no less than 1/3 of all the changes. This is nice because being two to work on the project, we progress faster. Concerning the changes, the stats interface (socket and page) certainly is the most affected area. It is now possible to reset counters and to change a server's weight without restarting... two features that many of you have been asking for years! The stats page now can also display a node name and description, as well as the exact status of a health check. The LB algorithms have now been moved to separate files, and a consistent hashing algorithm has been added. It allows hot addition or removal of servers without disturbing the load-balancing, which is desirable for caches. Also, the LB rework brought the opportunity to re-enable the old static round-robin algorithm, which can make sense for people who run more than 4000 servers in a single backend (practical limit of the dynamic RR algorithm). Last, some new ACLs have been added, to check for IP addresses in headers, and to check frontend's and backend's connections, queues and per-server average queue size. A few minor bugs were fixed, and those fixes as well as some minor riskless features have been backported into 1.3 to release 1.3.21.

September 26th, 2009

    I found it was important to acknowledge some people and companies' efforts for the project. So I added a new page listing significant contributions, most often features but sometimes fixes, in the form of patches, code, time or even money. A minor bug on the stats page which remained in 1.4-dev3 has also been fixed and is available in the latest snapshot.

September 24th, 2009 : 1.4-dev3 + sponsors

    Most of the internal changes planned for version 1.4 have been completed, so it was time to release a new clean snapshot. The architecture is now ready to permit keep-alive, SSL or FastCGI developments. Some more changes are planned but the remaining ones should be a lot easier to perform without breaking everything.

    Compared to latest stable 1.3.20 version, 1.4-dev3 provides new features, among which support for the CLF log format, RDP protocol load-balancing and persistence, a new interactive CLI, an improved HTML stats page, support for inspecting HTTP contents in TCP frontends and switching to HTTP backends (allowing HTTP+SSL to coexist on the same port), support for forcing of the TCP MSS on frontends, smart network optimizations to reduce the number of TCP packets in a session, runtime-configurable buffer sizes, support for more than 64k concurrent connections, config parser support for "no option xxx" to disable options that were enabled by default, and correct 1xx status code processing. Developments to support keep-alive have already started, and if time permits, SSL integration will be attempted. The code looks amazingly stable for the amount of changes, and will probably not change much anymore, so any bug found in this version must be reported and fixed. Also, new feature submissions should be based on this version. It will be easier to implement for submitters and for me to merge.

    Several large sites are already running on 1.4-dev2 with great success. This one should be even better, but given the number of changes, it should be monitored more closely at the beginning.

    Last, I have a very good news that I hope will give ideas to others : Exceliance and Loadbalancer.org have both agreed to contribute some manpower and money to implement the complete persistence framework that everyone is dreaming about into haproxy. That's a tough work and I'm not certain it will be ready for 1.4 (though it might, depending if I'm as late on my releases as usual). I would personally like to thank them both for their contributions. When you have to put your money in commercial solutions, please never forget to consult first the guys who involve time and money in opensource projects, because they are the ones who help the projects evolve and live !

August 9th, 2009 : 1.3.20

    Cristian Ditoiu from transfer.ro reported a major regression when testing 1.3.19. It would crash within a few minutes while 1.3.15.10 was OK. He offered to help so we could run gdb and debug the crash live. We finally found that the crash was the result of a regression introduced by a recent in 1.3.19. I really want to thank him because he spontaneously provided a lot of help and trust to debug this issue which at first glance looked impossible after reading the code and traces, but took less than an hour to spot and fix when caught live in gdb ! It's always pleasant when users show that level of involvement to chase bugs.

    Another bug was reported by Romuald du Song, who found that option tcplog would log using global parameters if no logger was defined. It can be either helpful or annoying. This is now fixed and a warning is emitted when such a configuration is encountered, so that people running off erroneous configs can easily fix them.

    This time I expect 1.3.20 to be the good one. It's always a good sign when we fix minor bugs or recent regressions introduced by bug fixes. 1.4-dev2 has also been released to help people track changes in the two versions in parallel.

July 27th, 2009 : 1.3.19

    Since 1.3.18 was released two months ago, it has been widely deployed, in part thanks to the slowloris tool which has caused HAProxy downloads to jump by 20-30%. This results in more exposure and new kinds of bugs to be discovered. The most annoying ones concern too short sessions which may sometimes be reported as server errors, random delays under low traffic conditions due to a scheduler bug, and the last one reported today by an Exceliance customer who was kind enough to provide lots of traces, some occasional pauses on interactive TCP traffic which might also happen on the last chunk of an HTTP response, although extremely rarely. Each one alone would have been enough to issue a new maintenance release, so here it is, 1.3.19. It also brings a small bunch of nice new features backported from the dev tree, among which the support for multiple configuration files, the support for more than 64k concurrent connections (tested at 190k by the heavy user), and a highly better reporting of config warnings and errors. So, as usual with maintenance releases... everyone is highly encouraged to upgrade.

    Since some of the bugs above were present in earlier versions, a new release was emitted for 1.3.15 and 1.3.14 too for the late users who have not upgraded yet. I really think it's the last one for 1.3.14. 1.3.15 might still see another one till the end of the year, and that will probably be all for this one after 20 months of free support :-)

    The first development version should be released too, but I need to update my release scripts first, they are inadequate and take me too much time to use, so stay tuned !

June 27th, 2009 : HAProxy to counter DoS attacks

    Since the announcement of the Slowloris tool, people seem to be discovering how fragile a default Apache setup can be ! Well, this is not news, as people who install Apache on high-traffic sites have been aware of this weakness for ages, and have been setting very low timeouts and disabling keep-alive in order to mitigate risks. Now that a tool is publicly advertised, I'm beginning to hear questions from worried site admins about what to do if their site is attacked. Also, we're seeing several sites and forums suggesting installing HAProxy in front of Apache servers to protect them (note that Nginx would probably do equally well).

    Indeed, HAProxy does not need a new thread nor process to accept a new connection, it only needs some RAM (16-32 kB per connection). Some people are already using it past 70000 concurrent connections, which cannot be achieved on Apache which needs an expensive thread or process per connection. More specifically, HAProxy will only forward complete and valid requests. This means that it will not bother Apache while the attacker is playing with its few thousands connections, and all valid requests will immediately pass through. And the icing on the cake is that HAProxy can kill requests which take too much time to complete, using timeout http-request (more than a few seconds is not to be considered normal).

    Once again, we observe a derivate use of a load-balancer, which is a bit expected : when a tool is designed to accept 10 times more load than the servers it feeds, there is nothing surprizing that it can be used to protect them ! Let's see if Apache evolves towards providing more tunables to mitigate such attacks... In the mean time, a drop-in anti-DoS configuration is available here.

May 10th, 2009 : 1.3.18

    Yan Qiao of Rocket Fuel Inc reported a crash on x86_64, which was pretty much unexpected ! He nicely offered to help troubleshooting by rebuilding with debugging on and leaving the process running in production to catch the error, then sent me an interesting core 1 week later, which revealed that a field in the struct session which was never touched had been changed due to the sharing of two pools of the same size. This field should have been initialized but was unfortunately not. The issue can only happen on x86_64 with HTTP logging enabled, due to the exact 1024 bytes of the struct session which allows its pool to be shared with the struct requri's. Thank you guys for your huge help and the risks you have taken leaving that process running!

    During a troubleshooting session with the T20 guys (Maxim Fedchishin, Jason Coward and Viktor Brilon from modX team, Hans from RightScale team), I came across an old leftover process doing nothing after a soft-reload. That issue is brought once in a while by various people, but it happens too rarely for anyone to get an opportunity to debug it. The guys accepted that I installed a debugger on their machine to see what the process was doing. It was deadlocked in free() during the reload. And that made sense : during a reload, the old process releases as much memory as possible to leave room for the new one. If the two signals sent by the second one are too close to each other, the second signal is sent while the first one has not completed releasing memory and we can have a recursion in the libc's free(), causing a deadlock. That has been fixed by implementing asynchronous signal delivery. Thank you guys for giving me the opportunity to catch that rare event!

    Problems aside, a few minor features were merged. The stats are now more readable, report max session rates and provide full 64-bit counters everywhere. It is now possible to forward invalid requests or responses without blocking them, but they will still be captured. The config parser now warns about possibly unwanted ordering of ACLs or reqxxx/rspxxx. Several wrong printf() format strings have been fixed. The build process now supports an alternative architecture, and the RPM spec file has been cleaned. A new balance hdr(header) algorithm has been added to balance depending on a header hash. A new option enables addition of the destination IP address in the X-Original-To header. And last but not least, the doc has been massively cleaned up and reorganised.

    With all these fixes, I released 1.3.18, as well as 1.3.15.9 and 1.3.14.13 which are probably among the last ones of their respective branches after 12 and 18 months of maintenance.

April 19th, 2009 : new performance record broken !

    It was a long time since my last 10 Gigabit tests, exactly one year. The Linux kernel has evolved a lot, so did HAProxy and even the Myri10GE driver. I knew we could get much throughput since we fixed the kernel splice() syscall. It was a good opportunity to start a new series of benchmarks again. In short, new records were broken. Full 10Gig line rate with 20% CPU, and the 100000 sessions/s barrier was crossed !

March 29th, 2009 : 1.3.17

    Bart Bobrowski of who's.amung.us reported abnormal CPU usage with the new version 1.3.16. After a full day of tests and code analysis, I failed to reproduce the issue here, and the bug appeared impossible to me. Bart then offered a lot of help with testing many patches, providing hundreds of megs of traces, so that I could finally fix the issue caused by a nasty race condition. I really appreciate it when users with extreme loads accept to take traces in production, with all the risks that this practise implies. Sometimes it's the only way to get a bug fixed.Thanks Bart!.
    Since other minor fixes and enhancements were pending, I released 1.3.17, which users of 1.3.16 are invited to upgrade to.

March 22th, 2009 : 1.3.16 !

    Minor fixes and enhancements have been added since the second release candidate. So, that's it, 1.3.16 is out and marks the new official stable release. As it has already received long testing from major users, I'm not worried about its stability, eventhough I expect that a few bugs will surface. Further development will continue in a new branch, and 1.3 will only receive fixes and minor enhancements.

March 10th, 2009 : 1.3.16 is getting closer !

    Second release candidate of 1.3.16 has been published. It brings a lot of new long-awaited features, among which TCP splicing support, conditional redirection, TCP content filtering, session rate reporting and limiting, invalid request/response capture, binding to specific network interfaces, per-process affinity for frontends and backends, a monotonic internal clock, and many other features.
    The internals have finally been reworked in layers so that forwarding can be processed without waking high levels up. HTTP is now on top of TCP and not a special case of it. A big advantage of these changes is that we can now touch the socket code without impacting HTTP and vice-versa, which had not yet been possible till there. This means that the risk of future regressions caused by feature additions will be significantly lowered. Thanks to these changes, a lot of complex tricks and specific cases are now handled more cleanly and in a more evolutive manner. New work on keep-alive, SSL integration and QoS will be easier.
    Once 1.3.16 is out, branch 1.3 will become the new stable branch, and support for 1.2 as well as 1.3.14 and 1.3.15 will slowly phase out.

March 9th, 2009

    Several minor bug fixes were pending since 1.3.15.7, so it was time to release 1.3.15.8 and 1.3.14.12. Those bugs are not stability bugs, rather load-time bugs (config parsing, etc...). Only one of them really justifies updating : if your configuration uses the "track" keyword in order to synchronize multiple servers states, the time taken to synchronize them grows with the number of servers. Among the changes, a backport of the doc updates was merged, covering the log format, so that the old docs should normally not be needed anymore.

December 4th, 2008

    Kai Krueger reported a nasty problem he encountered and analyzed. When a server goes down, it requeues all of its connections waiting in the queue into the global queue. But when a session completes after that, haproxy checks if there are pending connections that this server can handle, without taking into account the fact that the server is dead. So the server can progressively suck all pending connections from the global queue just after it has been marked down. Yes, I know, this is stupid. A check has been added so that it does not dequeue global connections when it's marked down, and releases 1.3.15.7 and 1.3.14.11 have been issued. There are very few setups which will trigger this problem, however it's quite annoying for those experiencing it.

October 12th, 2008

    Once in a while, a user reports that some old processes remain present after a soft-restart. I could never reproduce the issue until Manuel Soto sent me a truss output of a configuration with which the problem reproduces frequently. The cause is finally that haproxy still binds listening addresses to disabled instances, but does not try to stop them and refuses to exit as long as they remain present. I took the opportunity to fix a related problem causing warnings to be emitted when haproxy tried to stop backends, and a segfault in the configuration parser if ACLs were declared in a defaults section.

    That was enough to release 1.3.15.5 and 1.3.14.9. I recommend that any user of 1.3.14 or 1.3.15 upgrades, as these fixes present very minor risk and fix really annoying problems.

September 14th, 2008

    Several users reported on the mailing list that they were experiencing abnormal concurrent connection counts higher than the maxconn they configured. They were very prompt to send me configurations and screenshots of the stats report showing the problem. It was indeed a bug triggered every time a connection attempt to a server failed. I've fixed it along with another minor one, and released 1.3.15.4 and 1.3.14.8. Mongrel users are particularly exposed because they run with maxconn=1 and the server cannot accept more connections, so users may experience occasional errors when a server starts to reject connections. It would also be interesting to find why some connections fail to the servers.

September 3rd, 2008

    A cool video demonstration of the connection regulation mechanism (maxconn) has been posted on 37signals. It's clearly explained and explicit enough for people not much aware of the mechanism to understand it. Check it there, it's not too long and really worth seeing.

September 2nd, 2008

    While working on haproxy 1.3.16, I came across a few bugs in the code, so I issued 1.3.15.3 and 1.3.14.7. The only one annoying concerns 1.3.15 for people who use the "balance url_param ... check_post" construct to hash on parameters present in POST requests. There is a risk of crashing (but no server compromission though) with some invalid requests. Fortunately, this feature is very new and ver limited to niche users, but it needed a quick fix anyway. Other bugs are pretty minor and most of them concern small issues with how timeouts are handled.

July 20th, 2008 : two lines...

    Two lines... That's all what is needed with the new TCP content inspection system to stop half of the spams I got home. One of my major customers who uses HAProxy a lot has sponsored the development of some preliminary content inspection which is used to decide whether to forward a connection or not. The very first usage of this feature consists in checking that only SSL is spoken on a connection. But most likely more protocols will come soon. As a nice side effect, I could now add a delay before the HELO message of my SMTP server, and reject all robots which talk first (forbidden). And since many spam bots have small timeout values, many of them abort before the timeout is reached, resulting in my incoming spam rate dropping from about 300/hour to "only" 150/hour. Those who keep up with the time out slow down due to limited resources. The small addition simply consists in adding those two lines in the frontend :

    		tcp-request inspect-delay  35s
    		tcp-request content reject if REQ_CONTENT
    	  

June 21th, 2008

    haproxy versions 1.3.15.2 and 1.3.14.6 have been released to fix a major bug in request queue handling. The problem was that due to a design problem, it was possible for new requests to be immediately served by a server before a request in queue would be served. That caused some requests to remain in queue until they reached the queue timeout, after which either they would eventually be served, or return a 503 error code to the client.

    Since it was a design problem, it took a lot of time analyzing the root cause and finding a solution. However, as a positive side effect, the fix now makes the redispatch option work for requests which overflow a queue. That way, clients do not get a 503 error anymore but can be served by another server (which was the purpose of the redispatch option.

    Note that it is possible that 1.2 is also affected by the issue since some parts of the faulty code have not changed since. But it is very hard to determine if it is faulty or not, and backporting the fix would take even more time. Maybe I will eventually take a look at it if people complain about the issue.

    Update (2008/06/28): Alexander Staubo, who first noticed the problem, has run a new series of tests showing that the problem is definitely fixed. It also demonstrates the very nice positive effect of running with maxconn 1 with Rails servers.

May 25th, 2008

    Released haproxy versions 1.3.15.1 and 1.3.14.5 with minor fixes : build fix for GCC 4.3, fix for early truncate of stats output in certain circumstances, and better handling of large amounts of highly active sockets. I indeed discovered during testing that the sepoll poller was so much efficient that when running at gigabit speed with 80000 active sockets fighting for their CPU share, almost all of them were running in speculative mode, causing starvation of the remaining ones, which in turn caused the accept() call to be very rarely called. Delays of about 40 seconds have been observed on a 3.4 GHz Pentium 4 to get the stats page under such a load. The other pollers were not better BTW. The fix consisted in ensuring that polled events are checked at much often as the speculative ones. With this fix, the stats page responds in less than one second on such a saturated machine. There is still room for improvement relying on events prioritization though. Version 1.3.15 has been promoted as the recommended one since there has been no regression report. Version 1.2.18 was also released for users of 1.2 which experienced trouble building on BSD.

April 19th, 2008

    Released haproxy version 1.3.15 with many new features. The most important changes are stats updates (HTTP and UNIX), enhancements of server checks such as tracking and dynamic intervals, addition of the leastconn load-balancing algorithm, a fully transparent mode on Linux, better handling of connection failures (dead server avoidance and turn-around state), support for inter-site off-loading through redirects, updates to the build process, and large documentation updates. For more information, please check the ChangeLog. Due to the important number of changes, upgrade from earlier versions should be performed with a bit of care.

    Once again, a lot of code comes from contributions. I'd like to specially thank Krzysztof Oledzki for a lot of useful contributions, including the SNMP agent, and the guys at Nokia for the good work they have done on POST parameter hashing.

March 30th, 2008

    I finally assembled my new machines and installed the donated 10-Gig Myricom NICs. I ran a few benchmarks. Result: new bandwidth records set for HAPoxy: 9.897 Gbps and 35128 hits/s! It's possibly the highest bitrate achieved to date with an opensource load-balancer! BTW, even most commercial ones are commonly limited to 4 Gbps by hardware design. What's a bit frustrating for a precision-tweaker like me is that those NICs work out-of-the box on dirt cheap hardware, there's almost no joy passing beyond the first 4 Gbps :-)

March 8th, 2008

    Released haproxy maintenance version 1.3.14.3 to address several minor bugs and clean up the configuration manual a little bit. One annoying issue with backup servers in round robin mode was fixed. Nothing really important was changed in this version, this makes it a good candidate for distro updates.

February 23rd, 2008

    I finally decided to buy an expensive motherboard to upgrade my PC in order to begin testing with the 10Gig NICs. I selected an Asus P5E3-WS Pro because I needed PCI-X slots for my older cards. I've put a Core2 Duo E8200 (45nm) because I wanted a lot of silence. The mobo has 2 PCI-E 16x slots, which made it possible for me to run a back-to-back test between two 10Gig NICs. Since the board does not support PCI graphics cards, I had to boot on serial port (the only VGA card I've got running on this mobo was a cheap crappy GeForce 8400 GS which does not work under X). Well, my first test is quite encouraging : I can achieve 10 Gbps of HTTP traffic between the two NICs with the server and client on the same machine, which means that the hardware will be able to support haproxy under the same conditions. I tried with client + haproxy + server but the bit rate diminished to about 6-7 Gbps. I'm impatient to buy the 3 other mobos to build a full lab. I will mix 2 Athlons and one C2D so that I can experiment which one is better for which type of traffic. Stay tuned!

January 21th, 2008

    Released haproxy maintenance version 1.3.14.2 to address several minor bugs as well as a major one affecting Linux 2.6 users with the sepoll poller, which can result in truncated responses if the client closes the connection before the server completes its response. Note that version 1.3.13.2 was released too with those bugs fixed. The GNU Makefile was crappy and caused trouble in some build environments. It has been rewritten in a more flexible manner, while still providing full variable compatibility with existing build systems. Distribution packagers are encouraged to migrate over to this one. The new configuration manual is almost finished. All keywords and all their options have been documented. Only the logs section remains to be completed. This version has been merged with 1.3.14.2. Some minor robustness and performance tuning parameters have been added, mostly timeouts and backlog.

January 13th, 2008

    Worked all the day both in kernel and haproxy to get full transparent proxy to work on Linux. Now, with a small kernel patch, it's possible for haproxy to become completely transparent and just appear as a router, without touching either source nor destination addresses and ports. And all this without NAT, at the same performance level as in normal proxy mode. This will be great for people looking for SMTP/HTTPS/FTP relaying and load balancing. I'm even planning on installing it on my firewall ;-) Stay tuned for the updates, I will soon post the patches once cleaned up.

December 12th, 2007 : Santa Claus left a present for me at EXOSEC !

    Some of you might have already got their hands on this. For those who don't know yet, this beautiful piece of art is a 10 Gbps Ethernet NIC from Myricom. For a long time, I had been tempted by their legendary high performance network cards, which were said everywhere to be able to saturate a 10 Gig wire under Linux without putting too much stress on the CPU, using a mainstream opensource driver, and without resorting to dirty tricks such as TOE. What would a performance addict like me need more ?

    I finally decided to mail these guys and described how I'm currently used to benchmark HAproxy with aggregated Gigabit NICs, with a minimum of 4 NICs in a setup (1 for the client, 2 for the proxy, 1 for the server). 4 hours later, when I woke up, I had a mail from Charles Seitz, Myricom's CEO. He explained to me that he was pleased to offer me 4 NICs with cables, plus one spare of each just in case, as their contribution to the project... yes, I'm talking about a donation of five 10Gig NICs! That's awsome! And if it would not be enough for some of you to find them really cool, he also provided me with french-speaking contacts, free access to their support and important advices for the choice of motherboards to get the best out of those wonderful NICs! I don't even know the polite words to say in such circumstances :-)

    Today I've been monitoring the shipping steps at UPS. This evening, I noticed that they arrived at EXOSEC. After leaving the customer's, I went back there to find this big parcel on my desk, with its contents very carefully packed. I must say that I was both very excited and extremely careful while opening the packaging.

    The first thing I noticed after extracting the first NIC from its packaging was that it had a very clean design, as can be seen on this photo. They are also very thin as shown on the picture on the right, so there will be no problem putting two of them side-by-side in the proxy.

    The CX4 connector looks a bit fragile, but careful manipulation is the minimal requirement to use the highest speed standard Ethernet. From what I understood, this is the same connector as used on Infiniband, except that 10GE has terminators on the board.

    Well, obviously, there are very nice companies out there who deserve to be talked about! Their very generous support to open source projects leaves many others far behind. People say that Santa Claus lives in the North Pole, but now I know he lives in Arcadia in California :-)

    Thank you very much Charles, thanks very much Myricom. Be sure to read about my first test results here.




December 6th, 2007

    Released haproxy version 1.3.14. A good part of the changes comes from nice contributors of the mailing list. Most sensible changes include support for dynamic server weights offering support for slow start and graceful shutdown. The load balancer is now able to report its servers state to outer components, enabling the building of more complex multi-site architectures involving dynamic routing protocols such as BGP. People who were complaining about the rough configuration, rough statistics, or lack of logging to UNIX sockets, should really give this one a try. Rate of changes after this version should significantly drop in order to progressively switch the tree to a stable state.

October 18th, 2007

September 22th, 2007

September 20th, 2007

    Released haproxy version 1.3.12.2. It fixes several bugs affecting timeouts and retry counts when configs are split between frontends and backends. Some sanity checks on the configuration file were never executed, causing some erroneous configurations to be accepted without being fixed. Last, the license has been clarified in a few files from O'Reilly. All in all, it seems like keeping a supported version is already starting to pay off, as people are looking for something stable and report bugs very quickly. All version 1.3 users are encouraged to upgrade to 1.3.12.2.

September 5th, 2007

    Released haproxy version 1.3.12.1. It fixes a few bugs discovered in 1.3.12, notably one which could lead to crashes under Linux with speculative I/O when clients disconnect before the connection has been established to the server. As a workaround, it is possible to specify "nosepoll" in the "global" section. A "stats refresh <interval>" option has also been added because some people like to have the stats page automatically refresh. It's also possible to hide all failed servers on the stats page now. This version also contains the new configuration manual which has just been started but which helps understand how to use ACL.

July 15th, 2007

    Started writing the new Configuration Manual. It enumerates all configuration keywords and in what context they may be used. It also includes a few examples of ACLs. It is not finished yet, but I decided to publish it because people have really no other valuable sources of information to use content switching. It only covers version 1.3.12, and updates will only cover the latest version, making it far more readable. Please take a look at it and start from the examples in the examples/ directory from the sources. Any feedback is welcome :-)

June 17th, 2007

    Released haproxy version 1.3.12. It completes integration of ACL with Content Switching, and allows you to customize your error responses. Except for the ACL and a few bugs, there have been few changes since 1.3.11.4, and I intend to support 1.3.12 during development and cleanups of the next versions which may not be as reliable. Several big content providers use 1.3 to regulate the traffic to/from their web servers, and there is a real demand for a stable version with the new features and performance of 1.3. And considering that some of them even pay for this, I understand they want something really reliable.

June 3rd, 2007

    Released haproxy version 1.3.11.4. It fixes 2 long-standing bugs in timeout handling, which could sometimes cause 100% CPU usage during several seconds when a client had closed its write channel. Some small improvements to the I/O subsystem should save some CPU cycles on high bandwidth sites. It is now possible to finely tune the pollers for reduced latency.

May 14th, 2007

    Released haproxy version 1.3.11.3. It fixes the (hopefully) last bug affecting Linux users with speculative I/O processing, introduced in 1.3.9. This bug was also causing random timeouts. Do not use versions 1.3.11 to 1.3.11.2 as they are all broken.

    New in this release are a better timer management and a new memory manager which is able to self-manage its pools and free unused ones when memory is becoming scarce. It is also easier to code with this new one since it's not necessary anymore to declare pool sizes. Overall, yet another performance boost of 5% has been gained.

May 10th, 2007

May 9th, 2007

    Released haproxy version 1.3.10.1. It fixes a serious bug affecting Linux users with speculative I/O processing, introduced in 1.3.9. This bug was causing random timeouts on some traffic patterns, mostly noticeable in TCP mode but almost certainly in HTTP too. All Linux users of 1.3.9 and 1.3.10 should either upgrade or disable speculative I/O as a workaround, by starting haproxy with the -ds argument or by setting nosepoll in the global section.

May 9th, 2007

    Released haproxy version 1.3.10. This one adds ACL, SMTP health checks (thanks to Peter van Dijk), and URI hashing (thanks to Guillaume Dallaire). Also, the rbtree was replaced with a much faster tree, leading to an overall performance boost around 5%.

    The speculative I/O processing in 1.3.9 has introduced some bugs which have been fixed in this version. I feel confident that latest changes have brought their pile of bugs too. I will probably spend some time soon to do cleanup and stabilization work, eventhough both are not really compatible.

    I also want to thank all the people who contribute code and testing. You are more and more at each release. I'm impatient to clean up the remains of the old code, so that even more people can contribute code. Interestingly, all contributions till now were of high quality. This is probably induced by some sort of selection caused by the technical aspect of the product, and the skills required to use the development version. Thanks again to you all !

Apr 22th, 2007

    Done a quick benchmark at EXOSEC with haproxy 1.3.9 running on a nice single-core system equipped with many PCI-Express Gigabit NICs. The graph shows pretty decent results !

Apr 15th, 2007

    Released haproxy version 1.3.9. This one adds modularization to the pollers, which made it possible for me to finally implement support for FreeBSD kqueue(). I'd like to thank Olivier Warin for providing me a FreeBSD account during this development.

    A new concept was introduced too : speculative I/O. It is a new method consisting in reducing the number of calls to the expensive epoll_ctl() and epoll_wait() by attempting to access the file descriptors before being notified about their readiness. This provides an overall speed boost of 10%, which is quite much for just a poller.

Apr 3rd, 2007

    Released haproxy version 1.3.8.2 to fix a minor and a major bug. The minor bug caused the response rewrite to fail on the status line. The major bug which was introduced in 1.3.6 could cause the process to crash in some circumstances when rewriting the request line (method and/or URI). All users of 1.3.6 and later must upgrade.

Apr 1st, 2007

    Released haproxy version 1.3.8.1 to fix very minor bugs, and slightly improve performance. Request headers were not added if option httpclose was not set. Bruno Michel contributed a VIM script for syntax color highlighting.

Mar 25th, 2007

    Released haproxy version 1.3.8. Several bugs which might have caused crashes on erroneous configurations have been fixed. The response processing is now completed, which means that real configurations can now be written ; HAProxy 1.3.8 now is at least equivalent to 1.2.17 in terms of features.

    Just like with every release, several code optimization have led to small but noticeable performance increases, particularly on very high data bandwidth. The configuration errors are handled more gracefully now with indications about what failed and hints to resolve the issue. HAProxy now builds on MacOS 10.4 thanks to Dan Zinngrabe who provided a makefile. Also, it is now possible to send health checks to an alternate server address, thanks to a patch from Fabrice Dulaunoy.

    Users of 1.3 are encouraged to upgrade to 1.3.8 as it both fixes known bugs and converges towards something less tricky than previous versions.

Mar 17th, 2007

    Released haproxy version 1.2.17. I have backported Sin Yu's rbtree scheduler from version 1.3 since it proved to be stable. A few minor bugs were fixed, and two useful contributions were merged : support for user and group keywords as alternatives to numerical uid and gid from Marcus Rueckert, and the ability to prevent some source addresses from appearing in the X-Forwarded-For header, which is useful when combined with Stunnel for instance (patch from Bryan Germann). Thanks to both of them, contribs are always welcome !

    The architecture manual was updated to reflect new features in branch 1.2, with examples for stunnel and for load mitigation.

    Users of 1.2.16 with high loads are encouraged to upgrade to 1.2.17 as it offers them the high performance of branch 1.3 with the reliability of the stable branch 1.2.

Jan 27th, 2007

    Released haproxy version 1.3.7. I found a critical bug in the new parser in development branch, causing crashes when an empty header is passed. This was caused by a missing pointer assignment in the empty header processing path. All 1.3.6 users MUST upgrade to 1.3.7.

Jan 22th, 2007

    Released haproxy version 1.3.6. I spent a long time reworking the HTTP message parser. It now consists in a carefully hand-optimized 28-states FSM. The new code will look awful to goto haters, and will please FSM lovers. It's blazingly fast : parsing and indexing all of the 660 bytes of an HTTP request from Firefox on Freshmeat only takes 1.94 microsecond on my 1.7 GHz Pentium-M notebook, which means it can do it more than 500000 times a second!

    The request code has been cleaned up a lot and adapted to this new FSM. Adding layer7 rules based on new criteria is now trivial. The response code will be ported next, but the code was so much cleaner and faster that it was worth releasing one version before breaking everything. Several bugs were fixed since 1.3.5. I really consider 1.3.6 as the most likely reliable 1.3 release to date.

Jan 7th, 2007

    In order to support the new Linux Layer7 Switching project, I have implemented support for kernel TCP splicing using Alexandre Cassen's library. This is still experimental but already works remarkably well. On my notebook at 400 Mbps, haproxy's usage goes down from 65% to 5-10%. I have written some doc explaining how to setup up TCP splicing, with an example. Since the original code was provided for Linux kernel 2.6.19 only, I have backported the patches to kernel 2.6.16 and 2.4.33.

    The second great news is that Sin Yu has provided me with a useful patch for the second time : the task scheduler is now based on an rbtree and not on the dirty old dual-linked list anymore. It means that people who had performance problems and who had to set all their timeouts to the same value as a workaround will not have to do this anymore. I have tested, and the code works like a charm ! Thanks again Sin !

Jan 2nd, 2007

    After about 4500 new lines of code and some useful feedback from a bunch of brave beta-testers, I'm pleased to announce haproxy version 1.3.4 with the new Content-Switching features !!!

    It is now possible to select a backend (server pool + load balancing algo) depending on any parameter in the request, such as any part of the URI, the host name, etc.... As of now, I've merged Sin Yu's patch to permit switching based on a request regex, but the framework is ready to accept many other criteria. The HTTP request parser has been completely rewritten to support unlimited header inspection, and the statistics page has been rewritten, as can be seen on the demo page. It is far from being finished right now, but it seems pretty usable. The server state machine should be adapted though.

    There is still no doc, so please note that old configurations do still work, and that in order to switch from an instance to another backend, you need to use "reqisetbe <regex> <new_proxy>". Also, there's a config example here that will be worth any doc.

Dec 5th, 2006

    The load balancing article has been linked to from LinuxFR. The small 128 kbps uplink is currently running at full speed but the site is still responding thanks to haproxy queuing the connections to smoothen the traffic. Next time, I should also write an article on setting up the QoS with tc, because typing remotely with SSH is still very responsive under full load :-)

Jul 4th, 2006

    Opened development branch 1.3, which started with a major cleanup. Not sure yet about all features which will be merged, the first step is to clean up the code and make it modular. The API's licence has been switched to LGPL in order to later allow linking with binary external modules developped for applications covered by NDAs for example. Version 1.3.0 is exactly the same as 1.2.14+bugfixes so it is a stable starting point. It is available here.

⇐ Back to HAProxy

Contacts

Feel free to contact me at for any questions or comments :

  • Main site : http://1wt.eu/
  • e-mail :