The LMAX Architecture (2011)(martinfowler.com)
martinfowler.com
The LMAX Architecture (2011)
https://martinfowler.com/articles/lmax.html
35 comments
This isn't even close to unique. I['ve worked at a few HFT/MM places that all had a similar architecture in place before LMAX wrote this. This, multicast rings, and other patters are copied a lot in the industry because they work well for the problem space.
I'm not surprised. And I think you nailed the reason too:
> copied a lot in the industry because they work well for the problem space
Finance as an industry could be described as ... incestuous. The people move around within the small circle of firms, so any architectural knowledge gets cross-pollinated quite rapidly. From what I understand, the disruptor pattern itself was reasonably well known in the academic circles well before LMAX wrote about it - but I'm not sure anyone had put publicly available numbers on a heavily optimised version before them. (FWIW, I use a simplified disruptor in one of my own projects because it works well and simplifies the overall design. )
We should also be honest about the technical challenges: there are very few as interesting problem spaces as an exchange or a low-latency MM.
> copied a lot in the industry because they work well for the problem space
Finance as an industry could be described as ... incestuous. The people move around within the small circle of firms, so any architectural knowledge gets cross-pollinated quite rapidly. From what I understand, the disruptor pattern itself was reasonably well known in the academic circles well before LMAX wrote about it - but I'm not sure anyone had put publicly available numbers on a heavily optimised version before them. (FWIW, I use a simplified disruptor in one of my own projects because it works well and simplifies the overall design. )
We should also be honest about the technical challenges: there are very few as interesting problem spaces as an exchange or a low-latency MM.
So I worked at LMAX during 2009-2011 when Martin Thompson, Dave Farley, Mike Barker, Chris Smith and Danny Yates worked on the Disruptor. It was in use in 2010 inside LMAX.
The heritage of Martin and Dave was video games rather than finance and they had the most input into it.
From a friend who worked in games, the general idea behind the disruptor had been in use in games for a while.
The heritage of Martin and Dave was video games rather than finance and they had the most input into it.
From a friend who worked in games, the general idea behind the disruptor had been in use in games for a while.
In finance, I know it goes back to Island ECN (which became part of NASDAQ). They had a small group of heavy talent that spread out to a number of high profile banks and prop shops. I've worked at a few places with the same architecture and it always seems to lead back to those guys at Island by a couple degrees.
I wouldn't be surprised if there was some cross pollination. I regularly read game dev blogs for ideas and information. While trading tends to have higher latency requirements and stronger networking requireemnts, games have other issues (the PCI bus and graphics card), but in the end there are a lot of architectural similarities and both tend to be a small group of highly skilled developers so you have a lot more freedom to chose complex index and advanced tooling.
I wouldn't be surprised if there was some cross pollination. I regularly read game dev blogs for ideas and information. While trading tends to have higher latency requirements and stronger networking requireemnts, games have other issues (the PCI bus and graphics card), but in the end there are a lot of architectural similarities and both tend to be a small group of highly skilled developers so you have a lot more freedom to chose complex index and advanced tooling.
A ton of real-time audio/video apps use the same general architectural pattern.
I absolutely loved Thompson's blog. He hasn't posted in a decade, and that makes me sad. I learned so much from that. I used the term "mechanical sympathy" all the time now when speaking to starting developers and trying to explain how to write fast code.
Can you explain what a multicast ring is? The article itself just grossed over it.
Thanks!
Thanks!
You have many servers receiving orders in a network. Instead of simple TCP/IP packets, they go with multicast and all of them receive the same packets and keep the same state in memory. If one of them fails you can still depend on the others
Thank you!
you have a group of machines, and each usually has specific obligations: logging message to file and db, one connects and gets prices and broadcasts them into the ring, one does order checks, one sends orders to the outside, one does strategies. Except there are many for each responsibility (like one price feed per venue). Plus these all have back ups.
The multicast ring every machine multicasts their packets, and if a machine wants that data it just subscribes to that channel. So the logger might subscribe to every channel, the price feeds will send to a specific group of channels, the order routing systems will listen on a particular channel, and core might subscribe to the strategy channels, and the strategies will subscribe to the price feeds and core and send to the order channels.
This allows you to have hundreds of machines without any slowdown, and you can literally just drop a new machine on the right and it just works. No need to change anything else. It highly decouples the systems. You just need to have a very tight protocol and extremely well written network code (these are always custom implementations - I've written custom SSL/TLS, websocket, webservers, TCP itself, etc. for these things.
This also allows close to perfect replayability. The machine logging logs everything, and then when you want to analyze something, you generally have tools to replace the message back into the ring and get the exact same state.
And all these active machines also have backs up, so you have two machines taking orders and routing them for a venue: one is the currently active and doing all the work and heartbeating, but there is another that is just reading all the messages keeping the same state as the primary, but when the primary goes down, the secondary can seemlessly pick up and none of the other machines need to even know what happened. It makes for a very resiliant system.
Plus you can use any language you want. We wrote all the trading systems in C++, but we also had nodes on the ring in python, java, and even shell scripts listing for particular messages. I remember one hack I did when the futures market keeping hitting its limit down a couple days in a row, so we made a shell script that listened for a particular limit down message from the price feel to be broadcast into the ring, and when it got it, it send another message to each algo telling them to get out of anything and pause. took less than a day to write using bash.
Its a really nice way to handle event systems, just it is sometimes difficult to get others to understand the benefits and it requires a lot of implementation skill and experience.
The multicast ring every machine multicasts their packets, and if a machine wants that data it just subscribes to that channel. So the logger might subscribe to every channel, the price feeds will send to a specific group of channels, the order routing systems will listen on a particular channel, and core might subscribe to the strategy channels, and the strategies will subscribe to the price feeds and core and send to the order channels.
This allows you to have hundreds of machines without any slowdown, and you can literally just drop a new machine on the right and it just works. No need to change anything else. It highly decouples the systems. You just need to have a very tight protocol and extremely well written network code (these are always custom implementations - I've written custom SSL/TLS, websocket, webservers, TCP itself, etc. for these things.
This also allows close to perfect replayability. The machine logging logs everything, and then when you want to analyze something, you generally have tools to replace the message back into the ring and get the exact same state.
And all these active machines also have backs up, so you have two machines taking orders and routing them for a venue: one is the currently active and doing all the work and heartbeating, but there is another that is just reading all the messages keeping the same state as the primary, but when the primary goes down, the secondary can seemlessly pick up and none of the other machines need to even know what happened. It makes for a very resiliant system.
Plus you can use any language you want. We wrote all the trading systems in C++, but we also had nodes on the ring in python, java, and even shell scripts listing for particular messages. I remember one hack I did when the futures market keeping hitting its limit down a couple days in a row, so we made a shell script that listened for a particular limit down message from the price feel to be broadcast into the ring, and when it got it, it send another message to each algo telling them to get out of anything and pause. took less than a day to write using bash.
Its a really nice way to handle event systems, just it is sometimes difficult to get others to understand the benefits and it requires a lot of implementation skill and experience.
Interesting! Can you compare this to traditional web/enterprise pub-sub systems? Obviously this has a lot more custom network code, but is it otherwise analogous to something like NATS or a brokered (but stateless/unbuffered) topic-based system a la AMQP?
grossed -> glossed
(aiming to help non-native English readers, vs nitpicking-pedantry)
(aiming to help non-native English readers, vs nitpicking-pedantry)
As a non-native English speaker but someone who grew up in the US, you may have no idea how much I appreciate that. Thank you!
So much hype for a lock-free MPMC ring buffer. If you're familiar with the concurrency literature you'll see that nothing here was novel even at the time.
java had a proper memory model and the busy wait was a novelty. iirc, it was used mostly in a single thread processing (i.e. consumer) for the main business logic like matching forex order.
Realistically aside the disruptor LMAX had an interesting way for fail over by replaying all the input to reach the state.
Realistically aside the disruptor LMAX had an interesting way for fail over by replaying all the input to reach the state.
I think that -MC is misleading. It would imply anycast (like what you would use for a work distribution queue), but the disruptor is multicast, I.e. all receivers potentially receive all the data. It is also lossy, like UDP.
Can you point to some relevant literature?
Is there any didactic implementation of the Disruptor / multicast ring available somewhere? I've been curious in working through some practical example to understand the algorithm better.
LMAX have an open source version of the disruptor in GitHub https://github.com/LMAX-Exchange/disruptor
Here's a high level description in TLA+:
https://github.com/nicholassm/disruptor-rs/blob/main/verific...
(Disclaimer: I wrote it.)
There's also a spec for a Multi Producer Multi Consumer (MPMC) Disrupter.
(Disclaimer: I wrote it.)
There's also a spec for a Multi Producer Multi Consumer (MPMC) Disrupter.
Is the LMAX disruptor just a ring buffer? Anything else special about it?
It is a ring buffer indeed, with publish serialization (on producer side) and normally a busy wait. It sort of doesn't work on some VMs due to scheduling. (I mean the busy wait w/o a backoff)
The big win is how it allows bulk processing while taking serialization dependencies into account.
So you algos reading market data messages and yank all that is there and apply them in bunk instead of trying to do each one individually.
And your logger that is storing all the messages to a file and/or db knows what messages the algo has processed and can grab all the messages up to that point so it is always writing as much as it can and current.
So you algos reading market data messages and yank all that is there and apply them in bunk instead of trying to do each one individually.
And your logger that is storing all the messages to a file and/or db knows what messages the algo has processed and can grab all the messages up to that point so it is always writing as much as it can and current.
Is this ability that is the result of each component being able to read the index/sequence number of all the other components?
Sounds so much like an io_uring model.
I wrote this in Crystal lang to learn more about it and the language: https://github.com/nolantait/disruptor.cr
There is a Ruby version that is also very readable: https://github.com/ileitch/disruptor
There is a Ruby version that is also very readable: https://github.com/ileitch/disruptor
And here's a version in Rust:
https://crates.io/crates/disruptor
(Disclaimer: I wrote it.)
(Disclaimer: I wrote it.)
Thanks for that. I was always curious why the finance bro down the block had an LMAX license plate on his car.
Fun fact, log4j use disrupter
One of my favorite patterns.
The implementations of this for C# support a value type as the buffer object. You can get upward of a half billion events per second in a SPSC arrangement.
My use cases for it are primarily in gaming or where I have a latency critical element. C#/.NET isn't necessarily ideal for these use cases, but you can do clever things like align GC to the disruptor batch processing boundaries.
If you are finding trouble with this pattern on small machines (1-2 cores), consider adding a heuristic that checks the core count of the machine and swaps from busy to a yielding wait strategy if there aren't at least 4 cores. Yielding wait isn't always good enough, but it definitely works for testing and prototyping.
The implementations of this for C# support a value type as the buffer object. You can get upward of a half billion events per second in a SPSC arrangement.
My use cases for it are primarily in gaming or where I have a latency critical element. C#/.NET isn't necessarily ideal for these use cases, but you can do clever things like align GC to the disruptor batch processing boundaries.
If you are finding trouble with this pattern on small machines (1-2 cores), consider adding a heuristic that checks the core count of the machine and swaps from busy to a yielding wait strategy if there aren't at least 4 cores. Yielding wait isn't always good enough, but it definitely works for testing and prototyping.
Is the company that uses this perchance located in Europe?
See also: the discussion on the LMAX disruptor github documentation https://news.ycombinator.com/item?id=38313457
Event input queue, fully deterministic matching engine, and an output queue. Plus importantly, a sequencer to guarantee that the events going into the input queue have a known order.
Aspects of their architecture and design decisions have been made public through their podcast series, Signals and Threads.[0]
0: https://signalsandthreads.com/