Designing a Low Latency 10G Ethernet Core (2023)(ttchisholm.github.io)
ttchisholm.github.io
Designing a Low Latency 10G Ethernet Core (2023)
https://ttchisholm.github.io/ethernet/2023/05/01/designing-10g-eth-1.html
82 comments
It seems fun to be a high frequency FPGA trader designer. All my FPGA is much lower power consumption so I don't get to play with stuff like gigs of external SRAM or the QDR stuff or whatnot
I feel like it's also got to be super stressful
Imagine getting a call you system is down and is costing them millions per hour
Imagine getting a call you system is down and is costing them millions per hour
You can code a failsafe, or what they call in other industries, an interlock. If an order exceeds some sane limit, or a loss is incurred that exceeds a limit, or the limit-checker fails to proactively ping the interlock, then an independent mechanism shuts it down.
This is how, say, a modern x-ray machine prevents itself from failing in a way that radiation gets stuck on.
This is how, say, a modern x-ray machine prevents itself from failing in a way that radiation gets stuck on.
The way to think is the opposite: your system is making millions per hour.
What about when something goes wrong because you overlooked some detail?
Shit happens. I wrote the firmware for a multi-link HDLC card which earned vast amounts of money for the company that commissioned it. I was quite aware of my responsibility and so was the hardware team. We stress tested the crap out of it before releasing to production and fortunately it never locked up when it was in actual use but we had some pretty wild and very rare (so hard to trigger and reproduce) bugs which delayed deployment considerably. But none of this gave me the kind of feeling that working on fuel estimation software for aircraft gave. That's when there are in the most literal sense lives on the line, and that's a completely different kind of pressure. You simply can not fuck up. It also really rammed home the value of code review and having a good specification so that you can ensure that within the envelope of input parameters your software does what it is supposed to do.
Multi-link HDLC. That brings me back to 1985 and the Intel 8274, Zilog Z8530 and the Western Digital WD2511 (that implemented the layer 2 protocol in silicon).
Yes, that was around that time. There were a couple of problems, the first batch of chips we got was very early in the development stage of the chip, pre-production issues and there were some bugs in the chips themselves which could cause lock-up under some circumstances. We found ways to work around those and then of course there were all of the niceties around dealing with a device that generates an extremely high rate of interrupts. So the code would either have to attempt to service all channels on any interrupt or be re-entrant. I don't remember which solution we picked but in the end the thing was, once we had the bugs worked out fairly bullet proof.
One funny bit about the development process was that initially I was going to be nice and implement every layer as its own stand-alone bit of software communicating via a defined set of primitives with the layers above and below. But that was slow as molasses so in the end all of that elegance got discarded for an absolutely unholy sandwich that did all of the layers in a single chunk of code. But with the experience from the 'slow' version that was actually doable, I would have never been able to write that as the first implementation. Classic illustration of 'first make it work, then make it fast'.
One funny bit about the development process was that initially I was going to be nice and implement every layer as its own stand-alone bit of software communicating via a defined set of primitives with the layers above and below. But that was slow as molasses so in the end all of that elegance got discarded for an absolutely unholy sandwich that did all of the layers in a single chunk of code. But with the experience from the 'slow' version that was actually doable, I would have never been able to write that as the first implementation. Classic illustration of 'first make it work, then make it fast'.
Cool. We had a small company in San Diego called Metacomp (long gone now) design a Z8530 add on module for their 80186 based Multibus board. It had two Z8530 chips on it and you could have two modules on the base board. So 8 channels total per slot.
Yes, 8 was just about the maximum, simply because that would approach the limits of how many interrupts you could process before you'd inevitably start losing them. Also, space and power delivery limitations, and connector space on the backside made doing more than that very impractical. Or you had to take more than one interrupt per board but that was 'not done'. I've always loved working on that division line between hardware and software, as close to the metal as possible. Funny, I'd all but forgotten about this project, now I'm wondering if I still have the software somewhere, just searched for a bit and can't locate it, so there is a good chance I wrote it on company hardware. I did find another project that I'd forgotten about, a CAD program for sails for sailboats that I wrote in the 80's for TD Sails in the Netherlands. That was a great project to work on and I met some awesome people doing it.
True. If you put in the whole system maybe.
On the other hand, no nights or weekends.
This is becoming less and less true every day. 24/7 markets are coming whether you want them or not, and if you're not trading outside of US Equity hours you're leaving significant money on the table.
> This is becoming less and less true every day. 24/7 markets are coming whether you want them or not
This has still to exist for real. Every couple of years there is a resurgence of "we should have 24/7 equity markets", followed by milestone announcements of Nasdaq.
The reality is more... nuanced.
First, there has to be a wide enough window for corporate actions and news dissemination to happen. Not only is that a regulatory requirement, but anyway no sane investor would like to trade in such a window even if possible.
Second, there are already early sessions, pre opening sessions, and late sessions on most American markets. The liquidity there is almost inexistant, so I wouldn't say there is a huge demand for extended trading hours by non-retail participants.
And last, half the liquidity of US markets is not on lit books, and half the remaining lit liquidity is close to the auction. I don't think it would even be a net profit for lit exchanges to extend their hours in these conditions. As for dark pool, I think only 2 or 3 of the smaller ones are 24/7, like the one from IB.
> if you're not trading outside of US Equity hours you're leaving significant money on the table
HFTs need a lot of liquidity and tight spreads, that's pretty much only doable at scale in the US and Japan. Europe and EM are either not liquid enough, or only really able to sustain low capital ad-hoc strategies.
This has still to exist for real. Every couple of years there is a resurgence of "we should have 24/7 equity markets", followed by milestone announcements of Nasdaq.
The reality is more... nuanced.
First, there has to be a wide enough window for corporate actions and news dissemination to happen. Not only is that a regulatory requirement, but anyway no sane investor would like to trade in such a window even if possible.
Second, there are already early sessions, pre opening sessions, and late sessions on most American markets. The liquidity there is almost inexistant, so I wouldn't say there is a huge demand for extended trading hours by non-retail participants.
And last, half the liquidity of US markets is not on lit books, and half the remaining lit liquidity is close to the auction. I don't think it would even be a net profit for lit exchanges to extend their hours in these conditions. As for dark pool, I think only 2 or 3 of the smaller ones are 24/7, like the one from IB.
> if you're not trading outside of US Equity hours you're leaving significant money on the table
HFTs need a lot of liquidity and tight spreads, that's pretty much only doable at scale in the US and Japan. Europe and EM are either not liquid enough, or only really able to sustain low capital ad-hoc strategies.
Genuinely curious - would 24x7 low-latency trading matter then? Wouldn't after-hours trading (for your local timezone) be happening on an exchange in another location?
Or do you mean dark pools/private exchanges that may run 24x7?
Or do you mean dark pools/private exchanges that may run 24x7?
PoE is the way to use low voltage in electronics.
I'd want to know why they had such a system as a single point of failure
Failure modes can be a bug making a bad trade while doing so at 100% uptime.
I agree! Seems like very interesting work.
> It seems fun to be a high frequency FPGA trader designer.
Yes, if you think that doing pointless work is fun. What do you tell your grandchildren?
Yes, if you think that doing pointless work is fun. What do you tell your grandchildren?
It's a bit unfair to call it pointless. Empirical studies have shown that HFT increase liquidity in markets, leading to narrowed spreads, market depth, faster execution speeds, cross-market informational efficiency, etc. All of these are generally beneficial to other non-HFT market participants.
Now you might counter that it also brings some negatives (e.g. fleeting liquidity, market manipulation risk, etc.) and there's certainty some validity to that. But none of this supports an argument that it's pointless.
More generally, activities aren't pointless if you are earning money from it which pays your rent, puts food on your table, etc. A large portion of the world works mainly for this goal. It's perfectly valid to tell your grandchildren "I spent my life making sure we'd have enough money for you and your parents to live comfortably", and we shouldn't look down on someone who does that.
Now you might counter that it also brings some negatives (e.g. fleeting liquidity, market manipulation risk, etc.) and there's certainty some validity to that. But none of this supports an argument that it's pointless.
More generally, activities aren't pointless if you are earning money from it which pays your rent, puts food on your table, etc. A large portion of the world works mainly for this goal. It's perfectly valid to tell your grandchildren "I spent my life making sure we'd have enough money for you and your parents to live comfortably", and we shouldn't look down on someone who does that.
> HFT increases liquidity
On average with a few caveats yes, but it also causes issues by reducing the vastly more important signaling aspects of the market.
Trade volume for example can get almost completely divorced from economic reality.
On average with a few caveats yes, but it also causes issues by reducing the vastly more important signaling aspects of the market.
Trade volume for example can get almost completely divorced from economic reality.
> Also, it's not pointless to the person actually engaging in HFT if they are making money from it. Activities aren't pointless if they pay your rent, put food on your table, etc.
You have to look at the bigger picture. HFT companies only increase the barrier to entry to participate in the market. This means that while you may be able to put food on the table, you make it more difficult for other people to put food on the table.
You have to look at the bigger picture. HFT companies only increase the barrier to entry to participate in the market. This means that while you may be able to put food on the table, you make it more difficult for other people to put food on the table.
Markets should not primarily be a casino.
If you want to invest some of your income into a spread of companies? Go for it. But if all you are doing is trying to game the market? That's just gambling, go to a real casino and have some fun instead.
HFT should be driven by these types of market makers exactly for the purpose of decreasing the amount of unnecessary speculation and volatility in the markets. HFTs killing the spreads is one of the only reasons the markets didn't implode during covid despite the extraordinary fear and volatility that was impacting them.
You want a parasite? Look to hedge funds who just sit on piles of other people's cash to scoop obscene amounts off the top (relative to their actual contribution) for occassionally shuffling funds between financial instruments.
If you want to invest some of your income into a spread of companies? Go for it. But if all you are doing is trying to game the market? That's just gambling, go to a real casino and have some fun instead.
HFT should be driven by these types of market makers exactly for the purpose of decreasing the amount of unnecessary speculation and volatility in the markets. HFTs killing the spreads is one of the only reasons the markets didn't implode during covid despite the extraordinary fear and volatility that was impacting them.
You want a parasite? Look to hedge funds who just sit on piles of other people's cash to scoop obscene amounts off the top (relative to their actual contribution) for occassionally shuffling funds between financial instruments.
> HFTs killing the spreads is one of the only reasons the markets didn't implode during covid despite the extraordinary fear and volatility that was impacting them.
And why would this not be possible if everybody traded at the same, low, frequency? Alternatively, what is the minimum trading frequency required to save markets from collapsing?
By trying to answer these questions you see how ridiculous the situation with HFT really is.
And why would this not be possible if everybody traded at the same, low, frequency? Alternatively, what is the minimum trading frequency required to save markets from collapsing?
By trying to answer these questions you see how ridiculous the situation with HFT really is.
Sorry but I don't understand any of your arguments.
You are throwing things like "HFT increase the barrier to entry", "HFT make things unfair", but... how so?
What would _you_ do if you had the capability to trade at the speed of light while others cannot?
Trading faster does not magically make money appear in your account, it does not magically tell you what is going up and what is going down.
It seems people work under the assumption that HFTs make money by having the same ideas than others, but trading it faster. That is _not_ what market making is, that is _not_ what HFTs do, and trading 10ns "faster" is of absolutely no use to 99.99% of market participants. In fact, the vast majority of market participants purposefully *slow* their signals to avoid impacting the market. At any given point in time, the orderbook contains but a tiny fraction of the liquidity that most people are willing to trade.
HFTs usually run on a combination of 4 businesses, none of which provide any downside for the rest of the participants: Market making, events signals, order flow and arbitrage.
Market making: essentially they are paid (or more precisely, rebated) to "animate" the market (provide liquidity). Everyone is happier with tight spreads, and everyone is happier with some level of liquidity at the top of the book, I don't really see how that could be up to debate. This is a tough business, the market maker has to provide a contractual amount of inventory for buyers and sellers at the same time, with the risk that these assets can inadvertently increase or decrease in value. Market makers get a premium (the spread) in exchange for providing liquidity to market takers, taking the risk of holding that inventory for them.
Order flow: HFTs are usually more knowledgeable than managers (and even brokers) to perform trade execution. Remember, the price of an asset does not move "by itself". What makes the price of an asset _really_ increase is an imbalance between the size of buys and sells, causing the spread to be crossed increasingly in one direction, and eating higher and higher price levels => a market participant buying shares is self realizing the price increase. That is called market impact, and that is something nobody wants. In an ideal world, nobody would know the actual liquidity that other participants are willing to exchange on the market, that way no arbitrage would be possible, no market impact, everyone is happy. The problem though is that you need _some_ level of visibility otherwise you hinder so called "price discovery", and regulators don't like that because it's an open door for scams, hidden fees, etc. So a lot of investors just delegate the actual order passing to HFTs: you agree on a price you're happy with, and the HFT executes the orders for you. If they manage to get a better price: good for them, they can pocket the difference. If they didn't, they will have to eat the cost and do better next time. Again, it's a _difficult job_, because by very definition managers are going to try to give you _toxic flows_: if someone mandates you to buy 100 shares of Microsoft, it's because they think the price will go up, but you will quote them at the current stock price. I don't think anyone is negatively impacted by that business to be honest. I am quite happy to instantly do a trade at a known price and let an HFT take the risk of executing trades, finding liquidity, etc.
Arbitrage: This may be the most well known HFT strategy. Essentially benefiting from price disparity of more or less fungible assets. So if HSBC is priced the equivalent of $100 on the LSE and $101 on NYSE, they buy there and sell here. I don't think there is much debate on whether that's a good thing or not, everyone should be happy that prices are more or less consistent around the world. Most people also wouldn't want to bear the overnight nor settlement/FX risk of holding these.
Event signals: I guess that's what people overreact to, even though this is but a tiny and negligible part of the HFT businesses. There are strategies where the whole concept is just to be the first to react to some kind of event, and yes HFTs are the best placed to do such strategies. The thing is, this is not a very lucrative business for a simple reason, which I already stated before: at any point in time, the orderbook contains but a mere fraction of the available liquidity. And it takes *ample time* for the market impact of a trade to be absorbed. There are a lot of papers on the subject, but without entering in too much details: the bigger your order, the longer it will take for the book to absorb it and liquidity to come back. For most hedge funds doing statistical arbitrage, CTAs, etc, we are talking multiple *days*. So realistically, if you're in the business of reacting the fastest to some kind of news or event, there's really not much more you can do than cross the spread and eat one or two levels of the book. Even for the most liquid of US stocks, that's a couple hundred thousands dollar per trade on average, for a tiny return difference versus executing a couple of seconds later, so I hope you find a lot of these if you plan on getting rich.
TLDR, I think 99% of HFT hate is completely unjustified. They provide a legitimate service to most market participants, and as an investor I wouldn't want to do their job, nor would I think that they hinder in any way my investments, on the contrary.
You are throwing things like "HFT increase the barrier to entry", "HFT make things unfair", but... how so?
What would _you_ do if you had the capability to trade at the speed of light while others cannot?
Trading faster does not magically make money appear in your account, it does not magically tell you what is going up and what is going down.
It seems people work under the assumption that HFTs make money by having the same ideas than others, but trading it faster. That is _not_ what market making is, that is _not_ what HFTs do, and trading 10ns "faster" is of absolutely no use to 99.99% of market participants. In fact, the vast majority of market participants purposefully *slow* their signals to avoid impacting the market. At any given point in time, the orderbook contains but a tiny fraction of the liquidity that most people are willing to trade.
HFTs usually run on a combination of 4 businesses, none of which provide any downside for the rest of the participants: Market making, events signals, order flow and arbitrage.
Market making: essentially they are paid (or more precisely, rebated) to "animate" the market (provide liquidity). Everyone is happier with tight spreads, and everyone is happier with some level of liquidity at the top of the book, I don't really see how that could be up to debate. This is a tough business, the market maker has to provide a contractual amount of inventory for buyers and sellers at the same time, with the risk that these assets can inadvertently increase or decrease in value. Market makers get a premium (the spread) in exchange for providing liquidity to market takers, taking the risk of holding that inventory for them.
Order flow: HFTs are usually more knowledgeable than managers (and even brokers) to perform trade execution. Remember, the price of an asset does not move "by itself". What makes the price of an asset _really_ increase is an imbalance between the size of buys and sells, causing the spread to be crossed increasingly in one direction, and eating higher and higher price levels => a market participant buying shares is self realizing the price increase. That is called market impact, and that is something nobody wants. In an ideal world, nobody would know the actual liquidity that other participants are willing to exchange on the market, that way no arbitrage would be possible, no market impact, everyone is happy. The problem though is that you need _some_ level of visibility otherwise you hinder so called "price discovery", and regulators don't like that because it's an open door for scams, hidden fees, etc. So a lot of investors just delegate the actual order passing to HFTs: you agree on a price you're happy with, and the HFT executes the orders for you. If they manage to get a better price: good for them, they can pocket the difference. If they didn't, they will have to eat the cost and do better next time. Again, it's a _difficult job_, because by very definition managers are going to try to give you _toxic flows_: if someone mandates you to buy 100 shares of Microsoft, it's because they think the price will go up, but you will quote them at the current stock price. I don't think anyone is negatively impacted by that business to be honest. I am quite happy to instantly do a trade at a known price and let an HFT take the risk of executing trades, finding liquidity, etc.
Arbitrage: This may be the most well known HFT strategy. Essentially benefiting from price disparity of more or less fungible assets. So if HSBC is priced the equivalent of $100 on the LSE and $101 on NYSE, they buy there and sell here. I don't think there is much debate on whether that's a good thing or not, everyone should be happy that prices are more or less consistent around the world. Most people also wouldn't want to bear the overnight nor settlement/FX risk of holding these.
Event signals: I guess that's what people overreact to, even though this is but a tiny and negligible part of the HFT businesses. There are strategies where the whole concept is just to be the first to react to some kind of event, and yes HFTs are the best placed to do such strategies. The thing is, this is not a very lucrative business for a simple reason, which I already stated before: at any point in time, the orderbook contains but a mere fraction of the available liquidity. And it takes *ample time* for the market impact of a trade to be absorbed. There are a lot of papers on the subject, but without entering in too much details: the bigger your order, the longer it will take for the book to absorb it and liquidity to come back. For most hedge funds doing statistical arbitrage, CTAs, etc, we are talking multiple *days*. So realistically, if you're in the business of reacting the fastest to some kind of news or event, there's really not much more you can do than cross the spread and eat one or two levels of the book. Even for the most liquid of US stocks, that's a couple hundred thousands dollar per trade on average, for a tiny return difference versus executing a couple of seconds later, so I hope you find a lot of these if you plan on getting rich.
TLDR, I think 99% of HFT hate is completely unjustified. They provide a legitimate service to most market participants, and as an investor I wouldn't want to do their job, nor would I think that they hinder in any way my investments, on the contrary.
[deleted]
I think it would also be useful to distinguish between high frequency and low latency trading in these sorts of discussions. The groups of traders that do actual high frequency are not the same ones that are doing shenanigans with things like speculative Ethernet frame invalidation (at least on the same engine).
Can you explain why we need nanosecond trading for this?
Why can't we use trading speeds such that everybody can participate, not just a few "gatekeepers"?
The average internet connection should be fast enough for trading, by now.
Why can't we use trading speeds such that everybody can participate, not just a few "gatekeepers"?
The average internet connection should be fast enough for trading, by now.
> Can you explain why we need nanosecond trading for this?
Nanosecond trading is honestly a bit unnecessary. There's academic studies somewhere that if I remember correctly showed little change in benefit to the stability of the market past tens of microseconds I think.
Now the reason why these ultra-high speeds in general are beneficial is because they make collaboration at scale or frontrunning information increasingly difficult the lower you make the trading latency.
This means that while your market makers like Jane Street may integrate outside data for predicting broad trends, the core algorithms for resolving spreads boil down to "what is the most efficient way to match these together" with little to no room for manipulation.
Market manipulation can cause volatility but your HFT market makers are going to eat that volatility at an incredible rate. Massive panics that would have triggered chaos in the streets and people dumping their 401ks get reacted to and their prices factored in within moments and as such prices adjust gradually rather than spiking and cratering and spiking again. HFT takes the blood out of the water so to speak.
Or as another example, HFT market makers act like a loosely attached mass (ex: heavy weighted chains) damping a piece of vibrating industrial equipment. Without the mass damping the system and actively pulling energy out of the vibrations reflecting through the chains, the equipment would rattle and rattle and more than likely the bolts fixing it to the ground would start to work their way loose. And once they are loose, vibrations can start to amplify into oscillations (doubly so if it's a "smart" system involving humans where you experience PIOs) and the stress on the system grows greater and greater. With those chains however each and every little vibration gets nearly any excess energy sucked out of it by the irregular reflections produced within the chains. That loss of energy dampens the vibrations down and while the machine is unimpeded in performing its function inside of itself, the excess energy in the form of vibrations, etc is heavily suppressed or effectively eliminated entirely helping reduce failures, stresses, and wear.
(Sorry for the overworded analogy but the mass-vibration damping analogy is really quite fitting to the way markets operate).
With the average internet latency (say 50-100ms) you can game the system and gain information early with expensive high speed links sharing the news around the world in a way that has near zero value due to diminishing returns w/ lower latency. Especially once light comes into play. If the competitive latency is on the order of distances light can travel within a country, that means once you factor out the essential parts that can't be cut out/sped up, what's left is a very small sliver that can be split between algorithms to make as efficient of trades as possible, algorithms to try and game the system, and integrating knowledge from distances proportional to the time it takes for light to travel it.
In other words at that speed you just can't really play fuck fuck games. The winning move is finding better local strategies (vs global strategies like market manipulation or exploiting information before it propagates).
/rant (sorry I should probably edit this down because I repeat myself a lot here but it's late and i'm tired so i'm just sending it.
Nanosecond trading is honestly a bit unnecessary. There's academic studies somewhere that if I remember correctly showed little change in benefit to the stability of the market past tens of microseconds I think.
Now the reason why these ultra-high speeds in general are beneficial is because they make collaboration at scale or frontrunning information increasingly difficult the lower you make the trading latency.
This means that while your market makers like Jane Street may integrate outside data for predicting broad trends, the core algorithms for resolving spreads boil down to "what is the most efficient way to match these together" with little to no room for manipulation.
Market manipulation can cause volatility but your HFT market makers are going to eat that volatility at an incredible rate. Massive panics that would have triggered chaos in the streets and people dumping their 401ks get reacted to and their prices factored in within moments and as such prices adjust gradually rather than spiking and cratering and spiking again. HFT takes the blood out of the water so to speak.
Or as another example, HFT market makers act like a loosely attached mass (ex: heavy weighted chains) damping a piece of vibrating industrial equipment. Without the mass damping the system and actively pulling energy out of the vibrations reflecting through the chains, the equipment would rattle and rattle and more than likely the bolts fixing it to the ground would start to work their way loose. And once they are loose, vibrations can start to amplify into oscillations (doubly so if it's a "smart" system involving humans where you experience PIOs) and the stress on the system grows greater and greater. With those chains however each and every little vibration gets nearly any excess energy sucked out of it by the irregular reflections produced within the chains. That loss of energy dampens the vibrations down and while the machine is unimpeded in performing its function inside of itself, the excess energy in the form of vibrations, etc is heavily suppressed or effectively eliminated entirely helping reduce failures, stresses, and wear.
(Sorry for the overworded analogy but the mass-vibration damping analogy is really quite fitting to the way markets operate).
With the average internet latency (say 50-100ms) you can game the system and gain information early with expensive high speed links sharing the news around the world in a way that has near zero value due to diminishing returns w/ lower latency. Especially once light comes into play. If the competitive latency is on the order of distances light can travel within a country, that means once you factor out the essential parts that can't be cut out/sped up, what's left is a very small sliver that can be split between algorithms to make as efficient of trades as possible, algorithms to try and game the system, and integrating knowledge from distances proportional to the time it takes for light to travel it.
In other words at that speed you just can't really play fuck fuck games. The winning move is finding better local strategies (vs global strategies like market manipulation or exploiting information before it propagates).
/rant (sorry I should probably edit this down because I repeat myself a lot here but it's late and i'm tired so i'm just sending it.
> Can you explain why we need nanosecond trading for this?
Can you explain why you wouldn't want nanosecond for this?
I am quite happy to buy shares where it's the most convenient for me, knowing that someone, somewhere, is constantly arbitraging stock prices on all markets so that all prices are the same regardless.
What upside is there for me, as an investor, that this happens every minute instead of every nanosecond? None, it's objectively worse.
> Why can't we use trading speeds such that everybody can participate, not just a few "gatekeepers"?
But that makes no sense... HFTs are not _competing_ with regular market participants, on the contrary they are providing a service to them. The only competitor to HFTs are other HFTs, because, well, they are providing the same service.
If you want to "participate" in something and you feel hindered by HFTs, that basically means you are not a typical investor, but rather a market maker / arbitrager, so then... competition is fair game, have fun.
Your argument is akin to "we should limit the speed of UPS trucks because I want to compete but I cannot afford to buy fancy trucks". UPS does not prevent the normal operation of package delivery, nor are they competing with people receiving packages. They are only competing with other delivery companies.
> The average internet connection should be fast enough for trading, by now.
Well even if we were to put latency aside, no, your internet connection won't cut it anyway, at least not for any kind of direct market access.
First, not anybody can "do DMA". There are a vast quantity of regulations, licenses, tests, audits, and fees to be an exchange participant. Regulators don't let just anyone participate on public exchanges, even on emerging markets.
1) Be a regulated entity, pass individual certifications, have a risk, legal and compliance team, yearly assessments, etc.
2) Pay a lot of money for access to the exchange market data live ($10k-$100k/month/exchange).
3) Have all your algos certified, risk assessed, and staged on the scenarios the exchange will provide you with.
4) There is no multicast on internet, and anyway unless you have a Solarflare/Mellanox NIC at home, and a crazy good ISP, you won't ingest 10Gb linerate that simply.
5) You need all the things around the ecosystem... symbology, security master, etc...
6) Even cash equities need to be settled, you will need a custodian and a back office, stocks are not tokens that magically change hands. Someone need to post cash, update company records, custody collateral, handle corporate actions, etc
There is no world in which a _regular_ investor will touch the markets directly anyway, everybody goes through a broker or deals with HFTs for order flow, and it's not "just because they are faster", there is a world of complexity/regulation/etc that goes with it.
Even most of the top hedge funds in the world are not direct market participants, but rather use brokers.
The very reason multistrat funds even exist and are so attractive, is that you can net your trades internally at mid and avoid even having to use lit orderbooks, reducing your market impact.
So, no, I don't think anyone sees HFTs as a hindrance, nor are willing to be market participants. This sounds sounds a bit like a naive simplification of the state of things (I don't mean it derisively, it's not a trivial subject).
Can you explain why you wouldn't want nanosecond for this?
I am quite happy to buy shares where it's the most convenient for me, knowing that someone, somewhere, is constantly arbitraging stock prices on all markets so that all prices are the same regardless.
What upside is there for me, as an investor, that this happens every minute instead of every nanosecond? None, it's objectively worse.
> Why can't we use trading speeds such that everybody can participate, not just a few "gatekeepers"?
But that makes no sense... HFTs are not _competing_ with regular market participants, on the contrary they are providing a service to them. The only competitor to HFTs are other HFTs, because, well, they are providing the same service.
If you want to "participate" in something and you feel hindered by HFTs, that basically means you are not a typical investor, but rather a market maker / arbitrager, so then... competition is fair game, have fun.
Your argument is akin to "we should limit the speed of UPS trucks because I want to compete but I cannot afford to buy fancy trucks". UPS does not prevent the normal operation of package delivery, nor are they competing with people receiving packages. They are only competing with other delivery companies.
> The average internet connection should be fast enough for trading, by now.
Well even if we were to put latency aside, no, your internet connection won't cut it anyway, at least not for any kind of direct market access.
First, not anybody can "do DMA". There are a vast quantity of regulations, licenses, tests, audits, and fees to be an exchange participant. Regulators don't let just anyone participate on public exchanges, even on emerging markets.
1) Be a regulated entity, pass individual certifications, have a risk, legal and compliance team, yearly assessments, etc.
2) Pay a lot of money for access to the exchange market data live ($10k-$100k/month/exchange).
3) Have all your algos certified, risk assessed, and staged on the scenarios the exchange will provide you with.
4) There is no multicast on internet, and anyway unless you have a Solarflare/Mellanox NIC at home, and a crazy good ISP, you won't ingest 10Gb linerate that simply.
5) You need all the things around the ecosystem... symbology, security master, etc...
6) Even cash equities need to be settled, you will need a custodian and a back office, stocks are not tokens that magically change hands. Someone need to post cash, update company records, custody collateral, handle corporate actions, etc
There is no world in which a _regular_ investor will touch the markets directly anyway, everybody goes through a broker or deals with HFTs for order flow, and it's not "just because they are faster", there is a world of complexity/regulation/etc that goes with it.
Even most of the top hedge funds in the world are not direct market participants, but rather use brokers.
The very reason multistrat funds even exist and are so attractive, is that you can net your trades internally at mid and avoid even having to use lit orderbooks, reducing your market impact.
So, no, I don't think anyone sees HFTs as a hindrance, nor are willing to be market participants. This sounds sounds a bit like a naive simplification of the state of things (I don't mean it derisively, it's not a trivial subject).
> They are only competing with other delivery companies.
I don't think this is true. They are trading on the same market as everybody else. People used to make money buying in market A and selling in market B. Now HFT has mostly taken over that role, for instance.
Anyway, finance has created a shitload of abstraction, and at this point it isn't clear if "normal" people benefit from this or that it's just the incumbents.
To make another analogy, yes, feudal lords provided some benefits to the peasants, such as safety and shelter. But you really have to step away from the entire construct to see if they are really better off. This is what is missing in your argumentation, imho.
I don't think this is true. They are trading on the same market as everybody else. People used to make money buying in market A and selling in market B. Now HFT has mostly taken over that role, for instance.
Anyway, finance has created a shitload of abstraction, and at this point it isn't clear if "normal" people benefit from this or that it's just the incumbents.
To make another analogy, yes, feudal lords provided some benefits to the peasants, such as safety and shelter. But you really have to step away from the entire construct to see if they are really better off. This is what is missing in your argumentation, imho.
> People used to make money buying in market A and selling in market B. Now HFT has mostly taken over that role, for instance
Who is "People"?
Well now, I cannot claim to know what everyone did at any point in time, but allow me to be very doubtful that anyone other than a dedicated arbitrager/market maker would do that.
Or at least, I cannot conceivably see how one would do that and turn a profit without being also a market maker.
The economics of arbitraging are just too tight.
You need to spot and act on a cross exchange spread that is wider that:
- The exchange fees, for which market makers have rebates. So even at equal technological footing, MM will have an advantage over "People".
- The cost of funding, for which you will need to post IM, and thus already be a professional investor with a good amount of capital.
- If that was "before HFTs", then it was before odd lots as well. Now I don't know what the average price of a US share is, but let's say a couple hundreds of dollars, times 100 lot size. These "People" better be filthy rich in cash.
- Arbitraging at a couple of price levels (which is already a lot), minus costs, would yield you a couple bips per trade. To make any kind of real money doing that, you better do it a trillion times per day, and thus lock a huge amount of money.
- The market risk over the arbitraging period
I don't think I have seen anyone turn a profit arbitraging naively in the last 15 years that were not some kind of market maker already, or some ad-hoc strategy on very ad-hoc EM markets.
Who is "People"?
Well now, I cannot claim to know what everyone did at any point in time, but allow me to be very doubtful that anyone other than a dedicated arbitrager/market maker would do that.
Or at least, I cannot conceivably see how one would do that and turn a profit without being also a market maker.
The economics of arbitraging are just too tight.
You need to spot and act on a cross exchange spread that is wider that:
- The exchange fees, for which market makers have rebates. So even at equal technological footing, MM will have an advantage over "People".
- The cost of funding, for which you will need to post IM, and thus already be a professional investor with a good amount of capital.
- If that was "before HFTs", then it was before odd lots as well. Now I don't know what the average price of a US share is, but let's say a couple hundreds of dollars, times 100 lot size. These "People" better be filthy rich in cash.
- Arbitraging at a couple of price levels (which is already a lot), minus costs, would yield you a couple bips per trade. To make any kind of real money doing that, you better do it a trillion times per day, and thus lock a huge amount of money.
- The market risk over the arbitraging period
I don't think I have seen anyone turn a profit arbitraging naively in the last 15 years that were not some kind of market maker already, or some ad-hoc strategy on very ad-hoc EM markets.
> I don't think this is true. They are trading on the same market as everybody else. People used to make money buying in market A and selling in market B. Now HFT has mostly taken over that role, for instance.
They aren't though that's kind of the thing. HFT Market Makers (i.e. the big HFT firms) aren't just trading in the same market. They are the market. They coordinate the trades and effectively instantly sink every buy and sell into themselves and handle the matching of those spreads.
The market makers are competing against each other to offer access to the market to you.
They aren't though that's kind of the thing. HFT Market Makers (i.e. the big HFT firms) aren't just trading in the same market. They are the market. They coordinate the trades and effectively instantly sink every buy and sell into themselves and handle the matching of those spreads.
The market makers are competing against each other to offer access to the market to you.
How's that? HFT lowers the barrier of entry for ordinary participants who just want to invest their savings either directly or through something like a pension fund, because they get all those benefits I mentioned, and those benefits are useful when you are only executing a trade once a month or year.
It only increases the barrier to entry to people who want to do HFT themselves on a smaller scale, because they now can't compete on latency etc. But it's a bit of a circular argument to say that it's a bad thing that engaging in HFT increases the barrier of entry to HFT.
In any case, and to reiterate, I wasn't refuting a claim that HFT is a bad thing, I was refuting your original claim that it's pointless. For something to be pointless, there has to be no purpose to it whatsoever, which means it has to have no benefits.
It only increases the barrier to entry to people who want to do HFT themselves on a smaller scale, because they now can't compete on latency etc. But it's a bit of a circular argument to say that it's a bad thing that engaging in HFT increases the barrier of entry to HFT.
In any case, and to reiterate, I wasn't refuting a claim that HFT is a bad thing, I was refuting your original claim that it's pointless. For something to be pointless, there has to be no purpose to it whatsoever, which means it has to have no benefits.
We all know that middlemen are bad.
We don't need an HFT middleman.
We don't need an HFT middleman.
actually you do need another participant to take the other side of your trade. That’s not a middle man. Market makers, liquidity providers, are a key component of these markets, taking risk with their own capital (no systemic risk, no too big to fail, going to get a government is things go badly), requiring sophistication and specialization to survive in an ultra competitive environment with a high degree of uncertainty and risk exposure.
How is HFT a middle man? We're not talking about things like market makers in this discussion
If you're a normal market participant, HFT companies mainly make it harder for someone else to take more money off of the top of your underlying transaction (and you are, in general, going to be paying someone for that). HFT almost completely replaced hoards of the stereotypical wall street traders and I don't really see many people shedding tears for them.
It's pointless in terms of the skill wasted that could be used for actual cool things. But I agree that more market participants is better. And whether I'm buying or selling, I always want someone to trade with, vs not.
High frequency trading is far from the only pointless activity software and hardware developers are employed to do. At least in that realm they are pushing the state of the art, unlike presenting ads or yet another crud app.
Pushing the state of the art? That's only meaningful if the results are published. But I suspect that everything is kept secret.
Come on, it is not difficult to find better things to do with your brains. Like work in medicine, etc.
Come on, it is not difficult to find better things to do with your brains. Like work in medicine, etc.
You do realise that Jane Street (one of the big HFT market makers) publishes basically all their technical work and contributes to massive amounts of open source software? They are to my knowledge one of the main contributors to the OCaml ecosystem as well.
Will the grandchildren be able to hear you over the roar of the yacht?
Meh, at least HFT is making sure markets are efficient.
Now imagine telling your grandkids you worked on pushing short form content in absolutely every app to maximize engagement and rot brains?
Now imagine telling your grandkids you worked on pushing short form content in absolutely every app to maximize engagement and rot brains?
[deleted]
“You’re welcome for the education and good home full of books.”
less than 60ns loopback latency
There are some Ethernet switches with 4ns latency, and those do more than just sending and receiving, so there's clearly still an order of magnitude of improvement still available. 4ns is basically ~40 cycles of the bit clock for 10G Ethernet.
There are some Ethernet switches with 4ns latency, and those do more than just sending and receiving, so there's clearly still an order of magnitude of improvement still available. 4ns is basically ~40 cycles of the bit clock for 10G Ethernet.
It's a bit more complicated than that. Loopback implies that the data went off the wire and was sent back. That puts some serious constraints on the process that deals with that data, how it is stored and how it is fed back into the system. You can't just compare that with naked pass-through latency (which is what a switch would do), because the signal is already on the way out before it has even fully been received.
Essentially a few ns after the header is received it will already be passed on to the output port after some signal conditioning, which happens almost without latency. A typical 'high speed' switch will have 800 ns round trip, so that 60 ns quoted here is actually quite impressive.
Essentially a few ns after the header is received it will already be passed on to the output port after some signal conditioning, which happens almost without latency. A typical 'high speed' switch will have 800 ns round trip, so that 60 ns quoted here is actually quite impressive.
There are L1 crosspoint "switches" in the 2-5ns range depending on port density and similar modes in some Ethernet silicon. These are not Ethernet switches in any normal sense though. They only replicate the signal 1:1 or 1:n and do not dynamically switch the destination based on anything in the packet. The fastest L3 Ethernet switches on the market are ~90ns.
Are L1 switches like hubs or repeaters then? Sorry for my confusion.
They're more like digital patch panels. You can connect any pair of ports together in both directions (TX/RX), just like moving a patch cable around. You can also connect an incoming RX lane to multiple TX lanes (think optical splitter but electrical). You cannot merge signals together. Some products have FPGAs that add L2+ or multiplexing functionality with packet buffers, but that's not part of the 2-5ns path.
So, if I know the destination addresses on each port in advance I can program it to behave like an L2 switch (with proper multicast even), and I can use code to reprogram it whenever the traffic pattern changes?
The L1 path can't do any address matching and you can't overlap sources - like Ghostbusters, no crossing the streams. You can source port 1 to ports 2-4 (multicast, but static and unidirectional) and you can change a bidir connection between pairs of ports. The signal gets replicated regardless of what's in the frame. It wouldn't be practical to make it behave quite like an L2 switch.
Apologies, my post was not precise enough, but I think I got it.
> It wouldn't be practical to make it behave quite like an L2 switch.
What are typical configuration? If you’re allowed to talk about it.
> It wouldn't be practical to make it behave quite like an L2 switch.
What are typical configuration? If you’re allowed to talk about it.
No they don’t. At least not on the critical path. L1 switch’s at 4ns simply replicate the incoming electrical signal. That’s all they do.
They also do some packet accounting and cute features off the critical path. None of that at 4ns.
They also do some packet accounting and cute features off the critical path. None of that at 4ns.
In fairness, assuming 4/8 ports in, 4 ports out, operating at some ungodly GHz using a custom GaAs or SiGe chip, and working in gearboxed scrambled space with very clever input mac prefix mapping and output scrambled/gearboxed precomputation, one _could_ do around 8ns optimistically from the start of the first 64b/66b block (the 0x55... preamble) for 10GB. There's some stuff about preamble shrink that makes it wonky, but that is doable. And interestingly enough, for 25G-R, one can comfortably do this in about 4ns. I am not in fact aware of such a beastie existing for 25G, but I have seen 1 or 2 for 10G though.
Surprisingly, if ChatGPT is prompted _juust_ right, it will even give you a good way to do this.
Surprisingly, if ChatGPT is prompted _juust_ right, it will even give you a good way to do this.
The other funny bit is that one-way PCIe latency is 250ns-ish (don't quote me on the exact numbers), which imposes a hard 1us constraint on latency between two hosts.
You can go quicker with CXL, but not by much.
There are always DPUs like NextSilicon, NVidia's Blue*, etc - at under 100 ns of SoF to fast compute though.
> There are some Ethernet switches with 4ns latency […]
E.g.,
* https://www.arista.com/en/products/7130-connect
* https://www.arista.com/en/products/7130-series
E.g.,
* https://www.arista.com/en/products/7130-connect
* https://www.arista.com/en/products/7130-series
Cut Through switching needs what, 14 bytes? Thats ~12ns on 10G? How would you do it in 40 cycles?
why do you assume the full ethernet header?
IIRC. cut-through only needs the first 6 bytes. Since it only needs the destination address for the port lookup.
Potentially the first bit, on broadcast.
IIRC. cut-through only needs the first 6 bytes. Since it only needs the destination address for the port lookup.
Potentially the first bit, on broadcast.
Because you might want VLAN. Plus you cant just start blasting reply without a preamble, so its still 14 bytes after receiving just the dest MAC. Then you get potential IFG, FEC, scrambling, it all adds up, no way any switch can do 4ns without heavy lawyer talk in the small print.
Not my are of expertise, but afaik the neat part is that you don't do it in cycles. The "program" lives as the transistor layout, not as instructions
The cycles in question are not those of any processor, but those of the signal coming in the wire. No matter how it's implemented, a switch cannot react to bits that are still in the wire and haven't made it to the switch yet.
For switching, this is just not true. There's in/out solutions that are at this kind of scale, but they are by definition not implementing any switching logic.
> the design has less than 60ns loopback latency
What is the current state of the art figures?
What is the current state of the art figures?
I can see the guy is bored to death at Jane Street.
That's so cool.
LinkedIn tells me: https://uk.linkedin.com/in/ttchisholm
FPGA engineer with a focus on ultra-low latency networking at Jane Street.
Not sure i get the implication, something wrong with the company?
I’m not the parent, but I’d assume that this was an impressed ‘yikes.’ While Jane is not the most well known firm for ultra low latency trading, an FPGA engineer from that firm could be considered an expert practitioner in the field.
Some people just love dropping “yikes” type comments and then never explaining themselves.
A few of those same cringe “yikes” users appear to have downvoted my comment too. Truly a strange bunch.
Is this really unexpected? A person who knows ultra-low-latency networking with FPGAs writes an article about making low-latency networking hardware for a company that needs high-speed, low-latency communications (high-frequency trading)