Challenges and Research Directions for Large Language Model Inference Hardware(arxiv.org)
arxiv.org
Challenges and Research Directions for Large Language Model Inference Hardware
https://arxiv.org/abs/2601.05047
25 comments
> To address these challenges, we highlight four architecture research opportunities: High Bandwidth Flash for 10X memory capacity with HBM-like bandwidth; Processing-Near-Memory and 3D memory-logic stacking for high memory bandwidth; and low-latency interconnect to speedup communication.
High Bandwidth Flash (HBF) got submitted 6 hours ago! It's a great article, fantastic coverage of a wide section of the rapidly moving industry. https://news.ycombinator.com/item?id=46700384 https://blocksandfiles.com/2026/01/19/a-window-into-hbf-prog...
HBF is about having many dozens or hundreds of channels of flash memory. The idea of having Processing Near HBF, spread out, perhaps in mixed 3d design, would be not at all surprising to me. One of the main challenges for HBF is building improved vias, improved stacking, and if that tech advanced the idea of more mixed NAND and compute layers rather than just NAND stacks perhaps opens up too.
This is all really exciting possible next steps.
High Bandwidth Flash (HBF) got submitted 6 hours ago! It's a great article, fantastic coverage of a wide section of the rapidly moving industry. https://news.ycombinator.com/item?id=46700384 https://blocksandfiles.com/2026/01/19/a-window-into-hbf-prog...
HBF is about having many dozens or hundreds of channels of flash memory. The idea of having Processing Near HBF, spread out, perhaps in mixed 3d design, would be not at all surprising to me. One of the main challenges for HBF is building improved vias, improved stacking, and if that tech advanced the idea of more mixed NAND and compute layers rather than just NAND stacks perhaps opens up too.
This is all really exciting possible next steps.
Why is persistence such a big thing here? Non-flash memory just needs a tiny bit of power to keep its data. I don't see the revolutionary usecase.
Density is the key here, not persistence.
Thanks! This explains it.
Now I'm wondering how you deal with the limited number of write cycles of Flash memory. Or maybe that is not an issue in some applications?
Now I'm wondering how you deal with the limited number of write cycles of Flash memory. Or maybe that is not an issue in some applications?
During inference, most of the memory is read only.
Sounds fair. That's not the kind of machine I'd want as a development system though. And usually development systems are beefier than production systems. So curious how they'd solve that.
Yeah, it is quite specialized for inference. It's unlikely that you'd see this stuff outside of hardware specifically for that.
Development systems for AI inference tend to be smaller by necessity. A DGX Spark, Station, a single B300 node... you'd work on something like that before deploying to a larger cluster. There's just nothing bigger than what you'd actually deploy to.
Development systems for AI inference tend to be smaller by necessity. A DGX Spark, Station, a single B300 node... you'd work on something like that before deploying to a larger cluster. There's just nothing bigger than what you'd actually deploy to.
HBF, like expensive HBM, is targeted at AI data centers.
The KAIST professor discussed an HBF unit having a capacity of 512 GB and a 1.638 TBps bandwidth.
Weird to see no mention in this paper of persistent memory technologies beyond NAND flash. Some of them, like ReRAM, also enable compute-in-memory which the authors regard as quite important.
Why not, instead of passing the entire model through a processor and running it on every bit of data, pass the data (which is much smaller) through the model? As in, have compute and memory together in the silicon. Then you only need to shuffle the data itself around (perhaps by broadcast) rather than the entire model. That seems like it would use a LOT less energy.
Or is it not possible to make the algorithms parallel to this degree?
Edit: apparently this is called "compute-in-memory"
Or is it not possible to make the algorithms parallel to this degree?
Edit: apparently this is called "compute-in-memory"
This is done that way at the GPU layer of abstraction - generally (with some exceptions!) the model lives in GPU vram, and you stream the data batch by batch through the model.
The problem is that for larger models the model barely fits in VRAM, so it definitely doesn't fit in cache.
Dataflow processors like cerebras do stream the data through the model (for smaller models at least, or if they can have smaller portions of models) - each little core has local memory and you move the data to where it needs to go. To achieve this though, Cerebras has 96GB of what is basically L1 cache among its cores, which is... a lot of SRAM.
The problem is that for larger models the model barely fits in VRAM, so it definitely doesn't fit in cache.
Dataflow processors like cerebras do stream the data through the model (for smaller models at least, or if they can have smaller portions of models) - each little core has local memory and you move the data to where it needs to go. To achieve this though, Cerebras has 96GB of what is basically L1 cache among its cores, which is... a lot of SRAM.
Designing a concept sustainable RAM product and in working around multiplexing scaling challenges I somewhat accidentally developed a potential solution for hosting already-trained LLMs with very low energy and hardware in carbon and lignin;
> You have effectively designed a Diffractive Deep Neural Network (D^2NN) that doubles as a storage device.
Mode Division Multiplexing (MDM) via OAM Solitons potentially with gratings designed with Inverse Design of a Transition Map to be lasered possibly with a Galvo Laser. This would be a very low power way to run LLMs; on a lasered substrate
> You have effectively designed a Diffractive Deep Neural Network (D^2NN) that doubles as a storage device.
Mode Division Multiplexing (MDM) via OAM Solitons potentially with gratings designed with Inverse Design of a Transition Map to be lasered possibly with a Galvo Laser. This would be a very low power way to run LLMs; on a lasered substrate
In-memory processing:
https://en.wikipedia.org/wiki/In-memory_processing
Computational RAM: https://en.wikipedia.org/wiki/Computational_RAM
Computational RAM: https://en.wikipedia.org/wiki/Computational_RAM
Frontier models are now much bigger than an individual query, hence batching, MoE, etc. So this idea, while very plausible, has economic constraints, you'd need vast amounts of memory.
[deleted]
[deleted]
Yes, this is the #2 direction recommended by the paper. Do you have arguments re "Table 4 lists why PNM is better than PIM for LLM inference, despite weaknesses in
bandwidth and power" ?
There are advantages, I suppose it comes down to economics and which of the advantages/disadvantages are greater. Probably if PIM was to ever catch on, it'd start off in mobile devices where energy efficiency is a high priority. Still might be impractical though.
Yup, reads like the executive summary (in a good way).
Can’t we credit the first author in the title too? Come on.
The current title uses 79 characters of 80 character budget:
75% = title written by first author
22% = name of second author, endorsing work of first author
No we can't, that would be a crime against royalty :)
That appendix of memory prices looks interesting, but misses the recent trend.
Several years ago I was at one of the Berkley AMP Lab retreats at Asilomar, and as I was hanging out, I couldn't figure how I know this person in front of me, until an hour later when I saw his name during a panel :)).
It was always the network. And David Patterson, after RISC, started working on iRAM, that was tackling a related problem.
NVIDIA bought Mellanox/Infiniband, but Google has historically excelled at networking, and the TPU seems to be designed to scale out in the best possible way.