AMD EPYC 9004 Genoa Under-the-Lid – ServeTheHome(servethehome.com)
servethehome.com
AMD EPYC 9004 Genoa Under-the-Lid – ServeTheHome
https://www.servethehome.com/amd-epyc-9004-genoa-under-the-lid/
24 comments
Accounting: Hardware and software acquisition is very different at an accounting level in large orgs, and come from different budgets. Licensing and support can be easier to procure and plan for than hardware.
APIs/RBAC: Cloud infrastructure-as-code for block storage, object storage, DB, network, VM and serverless with the same security policy language accessible via HTTP is really incredible. I wish there were an open-source cloud equivalent, but I don't know of one that can do 20% of what AWS can do.
APIs/RBAC: Cloud infrastructure-as-code for block storage, object storage, DB, network, VM and serverless with the same security policy language accessible via HTTP is really incredible. I wish there were an open-source cloud equivalent, but I don't know of one that can do 20% of what AWS can do.
> I don't know of one that can do 20% of what AWS can do.
We only use a few AWS products: Route53, EC2, VPC, EBS, and S3/Glacier. This feels like <20% of what AWS can do today.
Are these use cases not yet covered? I'd be OK with Route53 not being in the box, but surely we have VMs, storage and backups sorted by now, right?
We only use a few AWS products: Route53, EC2, VPC, EBS, and S3/Glacier. This feels like <20% of what AWS can do today.
Are these use cases not yet covered? I'd be OK with Route53 not being in the box, but surely we have VMs, storage and backups sorted by now, right?
How do you deploy your applications onto your EC2 instances? Do you create application-specific AMIs, use something like Ansible, something Windows-specific like WinRM, run your own container setup on EC2, or something else?
Our software has integral CI/CD capabilities. All built in-house. Once our services are installed on a machine, they can be upgraded from a centralized system.
First time setup is manual, but it could be automated if we got bored of the task. The binary distribution requires zero dependencies on the machine and brings its own database with it so bootstrapping is very easy.
First time setup is manual, but it could be automated if we got bored of the task. The binary distribution requires zero dependencies on the machine and brings its own database with it so bootstrapping is very easy.
Keep in mind that you need to put this in a server with DIMMs, power supplies, and disks. It will likely come out to $20-30k all in for one server depending on how much margin your system integrator wants. You probably want two in case the first one fails, and you may want as many as 3-4 for true 24/7 availability, plus a backup service.
The math still doesn't look too good for the cloud, though.
The math still doesn't look too good for the cloud, though.
> putting the entire company on 1 socket
What about availability? Do you mean 1 socket, plus a hot backup in a separate location in case of hardware failure or some catastrophe?
IIRC, Hacker News runs on 1 machine plus a hot backup.
What about availability? Do you mean 1 socket, plus a hot backup in a separate location in case of hardware failure or some catastrophe?
IIRC, Hacker News runs on 1 machine plus a hot backup.
Most of the time 'availability' means network connectivity. Most of the time you can throw resources in the processing.
Does anyone know how they place the chiplets on to the blue substrate? It must be incredibly precise, presumably?
This doc has some info about current pitch size for the "bump map" of where the chiplets connect to the substrate.
https://www.opencompute.org/documents/odsa-openhbi-l-v1-0-rc...
Page 38 says "OpenHBI-L specifies the minimum bump pitch (min) to range from 110um to 130um for manufacturability considerations".
Edit: Also related and interesting...
"One of the key points AMD spoke about...is how each of the silicon dies are attached to the package. In order to enable a pin-grid array desktop processor, the silicon has to be affixed to the processor in a BGA fashion. AMD stated that due to the 7nm process, the bump pitch ...reduced from 150 microns on 12nm to 130 microns on 7nm. This doesn’t sound like much, however AMD stated that there are only two vendors in the world with technology sufficient to do this."
https://www.anandtech.com/show/14525/amd-zen-2-microarchitec...
https://www.opencompute.org/documents/odsa-openhbi-l-v1-0-rc...
Page 38 says "OpenHBI-L specifies the minimum bump pitch (min) to range from 110um to 130um for manufacturability considerations".
Edit: Also related and interesting...
"One of the key points AMD spoke about...is how each of the silicon dies are attached to the package. In order to enable a pin-grid array desktop processor, the silicon has to be affixed to the processor in a BGA fashion. AMD stated that due to the 7nm process, the bump pitch ...reduced from 150 microns on 12nm to 130 microns on 7nm. This doesn’t sound like much, however AMD stated that there are only two vendors in the world with technology sufficient to do this."
https://www.anandtech.com/show/14525/amd-zen-2-microarchitec...
I'm a digital physical design engineer.
Even though the 7nm number is kind of meaningless these days it is still useful to compare the scale.
The flip chip bumps on the top metal layer interface to the outside world. Then they get distributed out to the pins of the package which connect to the socket on the motherboard.
Socket AM4 has 1331 pins. Socket SP5 for the Epyc has 6096 pins. I've worked on some chips with over 10,000 bump/pins.
A modern chip has over 16 metal layers, with multiple via layers in between and probably another 40-60 base layers with the transistors. The technology to align the masks for each layer is on the nanometer level. In contrast the bumps for the chip to package are in the hundreds of micron level.
If we just go with 130 microns / 7 nanometers = 18,571 less precise for the alignment of the package vs alignment of the next mask layer while making the chip.
So it is certainly impressive to line up chips on the package but nowhere close to the precision for the mask layers
Even though the 7nm number is kind of meaningless these days it is still useful to compare the scale.
The flip chip bumps on the top metal layer interface to the outside world. Then they get distributed out to the pins of the package which connect to the socket on the motherboard.
Socket AM4 has 1331 pins. Socket SP5 for the Epyc has 6096 pins. I've worked on some chips with over 10,000 bump/pins.
A modern chip has over 16 metal layers, with multiple via layers in between and probably another 40-60 base layers with the transistors. The technology to align the masks for each layer is on the nanometer level. In contrast the bumps for the chip to package are in the hundreds of micron level.
If we just go with 130 microns / 7 nanometers = 18,571 less precise for the alignment of the package vs alignment of the next mask layer while making the chip.
So it is certainly impressive to line up chips on the package but nowhere close to the precision for the mask layers
[deleted]
The technology to do this is pretty old, and it's basically just a very precise pick-and-place machine (disclaimer - this is for 3D stacking, not necessarily AMD's thing). The rub is that it doesn't need to be that precise: surface tension from the soldering process will actually correct a small mis-placement as long as you aren't off by too much, so the tolerances are bigger than you think.
Most of those techniques and machinery is behind company secrecy, but yes, you need highly precise and specialized machinery to build that kind of things.
Linus Tech Tips did a tour of the Intel fab factory a few months ago: https://www.youtube.com/watch?v=2ehSCWoaOqQ
Linus Tech Tips did a tour of the Intel fab factory a few months ago: https://www.youtube.com/watch?v=2ehSCWoaOqQ
Maybe, I wonder if this can work like ic placement on pcbs where surface tension can help align things (solder will pull parts into place)
I too would love to know as well!
I too would love to know as well!
Under the lids is.. another layer of lids?
It’s lids all the way down. Still, quite an impressive number of them.
Each pair of lids make up a NAND gate.
Where is Genoa-x? moooarr cache plz.
Later in 2023. We will cover it like we did with Milan-X.
If we were to consolidate all of the processing power across our Azure/AWS VMs, we would not be at even 30% of what this part can handle. You could even throw in the IT stack of our entire parent org with room to spare.
The only reason we are still in the cloud is the deep compliance packages that our customers really enjoy. Even so, management is starting to consider alternatives.
I know a lot of us can't stand the idea, but the notion of putting the entire company on 1 socket is super compelling to me. On-prem turns into a dragon for me when I have to manage 10+ machines and related vendors. 2-3 machines and a handful of vendors is more manageable.
One thing is certain - The various cloud providers will take every ounce of AMD's performance innovation and make it feel like a negligible improvement to their end customers.