I don't know how long someone should expect to remain employed when making baseless allegations of scientific misconduct against his colleagues instead of doing actual work. Again, he did not have evidence to support his suspicion of fraud, and he admitted this at the time.
I'm sorry that the "most important names of the entire floorplanning academic community" are struggling with ML basics, but it is what it is. The "Chip Has Sailed" paper makes this pretty clear. This pattern is unfortunately common when ML comes for a new field -- some researchers adapt and build, and others fail and complain (or worse, don't really even try).
> it's a hoop that everyone who has ever published any such paper (including all the big names) has had to pass in order to be published
If the hoop doesn't match what modern chip design needs, we shouldn't expect researchers to hop through it. No one is comparing Vision Transformers against AlexNet on MNIST. Meanwhile, AlphaChip is already used in production to make real layouts for real chips. TPU is a big deal!
I think the one thing we agree on is that this field desperately needs large public benchmarks that are representative of modern chip design.
The "whistleblower"'s admission that he "did not have evidence to support his suspicion of fraud" is pretty damning. He fails to meet a much, much lower bar than direct observation -- he admitted he had no evidence at all to even support his suspicion of fraud.
> the most damning evidence you could possibly have is that you cannot reproduce the results
Seems like the "whistleblower" didn't even have that. From the paper by the AlphaChip authors: "We provided the committee with one-line scripts that generated significantly better RL results than those reported in Markov et al., outperforming their “stronger” simulated annealing baseline. We still do not know how Markov and his collaborators produced the numbers in their paper."
Synopsys disavowed Markov's paper:
"Regarding the CACM article that Igor Markov's comments and writings do not represent Synopsys views or opinions in any way. Synopsys is also aligned with you on the potential of Reinforcement Learning AI for chip design"
(https://x.com/JeffDean/status/1859431937640665474)
Unfortunately, there aren't publicly available benchmarks for modern technology node sizes, at least not that I'm aware of. Kahng compared on 45nm and 12nm chips, which are very different from a physical design perspective from the 4nm technology node size used by Dimensity 5G, or the sub-10nm technology node size of TPU. "MLContra" used a >100nm technology node size, which is just crazy.
Even if the AlphaChip authors redid Kahng's study properly, this still wouldn't give us useful information -- what matters is AlphaChip's ability to optimize chips in a real-life, production setting, for modern chips, where millions of dollars are on the line.
There's a lot of... passionate discussion in this thread, but we shouldn't lose sight of the big picture -- Google has used AlphaChip in multiple generations of TPU, their flagship AI accelerator. This is a multi-billion dollar project that is strategically critical for the success of the company. The idea that they're secretly making TPUs worse in order to prop up a research paper is just absurd. Google has even expanded their of AlphaChip use to other chips (e.g. Axion).
Meanwhile, MediaTek built on AlphaChip and is using it widely, and announced that it was used to help design Dimensity 5G (4nm technology node size).
I can understand that, when this open-source method first came out, there were some who were skeptical, but we are way beyond that now -- the evidence is just overwhelming.
I'm going to paste here the quotes from the bottom of the blog post, as it seems like a lot of people have missed them:
“AlphaChip’s groundbreaking AI approach revolutionizes a key phase of chip design. At MediaTek, we’ve been pioneering chip design’s floorplanning and macro placement by extending this technique in combination with the industry’s best practices. This paradigm shift not only enhances design efficiency, but also sets new benchmarks for effectiveness, propelling the industry towards future breakthroughs.”
--SR Tsai, Senior Vice President of MediaTek
“AlphaChip has inspired an entirely new line of research on reinforcement learning for chip design, cutting across the design flow from logic synthesis to floor planning, timing optimization and beyond. While the details vary, key ideas in the paper including pretrained agents that help guide online search and graph network based circuit representations continue to influence the field, including my own work on RL for logic synthesis. If not already, this work is poised to be one of the landmark papers in machine learning for hardware design.”
--Siddharth Garg, Professor of Electrical and Computer Engineering, NYU
"AlphaChip demonstrates the remarkable transformative potential of Reinforcement Learning (RL) in tackling one of the most complex hardware optimization challenges: chip floorplanning. This research not only extends the application of RL beyond its established success in game-playing scenarios to practical, high-impact industrial challenges, but also establishes a robust baseline environment for benchmarking future advancements at the intersection of AI and full-stack chip design. The work's long-term implications are far-reaching, illustrating how hard engineering tasks can be reframed as new avenues for AI-driven optimization in semiconductor technology."
--Vijay Janapa Reddi, John L. Loeb Associate Professor of Engineering and Applied Sciences, Harvard University
“Reinforcement learning has profoundly influenced electronic design automation (EDA), particularly by addressing the challenge of data scarcity in AI-driven methods. Despite obstacles including delayed rewards and limited generalization, research has proven reinforcement learning's capability in complex electronic design automation tasks such as floorplanning. This seminal paper has become a cornerstone in reinforcement learning-electronic design automation research and is frequently cited, including in my own work that received the Best Paper Award at the 2023 ACM Design Automation Conference.”
--Professor Sung-Kyu Lim, Georgia Institute of Technology
"There are two major forces that are playing a pivotal role in the modern era: semiconductor chip design and AI. This research charted a new path and demonstrated ideas that enabled the electronic design automation (EDA) community to see the power of AI and reinforcement learning for IC design. It has had a seminal impact in the field of AI for chip design and has been critical in influencing our thinking and efforts around establishing a major research conference like IEEE LLM-Aided Design (LAD) for discussion of such impactful ideas."
--Ruchir Puri, Chief Scientist, IBM Research; IBM Fellow
Unfortunately, commercial EDA companies generally have restrictive licensing agreements that prohibit direct public comparison.
Still, the fact that Google uses it for TPU is pretty telling - this is a multi-billion dollar, mission-critical chip design effort, and there's no way they'd make TPU worse just to prop up a research paper. MediaTek's production use is also a good indicator.
In the blog post, they announce MediaTek's widespread usage, the deployment in multiple generations of TPU with increasing performance each generation, Axion, etc.
Chips designed with the help of AlphaChip are in datacenters and Samsung phones, right now. That's pretty neat!
As for external usage of the method - MediaTek is one of the largest chip design companies in the world, and they built on AlphaChip. There's a quote from a MediaTek SVP at the bottom of the GDM blog post:
"AlphaChip's groundbreaking AI approach revolutionizes a key phase of chip design. At MediaTek, we've been pioneering chip design's floorplanning and macro placement by extending this technique in combination with the industry's best practices. This paradigm shift not only enhances design efficiency, but also sets new benchmarks for effectiveness, propelling the industry towards future breakthroughs."
You are now using multiple new accounts based on the name of one of the authors (Anna Goldie) and her husband (Gabriel). First this one ('gabegobblegoldi'), and then 'anna-gabriella'.
I think it is time for you to take a deep breath and think about what you are doing and why.
You seem to be obsessed with the idea that this work is overrated. MediaTek and Google don't think so, and use it in production for their chips, including TPU, Dimensity, Axion, and others. If you're right and they're wrong, using this method loses them money. If it's the other way around, then using this method makes them gain money.
Chatterjee settled his case. He has moved on. This is not some product being sold -- it is a free, open-source tool. People who see value in it use it; others don't, and so they don't. This is how it always works, and it's fine.
Sometimes training from scratch is able to match the results of pre-training, given ~5X more time to converge. Other times, though, it never does as well as a pre-trained model, converging to a worse final result.
This isn't too surprising -- the whole point of the method is to be able to learn from experience.
The quick-start guide in the repo that said you don't have to pre-train for the sample test case, meaning that you can validate your setup without pre-training. That does not mean you don't need to pre-train! Again, the paper talks at length about the importance of pre-training.
They optimize using a fast heuristic based on wirelength, congestion, and density, but they evaluate with full P&R. It is definitely interesting that they get good timing without explicitly including it in their reward function!
The Nature paper describes the importance of pre-training repeatedly. The ability to learn from experience is the whole point of the method. Pre-training is just training and saving the weights -- this is ML 101.
I'm glad you agree that HPWL is a proxy metric. Optimizing HPWL is a fun applied math puzzle, but it's not chip design.
I am unaware of a single instance of someone using SA to generate real-world, usable macro layouts that were actually taped out, much less for modern chip design, in part due to SA's struggles to manage congestion, resulting in unusable layouts. SA converges quickly to a bad solution, but this is of little practical value.
For AlphaChip, pre-training is just training. You train, and save the weights in between. This has always been supported by the Google's open-source repository. I've read Kahng's FAQ, and he fails to address this, which is unsurprising, because there's simply no excuse for cutting out pre-training for a learning-based method. In his setup, every time AlphaChip sees a new chip, he re-randomizes the weights and makes it learn from scratch. This is obviously a terrible move.
HPWL (half-perimeter wirelength) is an approximation of wirelength, which is only one component of the chip floorplanning objective function. It is relatively easy to crunch all the components together and optimize HPWL --- minimizing actual wirelength while avoiding congestion issues is much harder.
Simulated annealing is good at quickly converging on a bad solution to the problem, with relatively little compute. So what? We aren't compute-limited here. Chip design is a lengthy, expensive process where even a few-percent wirelength reduction can be worth millions of dollars. What matters is the end result, and ML has SA beat.
(As for conflict of interest, my understanding is Cadence has been funding Kahng's lab for years, and Markov's LinkedIn says he works for Synopsis. Meanwhile, Google has released a free, open-source tool.)
Prior to AlphaChip, macro placement was done manually by human engineers in any production setting. Prior algorithmic methods especially struggled to manage congestion, resulting in chips that weren't manufacturable.
Definitely a big part of it. Chips enable better EDA tools, which enable better chips. First it was analytic solvers and simulated annealing, now ML. Exciting times!
https://deepmind.google/discover/blog/how-alphachip-transfor...
And they've responded to the absurd degree of skepticism that followed:
"That Chip Has Sailed: A Critique of Unfounded Skepticism Around AI for Chip Design" - https://arxiv.org/abs/2411.10053