OP here. In my previous post [1], I argued that code generation is the kingpin behind reasoning models. The bottleneck is LLMs generate code lossily due to tokenization fragmentation and treating code like natural language instead of structured graphs. In this post I propose:
1. Parsing user prompts into input graphs (using controlled English like ACE)
2. Parsing code into output graphs (AST trees)
3. Using graph transformers to map input graphs → output graphs
Core hypothesis: eliminating tokenization fragmentation ("ladlen" → ["lad", "len"]) and preserving tree structure could improve FrontierMath accuracy from 26% to 35-40%. No benchmarks yet. Just theory and a plan to test the improvement.
I've built compilers, not transformers, so would love technical feedback on:
- Is tokenization & linear structure really the bottleneck in code generation, or am I missing bigger issues?
- Is 35-40% improvement plausible, or overly optimistic?
- For those working on graph transformers: what approaches look promising?
OP here. In my previous post [1], I argued that code generation is the kingpin behind reasoning models. The bottleneck is LLMs generate code lossily due to tokenization fragmentation and treating code like natural language instead of structured graphs. In this post I propose:
1. Parsing user prompts into input graphs (using controlled English like ACE)
2. Parsing code into output graphs (AST trees)
3. Using graph transformers to map input graphs → output graphs
Core hypothesis: eliminating tokenization fragmentation ("ladlen" → ["lad", "len"]) and preserving tree structure could improve FrontierMath accuracy from 26% to 35-40%. No benchmarks yet. Just theory and a plan to test the improvement.
I've built compilers, not transformers, so would love technical feedback on:
- Is tokenization & linear structure really the bottleneck in code generation, or am I missing bigger issues?
- Is 35-40% improvement plausible, or overly optimistic?
- For those working on graph transformers: what approaches look promising?
Thanks in advance!
[1] Previous post - https://manidoraisamy.com/reasoning-not-ai.html
HN thread - https://news.ycombinator.com/item?id=45683113