Ask HN: How to avoid LLMs struggling with Lisp parens?
2 comments
For common lisp, cl-mcp is great for this (https://github.com/cl-ai-project/cl-mcp). I have not had any parentheses issues while using claude opus and cl-mcp. Other lisp mcps are probably also good, but I can only vouch for cl-mcp.
I suspect the most the reliable approach is to stop treating the model as the syntax checker. Let it propose a small diff, then run the result through check-parens, the language parser, or a formatter before accepting the change. if valifation fails, feed back the excat parser error and smallest affected form rather than the entire file
This is frustrating for two reasons:
Firstly, LLMs are famously bad at counting characters (e.g. the number of "r"s in "strawberry"), so it's no wonder this approach of generating and counting characters doesn't work very well.
Secondly, balancing parentheses is trivial for traditional, non-LLM algorithms; so it feels like an entirely avoidable problem (without resorting to larger, more-expensive models).
Is anyone using LLMs successfully on Lispy projects? If so, what workflows, tooling, etc. have you found to work well? I've tried guiding them to use Emacs `check-parens` rather than counting "manually"; but maybe inferring from indentation might work better? Perhaps tree-based generation/tools would avoid introducing such problems in the first place?