This is in the works :) In the past and still today, OpenAI has much better and easier function call support, something we rely on.
We currently are running through LiteLLM, so while undocumented in theory other LLMs could work (in my experience they don't). I’m working on updating and fixing this.
Beyond saving tokens, this greatly improved the quality and speed of answers: the language server (most notably used to find the declaration/definition of an identifier) gives the LLM
1. a shorter path to relevant information by querying for specific variables or functions rather than longer investigation of source code. LLMs are typically trained/instructed to keep their answers within a range of tokens, so keeping shorter conversations when possible extends the search space the LLM will be "willing" to explore before outputting a final answer.
2. a good starting point in some cases by immediately inspecting suspicious variables or function calls. In my experience this happens a lot in our Python implementation, where the first function calls are typically `info` calls to gather background on the variables and functions in frame.
Correct. "Selection of Benchmark Implementations" is a better name here. We'll update this in the next iteration. The point in this subsection is indeed that the selection is not adequate for comparison.
This is not the only issue, even an adequate selection of perfectly idiomatic and identical implementations would not have resulted in accurate comparison.
> C/C++ Outlier
Correct, Section 4.5.2 details this. It is 8.9x for us.
> JS/TS Outlier
The main outlier on our machine is mandelbrot, 21x (Section 4.5.1). Our second outlier is n-body (not discussed).
We currently are running through LiteLLM, so while undocumented in theory other LLMs could work (in my experience they don't). I’m working on updating and fixing this.