This is my concern as well. How successful is it in selecting the correct tool out of hundreds or thousands?
Different to what this integration is pushing, the LLMs usage in production based products where high accuracy is a requirement (99%), you have to give a very limited tool set to get any degree of success.
There is a generalized reasoning that current LLMs still miss. It's hard to put a finger on it. Things like hallucinations show that there isn't a self awareness of thought. "Thinking" models are getting closer.
From most of my network trying to make products based on LLMs, excluding cost, the biggest hurdles are hallucinations, and seemingly "non-sensical" reasoning or communicating. Subtle choices that just "feel" not quite right. Particularly when the LLM is being constrained for some activity.
Open-ended chat doesn't show these flaws as often.
An interesting side effect of this low-level bit mapping is that various banks authorization logics can be manipulated to increase auth rates by subtle bit flipping across various fields.
All the big fintech companies have ML running over changes to identify what results in the highest auth rates on a per bin basis.
Different to what this integration is pushing, the LLMs usage in production based products where high accuracy is a requirement (99%), you have to give a very limited tool set to get any degree of success.