Yes, we've seen this work best with clear instructions around Salesforce, Attio etc. superglue does come with a number of observability and monitoring features, but of course this can be extended for a specific setup.
Adina here, one of superglue's creators. I'm curious to hear folks opinions on this. Also to clarify how agents are able to call APIs via superglue: the first step is to set up auth and systems on superglue so it can process and extract documentation and any other context for calling the APIs, which is then passed on to the agents
How well can agents work with APIs they’ve never seen before? We tested 41 APIs across 8 different LLMs to find out.
API execution is great for benchmarking, because it tests core qualities and limitations of LLMs:_the depth of the data they were trained on, their stateless architecture, context dependency, and reasoning.
Today we're releasing v2 of API-Bench:_a benchmark that tests how well LLMs can execute against APIs. Here are the results: https://superglue.ai/benchmark_v2
Tl;dr:_LLMs fail at integrations because they lack ground truth, lack state, lack debugging ability, and lack access to real system context - everything API integrations fundamentally require.
Here’s what we found:
1. LLMs are only as good as the data they’re trained on:_when docs change, APIs evolve, or systems are niche/long-tail, they use outdated patterns, guess missing pieces and hallucinate endpoints and parameters.
2. LLMs are stateless, but integrations are stateful:_auth handshakes, pagination, retries, multi-step flows all need memory but LLMs can’t persist intermediate values or reason across steps.
3. LLMs produce code that “looks right” but fails at runtime: LLMs cannot isolate the failing step and understand real error messages, so they can’t change what’s broken or retry with new hypotheses.
4. LLMs can’t reliably interpret imperfect API design:_humans can infer the intended function, LLMs will hallucinate what looks reasonable.
creator of superglue here. we are trying to crack one of the hardest problems in infra management, with the goal to get rid of nasty systems and expensive consulting projects. always looking for the gnarliest of systems and how to make them future-compatible.