A practical guide to building agents [pdf](cdn.openai.com)
cdn.openai.com
A practical guide to building agents [pdf]
https://cdn.openai.com/business-guides-and-resources/a-practical-guide-to-building-agents.pdf
38 comments
I tried doing LLM-based tests, coined it "agentic-tests" and it worked quite well:
The idea was to use Stagehand [1] as the testing framework and then integrate this with Linear, which is our ticketing system. During a hackathon I whipped something together: first the 'agent' read a UAT from linear, then it pass this into a quite heavily prompted Stagehand. The prompts instructed Stagehand to run the UAT after its best ability and make very structured notes on each step on what failed. Once the Stagehand process was done, 'the agent' reported which steps succeeded and failed and into Linear.
Fundamentally the idea was sound, but there were some limitations in both the Linear SDK and in Stagehand. With some better tooling (or a novel system) and I predict this sort of agentic testing will work very well, especially for exploratory testing where the agent may be prompted to act like either a 90 year old grandma or 16 year old turbogamer. Privacy-safe usage heatmaps may also be generated automatically to test out the UX, as each run yielded slightly different approaches to achieve the UATs.
Testing teams need not apply!
[1]: https://www.stagehand.dev/
The idea was to use Stagehand [1] as the testing framework and then integrate this with Linear, which is our ticketing system. During a hackathon I whipped something together: first the 'agent' read a UAT from linear, then it pass this into a quite heavily prompted Stagehand. The prompts instructed Stagehand to run the UAT after its best ability and make very structured notes on each step on what failed. Once the Stagehand process was done, 'the agent' reported which steps succeeded and failed and into Linear.
Fundamentally the idea was sound, but there were some limitations in both the Linear SDK and in Stagehand. With some better tooling (or a novel system) and I predict this sort of agentic testing will work very well, especially for exploratory testing where the agent may be prompted to act like either a 90 year old grandma or 16 year old turbogamer. Privacy-safe usage heatmaps may also be generated automatically to test out the UX, as each run yielded slightly different approaches to achieve the UATs.
Testing teams need not apply!
[1]: https://www.stagehand.dev/
[deleted]
Thanks for sharing this. I'm actually starting to explore integrating an agent into one of my SaaS solutions, based on a client request.
To be honest, my experience with agents is still pretty limited, so I’d really appreciate any advice, especially around best practices or a roadmap for implementation. The goal is to build something that can learn and reflect the company’s culture, answer situational questions like “what to do in this case,” assist with document error checking, and generally serve as a helpful internal assistant.
All of this came from the client's desire to have a tool that aligns with their internal knowledge and workflows.
Is something like this feasible in terms of quality and reliability? And beyond hallucinations, are there major security or roadblocks concerns I should be thinking about?
To be honest, my experience with agents is still pretty limited, so I’d really appreciate any advice, especially around best practices or a roadmap for implementation. The goal is to build something that can learn and reflect the company’s culture, answer situational questions like “what to do in this case,” assist with document error checking, and generally serve as a helpful internal assistant.
All of this came from the client's desire to have a tool that aligns with their internal knowledge and workflows.
Is something like this feasible in terms of quality and reliability? And beyond hallucinations, are there major security or roadblocks concerns I should be thinking about?
Ingesting documents and using natural language to search your org docs with an internal assistant sounds more like a good use case for RAG[1]. Agents are best when you need to autonomously plan and execute a series of actions[2]. You can combine the two but knowing when depends on the use case.
I really like the OpenAI approach and how they outlined the thought process of when and how to use agents.
[1] https://www.willowtreeapps.com/craft/retrieval-augmented-gen...
[2] https://www.willowtreeapps.com/craft/building-ai-agents-with...
I really like the OpenAI approach and how they outlined the thought process of when and how to use agents.
[1] https://www.willowtreeapps.com/craft/retrieval-augmented-gen...
[2] https://www.willowtreeapps.com/craft/building-ai-agents-with...
Interesting, and thanks for explanations.
In this case, the agent would also need to learn from new events, like project lessons learned, for example.
Just curious: can a RAG[1] system actually learn from new situations over time in this kind of setup, or is it purely pulling from what's already there?
In this case, the agent would also need to learn from new events, like project lessons learned, for example.
Just curious: can a RAG[1] system actually learn from new situations over time in this kind of setup, or is it purely pulling from what's already there?
Especially with a client, consider the word choices around "learning". When using llms, agents, or rag, the system isn't learning (yet) but making a decision based on the context you provide. Most models are a fixed snapshot. If you provide up to date information, it will be able to give you an output based on that.
"Learning" happens when initially training the llm or arguably when fine-tuning. Neither of which are needed for your use case as presented.
"Learning" happens when initially training the llm or arguably when fine-tuning. Neither of which are needed for your use case as presented.
Thanks for the clarification, really appreciate it. It helps frame things more precisely.
In my case, there will be a large amount of initial data fed into the system as context. But the client also expects the agent to act more like a smart assistant or teacher, one that can respond to new, evolving scenarios.
Without getting into too much detail, imagine I feed the system an instruction like: “Box A and Box B should fit into Box 1 with at least 1" clearance.” Later, a user gives the agent Box A, Box B, and now adds Box D and E, and asks it to fit everything into Box 1, which is too small. The expected behavior would be that the agent infers that an additional Box 2 is needed to accommodate everything.
So I understand this isn't "learning" in the training sense, but rather pattern recognition and contextual reasoning based on prior examples and constraints.
Basically, I should be saying "contextual reasoning" instead of "learning."
Does that framing make sense?
In my case, there will be a large amount of initial data fed into the system as context. But the client also expects the agent to act more like a smart assistant or teacher, one that can respond to new, evolving scenarios.
Without getting into too much detail, imagine I feed the system an instruction like: “Box A and Box B should fit into Box 1 with at least 1" clearance.” Later, a user gives the agent Box A, Box B, and now adds Box D and E, and asks it to fit everything into Box 1, which is too small. The expected behavior would be that the agent infers that an additional Box 2 is needed to accommodate everything.
So I understand this isn't "learning" in the training sense, but rather pattern recognition and contextual reasoning based on prior examples and constraints.
Basically, I should be saying "contextual reasoning" instead of "learning."
Does that framing make sense?
There is no memory that the LLM has from your initial instructions to your later instructions.
In practice you have to send the entire conversation history with every prompt. So you should think of it as appending to an expanding list of rules that you put send every time.
In practice you have to send the entire conversation history with every prompt. So you should think of it as appending to an expanding list of rules that you put send every time.
You can use a RAG / embedding database as a kind of memory and add pulled 'approved examples', 'feedback comments' etc alongside future prompts.
Here's my recommendation:
What you're attempting to do, integrating an agent in your business, is difficult. It is however relatively easy to fake. Just setup a quick RAG tool, plug it into your LLM, and you're done. From the outside, the only difference between a quick-n-dirty integration and a much more robust approach will be in numbers. One will be more accurate than the other, but you need to actually measure and count performance to establish it as a fact and not just a vibe.
First advice: build up a dataset and measure performance as you develop your agent. Or just don't, and deliver what hype demands.
As for advices ... and looking at those other commenters left ... If you want to do this seriously, I'd recommend that you hire someone who already did that kind of integration, at least as a consultant. Someone whose first reflex won't be to just tell you LLMs are fixed and can't learn but will also add this isn't a limitation since RAG pipelines are better suited for this task than fine-tuning [1].
Also RAG isn't a monolithic solutions, there are many, many variations. For your use-case, I'd consider more elaborate solutions than just baseline RAG, such as GraphRAG [2]. For the box problem above, you might want to consider integrating symbolic reasoning tools such as prolog, or consider using reasoning models and developing your own reinforcement learning environments. Needless to say, all of these aspects need to be carefully balanced and optimized to work together, and you need to follow a benchmark/dataset centric-approach to developing your solution. For this task consider frameworks that were designed to optimize llm/agentic workflows as a whole [3][4].
Shit is complex really.
[1] https://arxiv.org/abs/2505.24832 tells us generalization happens in LLM once their capacity for remembering things is saturated, and this might explain why fine-tuning has been less efficient than RAG so far.
[2] https://microsoft.github.io/graphrag/
[3] https://github.com/microsoft/Trace
[4] https://github.com/SylphAI-Inc/AdalFlow
What you're attempting to do, integrating an agent in your business, is difficult. It is however relatively easy to fake. Just setup a quick RAG tool, plug it into your LLM, and you're done. From the outside, the only difference between a quick-n-dirty integration and a much more robust approach will be in numbers. One will be more accurate than the other, but you need to actually measure and count performance to establish it as a fact and not just a vibe.
First advice: build up a dataset and measure performance as you develop your agent. Or just don't, and deliver what hype demands.
As for advices ... and looking at those other commenters left ... If you want to do this seriously, I'd recommend that you hire someone who already did that kind of integration, at least as a consultant. Someone whose first reflex won't be to just tell you LLMs are fixed and can't learn but will also add this isn't a limitation since RAG pipelines are better suited for this task than fine-tuning [1].
Also RAG isn't a monolithic solutions, there are many, many variations. For your use-case, I'd consider more elaborate solutions than just baseline RAG, such as GraphRAG [2]. For the box problem above, you might want to consider integrating symbolic reasoning tools such as prolog, or consider using reasoning models and developing your own reinforcement learning environments. Needless to say, all of these aspects need to be carefully balanced and optimized to work together, and you need to follow a benchmark/dataset centric-approach to developing your solution. For this task consider frameworks that were designed to optimize llm/agentic workflows as a whole [3][4].
Shit is complex really.
[1] https://arxiv.org/abs/2505.24832 tells us generalization happens in LLM once their capacity for remembering things is saturated, and this might explain why fine-tuning has been less efficient than RAG so far.
[2] https://microsoft.github.io/graphrag/
[3] https://github.com/microsoft/Trace
[4] https://github.com/SylphAI-Inc/AdalFlow
Sound advice and much appreciated.
In this case, I might team up with someone to help me add this feature to my SaaS. But I’ll definitely dive deeper into the subject. Thanks for the info and the links!
There's also (of course) the agentic rag, especially if your data is from a lot of different types of resources, and you will have some context / memory set up that it relies on, in actuality with a lot of context there's is not a lot of "learning" needed.
Incorporating more data or new data into the RAG pool is a form of “learning”, but in general agents don’t “learn” unless you give them a journal or allow them to modify their own prompt.
You can ingest new documents and data into the RAG system as you need
The complexity of an agent may range from something relatively simple to whatever level of complexity you want. So your projet sounds doable but you'll have to run some exploration to get proper answers. Regarding reliability, quality and security, it is as important to learn how to observe an agent system than learning how to implement an agent system. An agent/LLM-based solution is proven to work only if you observe that it actually works, experiments, tests and monitoring are not optional like eg in web development. As for security concerns, you'd want to take a look at the OWASP top 10 for LLMs: https://owasp.org/www-project-top-10-for-large-language-mode...
LLMs/agents indeed have their own new set of vulnerabilities.
That’s sound advice, really appreciate the link. Regarding your point about continuous monitoring, that’s actually the first thing I mentioned to the client.
It’s still highly experimental and needs to be observed, corrected, and tweaked constantly, kind of like teaching a child, where feedback and reinforcement are key.
I may share my experience with the HN community down the line. Thanks again!
It’s still highly experimental and needs to be observed, corrected, and tweaked constantly, kind of like teaching a child, where feedback and reinforcement are key.
I may share my experience with the HN community down the line. Thanks again!
I'm a big fan of https://github.com/humanlayer/12-factor-agents because I think it gets at the heart of engineering these systems for usage in your app rather than a completely unconstrained demo or MCP-based solution.
In particular you can reduce most concerns around security and reliability when you treat your LLM call as a library method with structured output (Factor 4) and own your own control flow (Factor 8). There should never be a case where your agent is calling a tool with unconstrained input.
In particular you can reduce most concerns around security and reliability when you treat your LLM call as a library method with structured output (Factor 4) and own your own control flow (Factor 8). There should never be a case where your agent is calling a tool with unconstrained input.
I guess I’ve got some reading and research ahead of me. I definitely would rather support the idea of treating LLM calls more like structured library functions, rather than letting them run wild.
Definitely bookmarking this for reference. Appreciate you sharing it.
Definitely bookmarking this for reference. Appreciate you sharing it.
Provided the goal is solid internal culture at the company, the company would have an automatic dart thrower that reliably hits a target that is the wrong target altogether.
Supposing you hired a consultant to be "culture keeper" for this company -- and she or he said, "I'm just going to reason about context by treating this culture as a body of text" -- you would instantly assume that they didn't have skin in the game and didn't understand how culture actually grows and accretes, let alone monitoring and validating eventual quality or reliability. We can't read about what rules apply in some foreign culture's situations and then remotely prescribe what to do socially in a foreign culture we've never set foot in. We can't accurately anticipate even the second-order effects of our recommendations in that situation.
We simply have to participate first. It would be better for this to be a role that involves someone inside of the company who does participate in navigating the culture themselves so that they make accurate observations from experience. A person trustworthy enough to steward this culture would also necessarily be trustworthy enough not to alarm the chief of HR. Based on my model of how work works, from experience, I am wondering if they imagine they want this sensitive role filled with a nonhuman 'trusted' advisor so that it can't ever become a social shadow power center within the firm.
Or maybe they don't want to admit that modeling culture is beyond the reach of their matter-of-fact internal process models and simulations, and they're just wishfully hoping you can abstract away all of the soft elements without producing social fever dreams or ever having to develop a costly true soft element model. But then you absolutely abstract away where the rubber meets the road! That's quite a roadblock, to be honest with you.
Supposing you hired a consultant to be "culture keeper" for this company -- and she or he said, "I'm just going to reason about context by treating this culture as a body of text" -- you would instantly assume that they didn't have skin in the game and didn't understand how culture actually grows and accretes, let alone monitoring and validating eventual quality or reliability. We can't read about what rules apply in some foreign culture's situations and then remotely prescribe what to do socially in a foreign culture we've never set foot in. We can't accurately anticipate even the second-order effects of our recommendations in that situation.
We simply have to participate first. It would be better for this to be a role that involves someone inside of the company who does participate in navigating the culture themselves so that they make accurate observations from experience. A person trustworthy enough to steward this culture would also necessarily be trustworthy enough not to alarm the chief of HR. Based on my model of how work works, from experience, I am wondering if they imagine they want this sensitive role filled with a nonhuman 'trusted' advisor so that it can't ever become a social shadow power center within the firm.
Or maybe they don't want to admit that modeling culture is beyond the reach of their matter-of-fact internal process models and simulations, and they're just wishfully hoping you can abstract away all of the soft elements without producing social fever dreams or ever having to develop a costly true soft element model. But then you absolutely abstract away where the rubber meets the road! That's quite a roadblock, to be honest with you.
Oh, your response basically captures hours of conversation I’ve had with that and many other clients. I completely agree, no AI tool can replace years of experience within a company. Trying to model that too literally risks creating misunderstandings, or worse, damaging trust and reputation.
It definitely will never be a replacement for HR or top executive thinking. At best, I’ll be proposing something much lighter, more like a glorified internal search tool for real user examples. To be honest, I’m still all figuring it out. Best case: a helpful resource guide. Worst case: it adds no real value.
The tricky part is, if I don’t provide something, even just a prototype, they’re already looking at other consultants who’ll happily promise the moon. And that’s my bigger concern: if I’m not involved, someone else might introduce a half-baked solution that interferes with the SaaS I’ve already built for them.
So now I’m in a position where I need to put together a clear, honest demo that shows what this tech can and can’t do, just to avoid further complications down the line.
Ironically, this all started when another “AI expert” sold them the idea.
I’ve been saying the same thing all along, we’re not quite there yet. Maybe one day, but not now.
I also get that businesses want to take full advantage of this tech when it’s pitched as a money-saving opportunity, the pressure to act fast is real.
I wonder how many other devs and consultants are facing similar situations?
It definitely will never be a replacement for HR or top executive thinking. At best, I’ll be proposing something much lighter, more like a glorified internal search tool for real user examples. To be honest, I’m still all figuring it out. Best case: a helpful resource guide. Worst case: it adds no real value.
The tricky part is, if I don’t provide something, even just a prototype, they’re already looking at other consultants who’ll happily promise the moon. And that’s my bigger concern: if I’m not involved, someone else might introduce a half-baked solution that interferes with the SaaS I’ve already built for them.
So now I’m in a position where I need to put together a clear, honest demo that shows what this tech can and can’t do, just to avoid further complications down the line.
Ironically, this all started when another “AI expert” sold them the idea.
I’ve been saying the same thing all along, we’re not quite there yet. Maybe one day, but not now.
I also get that businesses want to take full advantage of this tech when it’s pitched as a money-saving opportunity, the pressure to act fast is real.
I wonder how many other devs and consultants are facing similar situations?
Others have given some decent advice based on your comment, but would you be interested in a ~30 minute (video) call to dive a bit deeper so I can give more tailored suggestions?
IMHO this guide should have been called "a theoretical guide for building agents". In practice, you cannot build agents like that if you want them to do useful things.
Also, the examples provided are not only not practical but potentially bad practice. Why do you need a manager pattern to control a bunch of language translation agents when most models will do fine especially for latin-based languages? In practice a single LLM will not only be more cost-effective but also good for the overall user experience too.
Also, prompting is the real unsung hero that barely gets a mention. In practice you cannot get away with just a couple of lines describing the problem / solution at a high-level. Prompts are complex and very much an art form because and frankly, let's be honest, there is no science whatsoever behind them - just intuition. But in practice they do have enormous effect on the overall agent performance.
This guide is not aimed at developers to really educate how to build agents but at business executives and decision-makers who need a high-level understanding without getting into the practical implementation details. It glosses over the technical challenges and complexity that developers actually face when building useful agent systems in production environments.
Also, the examples provided are not only not practical but potentially bad practice. Why do you need a manager pattern to control a bunch of language translation agents when most models will do fine especially for latin-based languages? In practice a single LLM will not only be more cost-effective but also good for the overall user experience too.
Also, prompting is the real unsung hero that barely gets a mention. In practice you cannot get away with just a couple of lines describing the problem / solution at a high-level. Prompts are complex and very much an art form because and frankly, let's be honest, there is no science whatsoever behind them - just intuition. But in practice they do have enormous effect on the overall agent performance.
This guide is not aimed at developers to really educate how to build agents but at business executives and decision-makers who need a high-level understanding without getting into the practical implementation details. It glosses over the technical challenges and complexity that developers actually face when building useful agent systems in production environments.
>Also, the examples provided are not only not practical but potentially bad practice. Why do you need a manager pattern to control a bunch of language translation agents when most models will do fine especially for latin-based languages? In practice a single LLM will not only be more cost-effective but also good for the overall user experience too.
Usually that's how these agent tutorials work. I don't think there any open sourced large scale agent applications that anyone has published online yet, especially those agent that hand off to other agent sort of apps.
Usually that's how these agent tutorials work. I don't think there any open sourced large scale agent applications that anyone has published online yet, especially those agent that hand off to other agent sort of apps.
Has anyone solved scoped permissions in multi-agent systems? For instance, if a user asks an orchestrator agent to:
1) Search GitHub for an issue in their repo.
2) Fix the issue and push it to GitHub.
3) Search Jira for any tasks related to this bug, and update their status.
4) Post a message to slack, notifying the team that you fixed it.
Now, let’s assume this agent is available to 1000 users at a company. How does the system obtain the necessary GitHub, Jira, and Slack permissions for a specific user?
The answer is fairly obvious if the user approves each action as the task propagates between agents, but how do you do this in a hands-free manner? Let’s assume the user is only willing to approve the necessary permissions once, after submitting their initial prompt and before the orchestrator agent attempts to call the GitHub agent.
If anyone could offer any advice on this, I would really appreciate it. Thank you!
1) Search GitHub for an issue in their repo.
2) Fix the issue and push it to GitHub.
3) Search Jira for any tasks related to this bug, and update their status.
4) Post a message to slack, notifying the team that you fixed it.
Now, let’s assume this agent is available to 1000 users at a company. How does the system obtain the necessary GitHub, Jira, and Slack permissions for a specific user?
The answer is fairly obvious if the user approves each action as the task propagates between agents, but how do you do this in a hands-free manner? Let’s assume the user is only willing to approve the necessary permissions once, after submitting their initial prompt and before the orchestrator agent attempts to call the GitHub agent.
If anyone could offer any advice on this, I would really appreciate it. Thank you!
I would solve this using the equivalent of a service account - I would give that "agent" an identity - "CodeBot" or whatever - and then that as an actor which has permission to read things on Jira, permission to send notifications to Slack, permission to access the GitHub API etc.
Then I would control who had permission to tell that to do, and log everything in detail.
Then I would control who had permission to tell that to do, and log everything in detail.
We do this at Arcade.dev with oAuth scopes and token persistence for each agent. There is a grant required for each tool-user tuple the first time, but then it’s remembered
Yea we have been developing Polykey for this purpose. Sent you an email to discuss.
Equivalent guides from —
Google: https://ia600601.us.archive.org/15/items/google-ai-agents-wh...
Anthropic: https://www.anthropic.com/engineering/building-effective-age...
Google: https://ia600601.us.archive.org/15/items/google-ai-agents-wh...
Anthropic: https://www.anthropic.com/engineering/building-effective-age...
Tools are the only thing that matters, and are what you should focus on, not "agents" as a separate concept. Locking yourself into any particular agent framework is silly; they are nothing but LLM calling while-loops connected to JSON/XML parsers. Tools define and shape the entirety of an agent's capability to do useful things, and through MCP can be trivially shared with virtually any agentic process.
Yes, I wonder if it ever makes sense to partition an agent's toolkit across multiple "agents"—besides horizontal scaling. Why should one process have access to APIs that another doesn't? Authorization and secrets, maybe, but functionality?
Depending on the capability or context size of the model, it is easy to imagine a situation where specifying too many tools (or MCPs) can overwhelm it, affecting the accuracy.
> it is easy to imagine a situation where specifying too many tools (or MCPs) can overwhelm it, affecting the accuracy.
Answer to this is agent explorable interfaces a la HATEOAS. Define a single entry point to your MCP gateway and give the agent clues to drill down and find what it needs to use.
Answer to this is agent explorable interfaces a la HATEOAS. Define a single entry point to your MCP gateway and give the agent clues to drill down and find what it needs to use.
You may get some mileage out of protocols like Agent Network Protocol (ANP), Agent Interaction & Transaction Protocol (AITP), Agora "Protocol Documents" (PDs), agents.json, Airweave, or the Coral Protocol.
AITPs "message passthrough pattern" supposedly enables "progressive" delegation—where a general agent can forward requests to specialized agents.
agents.json may facilitate a sort of progressive disclosure across the "flows" and "links" of multi-step workflows that an agent is guided through, wherein subsequent actions or information is revealed as dependencies are met, limiting upfront information overload.
I've only skimmed the barest details of these protocols, guided by this paper: https://arxiv.org/html/2504.16736v2
AITPs "message passthrough pattern" supposedly enables "progressive" delegation—where a general agent can forward requests to specialized agents.
agents.json may facilitate a sort of progressive disclosure across the "flows" and "links" of multi-step workflows that an agent is guided through, wherein subsequent actions or information is revealed as dependencies are met, limiting upfront information overload.
I've only skimmed the barest details of these protocols, guided by this paper: https://arxiv.org/html/2504.16736v2
So, to give an example of what's worked really well for me: I'm working for an app hosting startup named Wasmer, and we host a decent of apps. Some of these are malicious. So to very effectively detect the malicious apps, we have an app-vetter agent named Herman. Herman reads the index page of every newly created app, alongside a printscreen of the index page, then he flags if he thinks that the app is malicious. Then some human (usually me) inspect the app and make the final decision of it should banned or not.
This allows us to scan a quite large amount of apps and filter the noisy non-malicious apps. Doing this with a 'dumb' service wouldn't really be feasible, and the context of an LLM fits perfectly where it gets both an image and the source code. An LLM is also quite 'omnipotent', in that it for example knows that DANA is a bank in Malaysia, something I personally have no idea about.
I think tedious and time consuming chores like this is a great way of using agents. Next in line for my experimentation is to utilize agents for 'fuzzy' integration testing, where the LLM simply has access to a browser + cli tools and UAT specifications, and may (in an isolated environment) do whatever it wants. Then it should report back any findings and improvements using an MCP integration towards our ticketing system. So to utilize the hallucinations to find issues.