LangChain Agent Testing Guide Tool (Free)

1 pointsby exordex5 mesi fa0 comments

Hey HN, If you're building LangChain agents, you've probably seen them break in creative ways - prompt injection bypassing your chain logic, tools getting called with bad parameters, or cascading failures when an API times out mid-chain.

I built Khaos to test these failure modes before production.

Example LangChain agent: ```python from langchain.agents import AgentExecutor, create_openai_functions_agent from khaos import khaosagent

  @khaosagent(name="research-agent", framework="langgraph")
  def agent(query: str) -> dict:
      executor = AgentExecutor(agent=agent, tools=tools)
      result = executor.invoke({"input": query})
      return {"response": result["output"]}

Test it: pip install khaos-agent khaos discover khaos run research-agent --pack security

Khaos injects: - 242+ security attacks - Prompt injection variations that bypass LangChain's prompt templates - Tool misuse - Malicious parameters in tool calls (e.g., os.system injection in code execution tools) - Chain failures - What happens when your 3rd step in a 5-step chain times out? - LLM faults - Rate limits, token overflows, model unavailability

  Why this matters for LangChain specifically:

  LangChain's abstraction layers can hide vulnerabilities:
  - Prompt templates can still be injected via tool outputs
  - AgentExecutor doesn't validate tool parameters
  - Chains fail silently or propagate corrupted state
  - ReAct/Plan-and-Execute patterns have unique attack surfaces

  Works with LangGraph, LCEL chains, and classic LangChain agents. Auto-instruments your chains to inject faults at each step.

  Repo: https://github.com/ExordexLabs/khaos-sdk
  Examples: https://github.com/ExordexLabs/khaos-examples/tree/master/code-execution-agent

0 comments

—