yilu331

2 tháng trướcdiscuss

there is a dashboard that you can use by running /claude-smart:dashboard command if there is any applied learnings, there will be a visual hint in the claude code response and show cite the exact learnings in the dashboard as well.

you could also ask claude code how does claude-smart learning helped your response to explain to you

yilu331

2 tháng trướcdiscuss

Hi HN, claude-smart is an open sourced claude code plugin that makes claude code self-improve.

The idea is memory alone does not prevent claude code to make the same mistake twice. It should be able to capture user corrections and turn into learnings that are generalized across different use cases and projects

We have done benchmark against other memory based solutions and found that memory based solution almost never capture any user correction but just storing user mentioned facts. In addition, our context footprint is a fraction of what claude-mem has, which make claude-smart super context efficient.

Try install it with ``` npx claude-smart install ```

and let us know what you think, your feedback is hugely appreciated!

yilu331

3 năm trướcdiscuss

LLM today makes it a lot easier to build chatbot without decision tree and handle more complex scenarios. What you would need to do is providing it sufficient tools to use and a scope for what it can do, so that the chat bot doesn't have to be mechanical

yilu331

3 năm trướcdiscuss

One of the valuable things AutoChain provides is simulated conversation evaluation. Testing and tuning those agent is very hard and expensive. checkout the evaluation output from AutoChain, for example.

"conversation_history":

      "user: I want to change the shipping address for my order",
      "assistant: Can you please provide the order ID?",
      "user: The order ID is 456",
      "assistant: The order with ID 456 has not yet shipped. I can update the shipping address for you. Please provide the new shipping address.",
      "user: The new shipping address is 234 Spear St, San Francisco",
      "assistant: The shipping address for your order with ID 456 has been successfully updated to 234 Spear St, San Francisco.",
      "user: Thank you for updating the shipping address for my order",
      "assistant: Thank your for contacting"

   "is_agent_helpful":
   {
      "rating": 5,
      "reason": "The conversation reaches the expected outcome. The user successfully provides the order ID and the new shipping address, and the assistant updates the shipping address for the order with ID 456. The assistant confirms the successful update and thanks the user for contacting."
   },
   "actions_took":

      {
         "tool": "check_order_status",
         "tool_input":
         {
            "order_id": "456"
         },
         "tool_output":
         {
            "status_code": 200,
            "order_id": "456",
            "order_status": "not_shipped",
            "tracking_url": "example.com/456",
            "shipping_address": "301 ivy street san francisco ca"
         }
      },
      {
         "tool": "change_shipping_address",
         "tool_input":
         {
            "order_id": "456",
            "new_address": "234 Spear St, San Francisco"
         },
         "tool_output":
         {
            "status_code": 200,
            "order_id": "456",
            "shipping_address": "234 Spear St, San Francisco"
         }
      }

   "num_turns": 8,
   "expected_outcome": "found order status and changed shipping address"

yilu331

3 năm trướcdiscuss

with openai function calling, it is definitely getting easier and easier to build llm agent into a prototype. There are a few benefits of using AutoChain. 1) get started easily; plug and play different ideas before building into a product 2) AutoChain introduces a novel evaluation framework for evaluating LLM agent's ability to perform multi-turn conversation under different user scenarios.

In addition, when it comes to prototyping for a specific use case, we found it is often more than just calling the model but also the orchestration process matters, for example, when should LLM agent stop answering questions, fix input argument, ask a custom clarifying questions and more.

Hope AutoChain makes your exploration easier and more robust!

yilu331

3 năm trướcdiscuss

great discussion. there are a lot of problems with LangChain for sure. hopefully we are trying to make it easier for exploring different ideas

yilu331

3 năm trướcdiscuss

exactly, i found the same thing which motivated me to create a lightweight version of it for rapid iteration. There are a lot of customizations i found needed when i try to wrap a practical prototype for LLM agent

yilu331

3 năm trướcdiscuss

AutoChain takes inspiration from LangChain and AutoGPT and aims to solve both problems by providing a lightweight and extensible framework for developers to build their own agents using LLMs with custom tools and automatically evaluating different user scenarios with simulated conversations. Experienced user of LangChain would find AutoChain is easy to navigate since they share similar but simpler concepts.

The goal is to enable rapid iteration on generative agents, both by simplifying agent customization and evaluation.

If you have any questions, please feel free to reach out to Yi Lu [email protected]

Submissions

Show HN: Claude-smart – Make Claude Code self-improve from every session

[untitled]

AutoChain, lightweight and testable alternative to LangChain

comments