HackerTrans
TopNewTrendsCommentsPastAskShowJobs

sumanyusharma

no profile record

Submissions

Launch HN: Hamming (YC S24) – Automated Testing for Voice Agents

129 points·by sumanyusharma·2 anni fa·66 comments

New code-focused LLM needle in the haystack benchmark

github.com
6 points·by sumanyusharma·2 anni fa·1 comments

Can LLMs find bugs in large Python codebases?

hamming.ai
5 points·by sumanyusharma·2 anni fa·3 comments

comments

sumanyusharma
·anno scorso·discuss
Congratulations on the launch. Few qs:

How do your agents decide a suspected issue is a validated vulnerability, and what measured false-positive/false-negative rates can you share?

How is customer code and data isolated and encrypted throughout reconnaissance, exploitation, and patch generation (e.g., single-tenant VPC, data-retention policy)?

Do the agents ever apply patches automatically, or is human review required—and how does the workflow integrate with CI/CD to prevent regressions?

Ty!
sumanyusharma
·2 anni fa·discuss
How is this different from Integuru? They posted a few weeks back here: https://news.ycombinator.com/item?id=41983409
sumanyusharma
·2 anni fa·discuss
Appreciate the info; I'll double-check Fidelity again!
sumanyusharma
·2 anni fa·discuss
I'm actually pretty interested in what you're building. Sure, Vanguard and Fidelity are well-established giants, but they've barely moved beyond standard ETFs for decades. Having the option to tweak weightings at a more granular level and do daily tax-loss harvesting at scale seems like a genuine step forward.

I also like that you're transparent about how you might eventually introduce additional revenue streams like margin lending or maybe even PFOF. Knowing that upfront is better than a sudden terms-of-service surprise down the road. Still, I'd hope you'll consider giving users some say over how their shares are handled — like opting out of lending — so your incentives stay aligned over the long run.

Congrats on hitting $10M AUM. I'm rooting for more low-fee alternatives that keep the user in the loop!
sumanyusharma
·2 anni fa·discuss
No plans for acquisition :)

Building product, talking to customers and making something people want!
sumanyusharma
·2 anni fa·discuss
We're focused on end-to-end evals focused on function-call accuracy, style, tone & latency of the conversations between our sims and your voice agent. Less focused on pure TTS evals at the moment!
sumanyusharma
·2 anni fa·discuss
Pipecat looks awesome! I'll run the examples over the weekend and try to see what the integration hooks need to look like: https://github.com/pipecat-ai/pipecat/tree/main/examples

It should be pretty straightforward at first glance!
sumanyusharma
·2 anni fa·discuss
Likely outsourced call centers since call complexity is low to medium. We also expect rapid adoption in industries like customer service, healthcare, and retail, where 24/7 availability could be high-impact for businesses and convenient for consumers!
sumanyusharma
·2 anni fa·discuss
Should be fixed now; could you try again please?
sumanyusharma
·2 anni fa·discuss
We forgot to enable non-US numbers in our config for the demo. (oops)

We're working on a fix right now!
sumanyusharma
·2 anni fa·discuss
I am curious - how was the team solving this at Kea?
sumanyusharma
·2 anni fa·discuss
I use Superwhisper (no affiliation, just a happy user), which runs a local Whisper model, to create most of my email drafts and post-meeting notes. I find Whisper more accurate than Mac’s built-in speech-to-text, plus I’m faster at speaking than typing.

Sometimes, I even ‘talk’ into Cursor’s chat window instead of typing. The only downside? It can get a bit annoying for others when you're talking to yourself all day.
sumanyusharma
·2 anni fa·discuss
I'm curious to learn more about what's blocking the widespread adoption of the LLM capabilities. Lack of knowledge, reliability, or something else?
sumanyusharma
·2 anni fa·discuss
This tracks. Text evals to test core logic and voice evals for overall end-to-end performance!
sumanyusharma
·2 anni fa·discuss
It's a bit of a catch-22.

Making current voice agents reliable is incredibly time-consuming and complex. This challenge has kept many teams from pushing their agents into production. Those who do launch often release a very limited, basic version to minimize risk. We frequently talk to teams in both camps.

As a result, there aren't many 'killer' voice products on the market right now. But as models improve, we'll see more voice-centric companies emerge.

Teams are already calling their agents by hand and keeping track of experiment runs in a spreadsheet. We're just automating the workflow and making it easier to run experiments!
sumanyusharma
·2 anni fa·discuss
Our customers, who build voice agents, are often asked by their customers to make their voice agents more human-like and flexible. Their clients — businesses like pest control and automotive repairs — value providing a personalized experience but want the convenience and reliability of a 24/7 booking and answering service.
sumanyusharma
·2 anni fa·discuss
Bolna looks awesome! We've considered going open-source, but we're not sure how to effectively manage a community.

I'll reach out async!
sumanyusharma
·2 anni fa·discuss
Absolutely agree that creating effective evals requires domain expertise. Right now, we're co-building evals with customers, but we're identifying which aspects can be productized.

Regarding text-based evals — part of testing voice agents involves assessing their core reasoning logic. To do that, we bypass the voice layer and simulate conversations via text. So yes, the core simulation engine is reusable for both conversational text and voice interactions.

We're also excited about shipping the ability to replay a simulated conversation inspired by a real user!
sumanyusharma
·2 anni fa·discuss
Yes! We're aiming to build a tool that both engineers and non-engineers love.

We've discovered that it's often faster for non-technical domain experts to iterate on prompts in a structured, eval-driven way, rather than relying on engineers to translate business requirements into prompts.

While storing prompts in code offers version control benefits, it can hinder collaboration. On the other hand, using a pure CMS for prompts enhances collaboration but sacrifices some modern software development practices.

We're working towards a solution that bridges this gap, combining the best of both approaches. We're not there yet, but we have a clear roadmap to achieve this vision!
sumanyusharma
·2 anni fa·discuss
I wonder if a more optimistic version of this could be used to train humans and improve their skills. I'm thinking along the lines of LeetCode / Project Euler, but more dynamic and personalized!

Few examples:

1) Customer service: Simulating challenging customer interactions could help reps develop patience and problem-solving skills.

2) Emergency responders: Creating realistic crisis scenarios (like 911 calls) that could improve decision-making under pressure.

3) Healthcare: Virtual patients with complex symptoms could speed up the learning rate for med students.

4) Conflict resolution: Practicing with difficult personalities could aid mediators and negotiators.

5) Sales: AI-simulated tough customers could help salespeople refine their pitches and objection-handling skills in a low-stakes environment.

Thoughts?