hellovai

no profile record

Submissions

A Cautionary Tale on Vibes

1 pointsby hellovai8 mesi fa1 comments

Deepseek R1: Adding Function Calling/Tool use

2 pointsby hellovaianno scorso0 comments

Show HN: GitHub Wrapped – analyzing every file change in every commit

6 pointsby hellovai2 anni fa0 comments

Show HN: PromptFiddle – Open-source WASM-based LLM playground

promptfiddle.com

2 pointsby hellovai2 anni fa0 comments

Show HN: Beating OpenAI's structured outputs on cost, accuracy and speed

8 pointsby hellovai2 anni fa2 comments

Prompts are using 4x more tokens than you need

7 pointsby hellovai2 anni fa2 comments

Show HN: BAML - A new programming language to get structured data from LLMs

18 pointsby hellovai2 anni fa2 comments

comments

7 mesi fadiscuss

(on of the creators of BAML here) yep! exactly!

that workaround we've found works quite well, but the problem is that its not sufficient to just retry in the case of failed schema matches (its both inefficient and also imo incorrect).

Take these two scenarios for example:

Scenario 1. My system is designed to output receipts, but the user does something malicious and gives me an invoice. during step 2, it fails to fit the schema, but then you try with step 3, and now you have a receipt! Its close, but your business logic is not expecting that. Often when schema alignment fails, its usually because the schema was ambiguous or the input was not valid.

Scenario 2. I ask the LLM to produce this schema:

    class Person {
      name string
      past_jobs string[]
    }

However the person only has ever worked at 1 job. so the LLM outputs: { "name": "Vaibhav", "past_jobs": "Google" }. Technically since you know you expect an array, you could just transform the string -> string[].

thats the algorithm we created: schema-aligned parsing. More here if you're interested: https://boundaryml.com/blog/schema-aligned-parsing

Benchmark wise, when we tested last, it seems to help on top of every model (especially the smaller ones) https://www.reddit.com/r/LocalLLaMA/comments/1esd9xc/beating...

Hope this helps with some of the ambiguities in the post :)

10 mesi fadiscuss

if you haven't tried the research -> plan -> implementation approach here, you are missing out on how good LLMs are. it completely changed my perspective.

the key part was really just explicitly thinking about different levels of abstraction at different levels of vibecoding. I was doing it before, but not explicitly in discrete steps and that was where i got into messes. The prior approach made check pointing / reverting very difficult.

When i think of everything in phases, i do similar stuff w/ my git commits at "phase" levels, which makes design decision easier to make.

I also do spend ~4-5 hours cleaning up the code at the very very end once everything works. But its still way faster than writing hard features myself.

10 mesi fadiscuss

Have you tried techniques that don’t require modifying the LLM and the sampling strategy for structure outputs? For example, schema aligned passing, where you build error tolerance into the parser instead of coercing to a grammar.

https://boundaryml.com/blog/schema-aligned-parsing

anno scorsodiscuss

if you share your prompt with me on promptfiddle.com i can play around with it and see how i can make it better!

anno scorsodiscuss

its a bit more nuanced than applicative lifting. parts of of SAP is that, but there's also supporting strings that don't have quotation marks, supporting recursive types, supporting unescaped quotes like: `"hi i wanted to say "hi""`, supporting markdown blocks inside of things that look like "json", etc.

but applicative lifting is a big part of it as well!

gloochat.notion.site/benefits-of-baml

anno scorsodiscuss

appreciate you tyring it. the reason it dropped the day was due to your type system not being understood by the LLM you're using.

the model replied with

       {
          "Text": "coffee liqueur",
          "Type": "Liqueur",
          "Liquor_type": "Liqueur",
          "Name_brand": null,
          "Unit_of_measure": "ounce",
          "Measurement_or_unit_count": "3/4"
        },

but you expected a { Text: string, Type: IngredientType, Liquor_type: LiquorType or null, Name_brand: string or null, Unit_of_measure: string, Measurement_or_unit_count: string, }

there's no way to cast `Liqueur` -> `IngredientType`. but since the the data model is a `Ingredient[]` we attempted to give you as many ingredients as possible.

The model itself being wrong isn't something we can do much about. that depends on 2 things (the capabilities of the model, and the prompt you pass in).

If you wanted to capture all of the items with more rigor you could write it in this way:

    class Recipe {
        name string
        ingredients Ingredient[]
        num_ingredients int
        ...

        // add a constraint on the type
        @@assert(counts_match, {{ this.ingredients|length == this.num_ingredients }})
    }

And then if you want to be very wild, put this in your prompt:

   {{ ctx.output_format }}
   No quotes around strings

And it'll do some cool stuff

anno scorsodiscuss

have you tried schema-aligned parsing yet?

the idea is that instead of using JSON.parse, we create a custom Type.parse for each type you define.

so if you want a:

   class Job { company: string[] }

And the LLM happens to output:

   { "company": "Amazon" }

We can upcast "Amazon" -> ["Amazon"] since you indicated that in your schema.

https://www.boundaryml.com/blog/schema-aligned-parsing

and since its only post processing, the technique will work on every model :)

for example, on BFCL benchmarks, we got SAP + GPT3.5 to beat out GPT4o ( https://www.boundaryml.com/blog/sota-function-calling )

anno scorsodiscuss

really cool to see BAML on here :) 100% align on so much of what you've said here. its really about treating LLMs as functions.

anno scorsodiscuss

yea! even deepseek. Calling an external function / tool calling is really just a data extraction problem.

say you have a tool:

def calculator(expr: str) -> float

then the model just needs to say:

{ "function": "calculator", "args": { "expr": "5 + 10" } }

then in your code you can easily pass that to the "calculator" function and get the result, then hand the result back to the model. Making it feel like the model can "call" an external function.

deep seek can also do this: https://www.boundaryml.com/blog/deepseek-r1-function-calling

2 anni fadiscuss

i 100% percent agree. people get so caught up on trying to do everything 90% right with AI, but they forget there's a reason most websites offer at least 2 9's of uptime.

2 anni fadiscuss

We have some preliminary data with llama3.1 and we find that the smaller model gets to around 70% with BAML (+20% from base), but we'll update this dashboard with llama3.1 by end of week!

2 anni fadiscuss

Take a look at BAML (boundaryml.com)

Its a different take that leverages a DSL to make prompting cleaner and fixes a few other ergonomic issues along the way.

you can try it online at promptfiddle.com

2 anni fadiscuss

nothing specific, but you can try our prompt / datamodel out on https://www.promptfiddle.com

or if you're open to share your prompt / data model with, I can send over my best guess of a good prompt! We've found these models works even with over 50+ fields / nested and whatnot decently well!

2 anni fadiscuss

we recently added dynamic type support with this snippet! (docs coming soon!)

Python: https://github.com/BoundaryML/baml/blob/413fdf12a0c8c1ebb75c...

Typescript: https://github.com/BoundaryML/baml/blob/413fdf12a0c8c1ebb75c...

Snippet:

async def test_dynamic():

    tb = TypeBuilder()

    tb.Person.add_property("last_name", tb.string().list())

    tb.Person.add_property("height", tb.float().optional()).description(
        "Height in meters"
    )


    tb.Hobby.add_value("chess")

    for name, val in tb.Hobby.list_values():
        val.alias(name.lower())

    tb.Person.add_property("hobbies", tb.Hobby.type().list()).description(
        "Some suggested hobbies they might be good at"
    )

    # no_tb_res = await b.ExtractPeople("My name is Harrison. My hair is black and I'm 6 feet tall.")
    tb_res = await b.ExtractPeople(
        "My name is Harrison. My hair is black and I'm 6 feet tall. I'm pretty good around the hoop.",
        {"tb": tb},
    )

    assert len(tb_res) > 0, "Expected non-empty result but got empty."

    for r in tb_res:
        print(r.model_dump())

2 anni fadiscuss

oh thats really interesting, how often do you get errors like that?

fyi, we actually fix those specific errors in our parser :)

2 anni fadiscuss

The main drawback is really when you attempt to do more advanced prompting techniques like chain-of-thought or reasoning.

forcing those parts to be json, can be hard and unnecessarily constrain the model. e.g. https://www.promptfiddle.com/Chain-of-Thought-KcSBh

try pressing run tests and you'll see what i mean! this method or doing chain of thought works a bit better

2 anni fadiscuss

;) https://www.promptfiddle.com/structured-summary-66myE (sorry bad syntax highlighting when including baml code in baml code)

{ author: "Sam Lijin"

key_points: [ "Structured output from LLMs, like JSON, is a common challenge."

  "Existing solutions like response_format: 'json' and function calling often disappoint."

  "The article compares multiple frameworks designed to handle structured output."

  "Handling and preventing malformed JSON is a critical concern."

  "Two main techniques for this: parsing malformed JSON or constraining LLM token generation."

  "Framework comparison includes details on language support, JSON handling, prompt building, control, model providers, API flavors, type definitions, and test frameworks."

  "BAML is noted for its robust handling of malformed JSON using a new Rust-based parser."

  "Instructor supports multiple LLM providers but has limitations on prompt control."

  "Guidance, Outlines, and others apply LLM token constraints but have limitations with models like OpenAI's."

]

take_way: "Consider using frameworks that efficiently handle malformed JSON and offer prompt control to get the desired structured output from LLMs."

}

2 anni fadiscuss

that's a great question, there's three main benefits:

1. seeing the full prompt, even though that python code feels leaner, somehow you need to convert it to a prompt. a library will do that in some way, BAML has a VSCode playground to see the entire prompt + tokenization. If we had to do this off of python/ts, we would run into the halting problem and making the playground would be much much harder.

2. there's a lot of codegen we do for users, to make life easier, e.g. w/o BAML, to now do streaming for the resume, you would have to do something like this:

class PartialResume: name: Optional[str] education: List[PartialEducation] skills: List[str]

and then at some point you need to reparse PartialResume -> Resume, we can codegen all of that for you, and give you autocomplete, type-safety for free.

3. We added a lot of static analysis / jump to definition etc to JINJA (which we use for strings), and that is much easier to navigate than f-strings.

4. Since its code-gen we can support all languages way easier, so prompting techniques in python work the same exact way for the same code in typescript.

2 anni fadiscuss

the main one is that most people don't own the model. so if you use openai / anthropic / etc then you can't use token masking. in that case, reprompting is pretty much the only option

2 anni fadiscuss

thats pretty cool! We'll update the page after taking a look at the library!

hellovai · HackerTrans