Interesting that none of the new features (DALLE-3, Advanced Data Analysis, Browse with Bing) are usable without enabling history (and therefore, using your data for training).
Logit-bias guidance goes a long way -- LLM structure for regex, context-free grammars, categorization, and typed construction. I'm working on a hosted and model-agnostic version of this with thiggle
We've found the same. A lot of usage through our LLM Categorization endpoint. The toughest problem was actually constraining the model to only output valid categories and not hallucinate new ones. And to only return one for single-classification (or multiple if that's the mode).
I just released a zero-shot classification API built on LLMs https://github.com/thiggle/api. It always returns structured JSON and only the relevant categories/classes out of the ones you provide.
LLMs are excellent reasoning engines. But nudging them to the desired output is challenging. They might return categories outside the ones that you determined. They might return multiple categories when you only want one (or the opposite — a single category when you want multiple). Even if you steer the AI toward the correct answer, parsing the output can be difficult. Asking the LLM to output structure data works 80% of the time. But the 20% of the time that your code parses the response fails takes up 99% of your time and is unacceptable for most real-world use cases.
For a less dramatic strategy with LLMs that expose the tokenizer vocabulary, you can use context-free grammars to constrain the logits according to the parser so that the LLMs only generate valid next tokens for the language.[0]
There's probably a better API that wraps generate, but there's a bit more work than the logit mask.
You have to go one token at a time, otherwise the masking becomes combinatoric rather than linear (two tokens at a time -- need to generate all two token pairs, etc.).
(author here) That's interesting! Maybe there's a way to quantify the cumulative probability of the squashed tokens (i.e., if you constrain to 'true' and 'false', what's the distribution of the other tokens).
For now, this is a good way to make sure that I can parse the output reliably in the minimal amount of completions (instead of looping until conformant).
Similar strategies with the logitsprocesor. It is a more generalized version that's not just constrained to JSON parsing, but any regex. JSONformer/clownfish try to parse the types syntactically.
A regex is a better fit for a different class of problems. You might implement a JSONformer/clownfish with this instead.