How streaming LLM APIs work(til.simonwillison.net)
til.simonwillison.net
How streaming LLM APIs work
https://til.simonwillison.net/llms/streaming-llm-apis
25 comments
That gif is really cool! I built a Python package magentic [0] which similarly parses the LLM streamed output and allows it to be used before it is finished being generated. There are plenty of use cases / prompts that can be refactored into a "generate list, then generate for each item" pattern to take advantage of this speedup from concurrent generation.
[0] https://magentic.dev/streaming/#object-streaming
[0] https://magentic.dev/streaming/#object-streaming
When you ask to return JSON data using streaming, you will notice that the response is incomplete and unparseable by JSON libraries, resulting in malformed errors. You will have to wait for the entire stream to complete.
To solve this problem I tried to define a spec and built a lib for it:
- [lib] https://github.com/st3w4r/openai-partial-stream/tree/main
- [spec] https://github.com/st3w4r/openai-partial-stream/blob/main/sp...
To solve this problem I tried to define a spec and built a lib for it:
- [lib] https://github.com/st3w4r/openai-partial-stream/tree/main
- [spec] https://github.com/st3w4r/openai-partial-stream/blob/main/sp...
Very interesting. I tried to solve this problem too, and my code parses incomplete JSON allowing partial values and fully complete values to be accessed.
Why do you wait for the entire stream to be complete? Some objects in the JSON structure can be shown to be complete before the stream ends.
Why do you wait for the entire stream to be complete? Some objects in the JSON structure can be shown to be complete before the stream ends.
Yeah, it's an interesting problem to solve. The library is designed to parse incomplete json without waiting for the stream to finish.
I’ve been using the ijson Python library for that - I have notes on that here: https://til.simonwillison.net/json/ijson-stream
Pydantic also have support for parsing partial JSON. https://docs.pydantic.dev/latest/concepts/json/#partial-json...
from pydantic_core import from_json
partial_json_data = '["aa", "bb", "c'
result = from_json(partial_json_data, allow_partial=True)
print(result)
#> ['aa', 'bb']
That's neat, I hadn't seen that. Docs were lacking so I submitted a PR: https://github.com/pydantic/jiter/pull/143
Nice, it looks like a good library to build on top of. I like the available events: start_map, end_map, etc. I did try a library in JS that had similar ones, but it lacked the granularity to cover all use cases for individual fields instead of an entire item. I'll keep a note of this one if I do Python JSON streaming.
These are great. I've been working on trying to get markup working with streaming and it's a seemingly hard problem. This should help with figuring it out!
Here’s a bit more info on generating streams like this: https://parnassus.co/building-a-copilot-1-server-fundamental...
I’m slowly building a copilot stack, and end up wrapping multiple layers of streaming: SSE as in this article, parsed on the fly as it streams from JSON (ie parsi g incomplete invalid JSON), parsed on the fly as it streams to extract Markdown, parsed on the fly as it streams to format that Markdown and render it. You can read about this here: https://parnassus.co/building-a-copilot-2-parsing-converting...
I’m slowly building a copilot stack, and end up wrapping multiple layers of streaming: SSE as in this article, parsed on the fly as it streams from JSON (ie parsi g incomplete invalid JSON), parsed on the fly as it streams to extract Markdown, parsed on the fly as it streams to format that Markdown and render it. You can read about this here: https://parnassus.co/building-a-copilot-2-parsing-converting...
Interesting subject, but came here to comment that you are “doing the lord’s work” by writing an LLM tool for Delphi developers. All six of them! (i kid) Best of luck with Owl.
Thankyou! I plan to expand it to other under-served languages. Delphi is a fun starting one :)
I have been working with streaming LLMs and Server Sent Events. It provides a very simple interface to work with, but you can feel SSE was never designed for this use case. As mentioned in the blog post:
> Annoyingly these can't be directly consumed using the browser EventSource API because that only works for GET requests, and these APIs all use POST.
It is not designed to send data to open a connection. You will then struggle to work with this streaming approach using frameworks and libraries that are based on the EventSource API.
> Annoyingly these can't be directly consumed using the browser EventSource API because that only works for GET requests, and these APIs all use POST.
It is not designed to send data to open a connection. You will then struggle to work with this streaming approach using frameworks and libraries that are based on the EventSource API.
EventSource is really really limited. However, you can instead use Fetch via something like https://github.com/Azure/fetch-event-source to consume SSEs.
This looks very good. The Fetch API is a nice one, so leveraging it sounds perfect. Thanks for the link.
[deleted]
OpenAI streaming has many peculiarities at production scale.
e.g. you will get “half-chunks” occasionally which are not parseable on their own and must be concatenated with the previous or subsequent chunk for parsing.
e.g. you will get “half-chunks” occasionally which are not parseable on their own and must be concatenated with the previous or subsequent chunk for parsing.
I do actually wonder if it's more efficient to use something like MessagePack instead of using JSON. It's a lot of strings so it may not matter too much I guess.
Looks like a ton of wasted data on extraneous fields
[deleted]
Most interesting is combining this with web components and having GPT directly output streams with small web components.