How (and why) to implement streaming in your LLM application(kusho.ai)
kusho.ai
How (and why) to implement streaming in your LLM application
https://kusho.ai/blog/how-to-implement-streaming-in-your-llm-application
4 comments
hey, thanks for pointing this out! we work on english only applications so missed this. will add a note to the blog
You need to be handling LLM output as partial UTF-8 encoded sequences, not as complete UTF-8 encoded strings.
In this case, this means you create one TextDecoder for the entire response, then call `decode` for each streamed token, with the `stream` option set to `true` [0]. That way, incomplete UTF-8 codepoints can be completed by future tokens, and not discarded.
If you do not do this, most languages other than English will not output properly. They will be interspersed with garbage, characters will be missing. Certainly anything using Chinese, Japanese, or Korean characters. It can become entirely unreadable and unusable.
This issue is present in the vast majority of scripts and interfaces that handle streaming responses from LLMs, and it's a huge pet peeve of mine.
I've also forwarded this concern to the author over email.
[0]: https://developer.mozilla.org/en-US/docs/Web/API/TextDecoder...