Count tokens used by GPT-4 and Llama for large texts (> 50k characters)(huggingface.co)

Count tokens used by GPT-4 and Llama for large texts (> 50k characters)

https://huggingface.co/spaces/Xenova/the-tokenizer-playground

1 comments

xenova3 anni fa

This web-app fixes the two main problems of OpenAI's tokenizer playground: (1) being capped at 50k characters, and (2) not supporting GPT-4/GPT-3.5 tokenizers.

Everything runs in-browser thanks to Transformers.js.