Calculate the number of language model tokens for a string(blog.mastykarz.nl)

17 pointsby waldekmano passado3 comments

https://blog.mastykarz.nl/calculate-number-language-model-tokens-string/

3 comments

sva_ano passado

  import tiktoken
  enc = tiktoken.encoding_for_model('o1-mini')
  len(enc.encode('some string'))

Does that work on anything other than openai?

topicseedano passado

No but most models have their own libs you can use