OpenAI's Tokenizer Page for OSS Models(github.com)3 points·by rgs224·vor 3 Jahren·1 commentsgithub.comOpenAI's Tokenizer Page for OSS Modelshttps://github.com/1rgs/tokenwiz1 commentsPost comment[–]rgs224·vor 3 JahrenreplyI kept having to debug prompt issues with open modelsSo I built OpenAI's Tokenizer page for all tokenizers on HuggingFace: Llama, Mistral, GPT2, MPT, Persimmon, T5 etchttp://tokenwiz.rahul.gs
So I built OpenAI's Tokenizer page for all tokenizers on HuggingFace: Llama, Mistral, GPT2, MPT, Persimmon, T5 etc
http://tokenwiz.rahul.gs