OpenAI's Tokenizer Page for OSS Models(github.com)3 points·by rgs224·قبل 3 سنوات·1 commentsgithub.comOpenAI's Tokenizer Page for OSS Modelshttps://github.com/1rgs/tokenwiz1 commentsPost comment[–]rgs224·قبل 3 سنواتreplyI kept having to debug prompt issues with open modelsSo I built OpenAI's Tokenizer page for all tokenizers on HuggingFace: Llama, Mistral, GPT2, MPT, Persimmon, T5 etchttp://tokenwiz.rahul.gs
So I built OpenAI's Tokenizer page for all tokenizers on HuggingFace: Llama, Mistral, GPT2, MPT, Persimmon, T5 etc
http://tokenwiz.rahul.gs