OpenAI's Tokenizer Page for OSS Models(github.com)3 ポイント·投稿者 rgs224·3 年前·1 コメントgithub.comOpenAI's Tokenizer Page for OSS Modelshttps://github.com/1rgs/tokenwiz1 コメントコメントを投稿[–]rgs224·3 年前返信I kept having to debug prompt issues with open modelsSo I built OpenAI's Tokenizer page for all tokenizers on HuggingFace: Llama, Mistral, GPT2, MPT, Persimmon, T5 etchttp://tokenwiz.rahul.gs
So I built OpenAI's Tokenizer page for all tokenizers on HuggingFace: Llama, Mistral, GPT2, MPT, Persimmon, T5 etc
http://tokenwiz.rahul.gs