HackerTrans
TopNewTrendsCommentsPastAskShowJobs

rgs224

no profile record

Submissions

OpenAI's Tokenizer Page for OSS Models

github.com
3 points·by rgs224·vor 3 Jahren·1 comments

comments

rgs224
·vor 2 Jahren·discuss
was able to get around waitlist: copilot-workspace.githubnext.com /<owner>/<repo>?task=<description>
rgs224
·vor 3 Jahren·discuss
this is very cool, reminds me of the tldraw feature
rgs224
·vor 3 Jahren·discuss
I kept having to debug prompt issues with open models

So I built OpenAI's Tokenizer page for all tokenizers on HuggingFace: Llama, Mistral, GPT2, MPT, Persimmon, T5 etc

http://tokenwiz.rahul.gs