Show HN: Skeletoken, a Python package for editing model tokenizers(github.com)

1 pointsby stephantulvor 5 Monaten0 comments

Show HN: Skeletoken, a Python package for editing model tokenizers

https://github.com/stephantul/skeletoken

Hello!

I just released a new version of Skeletoken, a package for editing tokenizers. New in this version is the ability to automatically adapt a model to an edited tokenizer.

For example, you can a new token to your tokenizer, and then ask skeletoken to add new token indices in the correct embedding tables.

Let me know what you think!

0 comments

—