Show HN: Like grep but for natural questions. Mixtral 8x7B – 28 tok/s on 8GB GPU(github.com)
github.com
Show HN: Like grep but for natural questions. Mixtral 8x7B – 28 tok/s on 8GB GPU
https://github.com/moritztng/fltr
0 comments
—
https://github.com/moritztng/fltr
—
Example:
fltr --file emails.txt --prompt "Is the following email spam? Email:" --batch-size 32
It will output all lines in the file where the answer is yes.
Text file input tokens per second:
- Nvidia RTX 3070 with 8GB memory: Mistral 7B: ~52 tok/s, Mixtral 8x7B: ~28 tok/s
- Intel I5-6500 with 8GB memory: Mistral 7B: ~5 tok/s, Mixtral 8x7B: ~2 tok/s
Supports: Linux (x86_64) & macOS (x86_64 & arm64)