Show HN: 5MB Rust binary that runs HuggingFace models (no Python)(github.com)
github.com
Show HN: 5MB Rust binary that runs HuggingFace models (no Python)
https://github.com/Michael-A-Kuykendall/shimmy/releases
Hi HN! I built Shimmy, a lightweight AI inference server that can now load HuggingFace SafeTensors models directly without any Python dependencies.
2 コメント
that's great !
I will try it, does it support GPU like cuda?
one question, can I use it as a library in my rust project, or I can only call it through new process with exe file?
I will try it, does it support GPU like cuda?
one question, can I use it as a library in my rust project, or I can only call it through new process with exe file?
GPU/CUDA: Yes, but disabled by default for faster builds. To enable: remove LLAMA_CUDA = "OFF" from config.toml and rebuild with CUDA toolkit installed.
Rust library: Absolutely! Add shimmy = { version = "0.1.0", features = ["llama"] } to Cargo.toml. Use the inference engine directly:
let engine = shimmy::engine::llama::LlamaEngine::new(); let model = engine.load(&spec).await?; let response = model.generate("prompt", opts, None).await?;
No need to spawn processes - just import and use the components directly in your Rust code.
Rust library: Absolutely! Add shimmy = { version = "0.1.0", features = ["llama"] } to Cargo.toml. Use the inference engine directly:
let engine = shimmy::engine::llama::LlamaEngine::new(); let model = engine.load(&spec).await?; let response = model.generate("prompt", opts, None).await?;
No need to spawn processes - just import and use the components directly in your Rust code.