The goal of the project is to bring open ABI and FFI for machine learning systems.
- Stable, minimal C ABI designed for kernels, DSLs, and runtime extensibility.
- Zero-copy interop across PyTorch, JAX, and CuPy using DLPack protocol.
- Compact value and call convention covering common data types for ultra low-overhead ML applications.
- Multi-language support out of the box: Python, C++, and Rust (with a path towards more languages).
XGrammar is an open-source library for efficient, flexible, and portable structured generation. Bring 2x-10x speedup in grammar grammar-guided(JSON and CFG) LLM serving.
There is also vulkan support which should be more universal(also included in the post), for example, the post also shows running LLM on a steamdeck APU.
Checkout the latest docs https://mlc.ai/mlc-llm/docs/ MLC started with demos and it evolved lately, with API integrations, documentations into an inference solution that everyone can reuse for universal deployments
It certainly also involves generating code(e.g. WebGPU, vulkan) that are more akin to traditionally compiler, and more like graph and memory optimization. So indeed more than packaging.
tvm runtime is pretty decent(~700k-2M level depending on dependency included), you can checkout tvm community and bring up the question there, i think there might be some common interest. There are impl of runtime for vulkan, metal that can be used as reference.
I think instead what would be needed is a wgpu native runtime support for TVM.
Like the implementations in tvm vulkan, then it will be naturally link to any runtime that provides webgpu.h
Then yah the llm_chat.js would be high-level logic that targets the tvm runtime, and can be implemented in any language that tvm runtime support(that includes, js, java, c++ rust etc).
Support webgpu native is an interesting direction. Feel free to open a thread in tvm discuss forum and perhaps there would be fun things to collaborate in OSS
The WGSL are generated and compiled through TVM and embedded into the wasm.
I think what you mean is wgpu native support. At the moment the web gpu runtime dispatches to the js webgpu environment. Once TVM runtime comes with wgpu native support (like the current ones in vulkan or metal), then it is possible to leverage any wgpu native runtime like what Zig provide.
Additionally, currently tvm natively support targets like vulkan, metal directly which allows targeting these other platforms
- Stable, minimal C ABI designed for kernels, DSLs, and runtime extensibility. - Zero-copy interop across PyTorch, JAX, and CuPy using DLPack protocol. - Compact value and call convention covering common data types for ultra low-overhead ML applications. - Multi-language support out of the box: Python, C++, and Rust (with a path towards more languages).