What I added (37 commits):
- Tool calling that works — streaming + non-streaming, supports MiniMax and Hermes/Qwen3 formats. 4/4 accuracy on structured function calling benchmarks.
- Reasoning separation — MiniMax-M2.5 mixes reasoning into its output with no tags. Built a heuristic parser that cleanly separates reasoning from content (0% leak rate, was 60%
with the generic parser).
- Prompt cache for SimpleEngine — persistent KV cache across requests. On 33K-token coding agent contexts: TTFT goes from 28s to 0.3s on cache hit. This is the single biggest
improvement for multi-turn use.
- 1500+ tests — parsers, engine, server, tool calling. The upstream had minimal test coverage.
Benchmarks (Mac Studio M3 Ultra, 256GB):
Qwen3-Coder-Next-6bit (80B MoE, 3B active):
- Decode: 65 tok/s
- Prefill: 1090-1440 tok/s
- TTFT (cache hit, 33K context): 0.3s
MiniMax-M2.5-4bit (229B MoE):
- Decode: 33-38 tok/s
- Deep reasoning with tool calling
I built this to run OpenClaw locally on my Mac instead of paying for cloud APIs. Qwen3-Coder-Next at 65 tok/s with tool calling is genuinely usable — not a toy demo.
Quick start:
pip install git+https://github.com/raullenchai/vllm-mlx.git
python -m vllm_mlx.server \
--model lmstudio-community/Qwen3-Coder-Next-MLX-6bit \
--tool-call-parser hermes --port 8000
GitHub: https://github.com/raullenchai/vllm-mlx
Two findings from benchmarking 7 models across 5 agent frameworks:
1. Qwen family gets 100% tool calling across every framework tested. Non-Qwen models (Llama, DeepSeek-R1) vary wildly — 40% to 100% depending on framework.
2. smolagents (HuggingFace) sidesteps structured function calling entirely by using code generation. DeepSeek-R1 goes from 40% with structured FC to 100% with smolagents.
Speed-wise, MLX's unified memory means zero CPU↔GPU copies. On an M3 Ultra: Qwen3.5-9B hits 108 tok/s (vs ~41 on Ollama), Qwen 3.6 35B does 100 tok/s with only 3B active params.
The full benchmark data is in the README. Happy to discuss the MLX performance characteristics or tool calling architecture.