mratsim·3 miesiące temu·discussIn my case with vLLM on dual RTX Pro 6000gpt-oss-120b: (unknown prefill), ~175 tok/s generation. I don't remember the prefill speed but it certainly was below 10kNemotron-3-Super: 14070 tok/s prefill, ~194.5 tok/s generation. (Tested fresh after reload, no caching, I have a screenshot.)Nemotron-3-Super using NVFP4 and speculative decoding via MTP 5 tokens at a time as mentioned in Nvidia cookbook: https://docs.nvidia.com/nemotron/nightly/usage-cookbook/Nemo...
mratsim·6 lat temu·discussNim memory management is tied to the type you use.You either use:- an object. Which is on the stack and is either trivial or uses destructors and is suitable for embedded due to deterministic memory management- a pointer object. Which is a raw pointer like C *. Managed directly via raw malloc/free or Nim malloc/free. Suitable for embedded- a reference type. Which is managed by one of Nim GC or is an error if you use gc:none
gpt-oss-120b: (unknown prefill), ~175 tok/s generation. I don't remember the prefill speed but it certainly was below 10k
Nemotron-3-Super: 14070 tok/s prefill, ~194.5 tok/s generation. (Tested fresh after reload, no caching, I have a screenshot.)
Nemotron-3-Super using NVFP4 and speculative decoding via MTP 5 tokens at a time as mentioned in Nvidia cookbook: https://docs.nvidia.com/nemotron/nightly/usage-cookbook/Nemo...