[dead]
Total Parameters: 1 trillion (1T)
Active Parameters: 32 billion (32B)
Number of Experts: 384, with 8 experts activated per token
Context Length: 256K tokens (upgraded from 128K in the original K2)
Model Layers: 61 layers (including 1 dense layer)
Attention Mechanism: MLA (Multi-head Latent Attention)
Activation Function: SwiGLU
Attention Hidden Dimension: 7168
Vocabulary Size: 160K
Training Data: 15.5 trillion tokens
Knowledge Cutoff: April 2025
License: Apache 2.0 (open-source, commercially usable)
There are no benchmarks yet for that specific model: