The method: collect 3D activation histograms (layer × expert × rank) over domain-specific corpora, rank experts by a rank-weighted utilisation score, remove the bottom 128 of 256 per layer directly in the GGUF — no finetuning, no safetensors loading, struct-level I/O only.
The earlier work established that a 50% reduction in parameter count could result in near parity in performance in Python coding compared to the full model while losing any significant capacity to code in HTML, while the similarly constructed WEB coding specialist likewise performed at near equivalence to the full model on Web tasks but failed significantly on Python tasks.
This new work substantially completes the full proof of concept that a general MoE model can be histographically indexed and specialist models can be extracted at significant reduction in expert count and memory footprint to give an end user with constrained VRAM access to models, within a given domain, that heretofore would have been inaccessible.
Next will be a full decomposition of the new Gemma4-25b-a4b model to show applicability of this idea to a different base architecture, along with further development of the CoE (College of Experts) orchestration framework — designed to integrate a set of disk-resident specialist models into a functional system where the collective intelligence of serially-invoked specialists exceeds that of any single general model runnable within the same VRAM budget.
I wanted to share a proof-of-concept from the College of Experts project on the separability of machine intelligence.
We hypothesized that domain knowledge in monolithic Mixture-of-Experts models is not holographically entangled across all routing layers, but physically separable. Using histographic activation profiling across 10 coding languages, we surgically extracted the 256 experts responsible for Python from Qwen3-Coder-Next-80B and separately extracted the 256 experts responsible for Web/Frontend logic. We used a bias activation function across the 48 layers which modified the expert ranking and selected experts up to the expert budget of 256 per layer.
The resulting 40B Python Specialist retains a 93% score on HumanEval (compared to the 80B model's 94%), despite losing half its parameters. Conversely, the 40B Web Specialist retains near-perfect UI generation capabilities while completely losing the ability to emit raw Python logic. Note that this was achieved strictly via weight-slicing the unmodified .gguf file, with zero post-surgery fine-tuning.
The repo linked above contains Demo v1.5, which uses a fast ONNX supervisor (DML/CUDA) to hot-swap these massive 40B lobes via Ollama, allowing 80B-class MoE routing on consumer hardware (29GB VRAM footprint).
The earlier work established that a 50% reduction in parameter count could result in near parity in performance in Python coding compared to the full model while losing any significant capacity to code in HTML, while the similarly constructed WEB coding specialist likewise performed at near equivalence to the full model on Web tasks but failed significantly on Python tasks.
This new work substantially completes the full proof of concept that a general MoE model can be histographically indexed and specialist models can be extracted at significant reduction in expert count and memory footprint to give an end user with constrained VRAM access to models, within a given domain, that heretofore would have been inaccessible.
Next will be a full decomposition of the new Gemma4-25b-a4b model to show applicability of this idea to a different base architecture, along with further development of the CoE (College of Experts) orchestration framework — designed to integrate a set of disk-resident specialist models into a functional system where the collective intelligence of serially-invoked specialists exceeds that of any single general model runnable within the same VRAM budget.
All 8 models (Q4_K_M GGUF, ~18B, Ollama-ready), histograms, masks, corpora and pipeline scripts: GitHub: https://github.com/JThomas-CoE/College-of-Experts-AI HF: https://huggingface.co/JThomas-CoE