
We present Complexity-Deep, a transformer architecture introducing two key innovations: (1) Token-Routed MLP, a deterministic expert routing mechanism based on token identity that achieves perfect load balancing without auxiliary losses, and (2) Mu-Guided Dynamics, a simplified PID-inspired system that accumulates context across layers and influences both attention and expert routing. Our Token-Routed MLP uses modulo-based routing (expert_id = token_id mod N) ensuring uniform expert utilization regardless of token frequency distribution. We further introduce CGGR (Contiguous Group GEMM Routing), a Triton-based kernel optimization achieving 5-6x speedup over batched matrix multiplication. The architecture incorporates modern techniques including Grouped Query Attention (GQA), RoPE positional embeddings, QK Normalization, and Flash Attention via SDPA. Our 1.5B parameter model demonstrates the viability of deterministic routing as an alternative to learned routing in Mixture-of-Experts architectures.
| selected citations These citations are derived from selected sources. This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically). | 0 | |
| popularity This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network. | Average | |
| influence This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically). | Average | |
| impulse This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network. | Average |
