Qwen3.5-35B-A3B: 35 Billion Parameters, 3 Billion Active

Alibaba’s Qwen team released Qwen3.5-35B-A3B, a mixture-of-experts language model containing 35 billion total parameters but routing each token through only 3 billion active parameters. The architecture uses 256 experts with 8 routed plus 1 shared expert activated per token, each with an intermediate dimension of 512. Native context length sits at 262,144 tokens, extensible to roughly one million.

The model pairs sparse MoE layers with Gated DeltaNet linear attention blocks in a repeating pattern: every four sublayers, three use linear attention and one uses standard gated attention with grouped-query heads (16 query, 2 KV). This hybrid layout reduces the memory and compute overhead associated with full quadratic attention while retaining its benefits at regular intervals.

Architecture Details

The layer layout follows a 10 × (3 × DeltaNet-MoE → 1 × Attention-MoE) pattern, meaning 30 of 40 sublayers use linear attention. This is a significant departure from standard transformer stacks and directly targets inference throughput on memory-constrained hardware.

Benchmark Performance

Despite activating under 10% of its total parameters, the model posts numbers that track surprisingly close to far larger configurations. Selected results against relevant comparisons:

On TAU2-Bench (agent tasks), the 3B-active model actually outscores both its dense 27B sibling and GPT-5-mini. Coding benchmarks show the expected gap — roughly 3-6 points behind the dense 27B variant on SWE-bench and LiveCodeBench — but the model still posts 69.2% on SWE-bench Verified, which would have been state-of-the-art a year ago.

Vision Capabilities

Unlike previous small MoE releases, this model ships with an integrated vision encoder. Selected multimodal results:

The vision numbers are notable — outperforming both Claude Sonnet 4.5 and GPT-5-mini on document understanding and mathematical vision tasks while activating a fraction of the compute.

Practical Implications

The 3B active parameter count puts this model’s inference memory footprint in the range of models like Phi-3-mini or Llama-3.2-3B, while delivering performance that competes with 22-27B active-parameter models. For teams running local inference on consumer GPUs (RTX 4070-class and above), this is a meaningful density improvement.

The model is compatible with Hugging Face Transformers, vLLM, SGLang, and KTransformers. Alibaba also offers a hosted version under the name “Qwen3.5-Flash” through their Model Studio API, which includes a 1M default context window and built-in tool use.

Language coverage spans 201 languages and dialects, which matters for deployment scenarios outside the English-centric defaults most models optimize for.

The key question for infrastructure teams: does the MoE expert-loading overhead on consumer hardware negate the active-parameter savings? Real-world throughput benchmarks on specific GPU configurations will determine whether this model delivers on the efficiency promise or whether the 256-expert routing adds enough latency to close the gap with dense alternatives.

References

1. Qwen3.5-35B-A3B Model Card — Hugging Face
2. Qwen3.5 Blog Post — Qwen Official Blog
3. Alibaba Cloud Model Studio — Hosted API

---