# Qwen3.8-Flash-Next Qwen3.8-Flash-Next is an open-weight multimodal [[AI Mixture of Experts (MoE)|MoE]] model from [[Qwen]], released on 24 August 2026, and, more interestingly, **an early preview of the architecture Qwen 4 will use**. 125B parameters, 6B active, plus a 51B n-gram embedding table that can live outside GPU memory. It ships under Alibaba's own Qwen Community 1.0 licence rather than Apache 2.0. The hosted version on Qwen Cloud is called Qwen3.8-Flash and adds 1M context by default and built-in tools. [[Simon Willison]]'s one-line summary: "It's pretty big: 125B parameters but only 6B active which means it gets a significant performance boost." He ran it on a DGX Spark with the Unsloth quants (72.5 GB UD-IQ1_S and 78.9 GB UD-Q2_K_XL) and got decent pelicans out of the 2-bit at `xhigh` effort. ## What's new in the architecture Alibaba frames this as the answer to "how efficiently can we scale" rather than "how much". Four pieces: - **Hybrid attention with Qwen Sparse Attention (QSA).** The Gated DeltaNet + Gated Attention pairing of Qwen 3.5 to 3.8 becomes Gated DeltaNet + QSA. QSA selects *micro-blocks* of tokens rather than individual tokens, with a small MQA indexer (4 query heads, 1 shared key head) and a budget of 512 blocks / 2,048 tokens per query. Same family of idea as DeepSeek's DSA and CSA2 (see [[DeepSeek V4.1 Flash]] and [[AI Sparse Attention]]); the point is long-context latency for agentic loops - **Gated Residual.** Widened residual streams (4 branches, bottleneck rank 320) with a data-dependent read gate and a per-branch write gate. More expressiveness per layer, stable training, small inference cost - **N-gram Embedding.** 20 million bigram/trigram entries (51B parameters) indexed at layer 2. Embeddings scale parameters with almost no compute and are easy to offload, so this is a way to make a model "know more" without making it slower. It's the same trick as DeepSeek's Engram memory, and it's why the 51B can sit on SSD or system RAM with a modest speed penalty (an HN report on a Mac Studio: ~300 prefill / 33 tokens/s with SSD offload versus 550 / 40 in RAM) - **Training recipe.** Muon and AdamW applied to different weight categories; no batch-size warm-up, straight to target batch size, guided by refitted scaling laws Layout: 48 layers as 12 blocks of three (Gated DeltaNet → MoE) followed by one (QSA → MoE); 512 experts, 10 routed + 1 shared active, expert width 640; one MTP layer; 262K native context, extensible to 1M. Vision encoder included. ## Benchmarks Alibaba's table, against its own siblings, DeepSeek V4 Flash 0731, and Claude Opus 4.6 Max: | Benchmark | **Flash-Next** (125B/6B) | [[Qwen3.8-27B]] | Qwen3.7-Plus (397B/17B) | DS V4 Flash (284B/13B) | Opus 4.6 Max | |---|---|---|---|---|---| | DeepSWE 1.1 | **58.7** | 42.2 | 16.5 | 54.4 | | | SWE-bench Pro | **62.5** | 61.7 | 55.8 | 56.0 | 53.4 | | SWE-bench Multilingual | **81.0** | 73.8 | 75.8 | | 77.5 | | NL2Repo-Bench | 48.1 | 42.3 | 41.1 | **54.2** | 47.6 | | CoWorkBench | **73.9** | 70.7 | 65.1 | 45.1 | 68.2 | | JobBench | **55.7** | 33.4 | 27.6 | 41.3 | 36.6 | | Agents' Last Exam (score) | **51.2** | 42.9 | 33.6 | | | | Toolathlon Verified | **73.5** | 67.1 | 50.6 | 70.3 | | | IFBench | **81.3** | 79.5 | 79.1 | 79.2 | 62.5 | | GPQA Diamond | **91.7** | 89.2 | 90.3 | 90.8 | 91.3 | | HLE | 35.9 | 30.8 | 34.7 | 33.8 | **40.0** | | LiveCodeBench v6 | **91.9** | 90.3 | 89.6 | 90.6 | 88.8 | Vision rows follow the same pattern: it leads OSWorld 2.0, AndroidWorld (84.5), RealWorldQA, ERQA, and long-video understanding among the models listed. The read: at 6B active it matches or beats a 17B-active model three times its size and a 13B-active DeepSeek on most agentic rows. That's the whole argument for the architecture. It's also Alibaba's table, measured mostly through the [[Claude Code]] harness, with the usual in-house benchmarks mixed in, and the comparison column is Opus 4.6, not anything current. ## Why it matters Three labs in one month shipped the same shape of model: many total parameters, very few active, a large lookup memory that lives off-GPU, and sparse attention designed for the agent loop rather than the chat box. [[DeepSeek V4.1 Flash]] (552B / 8B-16B, 196B Engram) and this one are the clearest examples. The frontier for open weights is no longer "biggest model" but "most capability per active parameter", and the design choices are converging. For local use it's a mixed gift. 6B active means it decodes fast. 125B plus 51B of embeddings means it doesn't fit on one consumer card; you want a DGX Spark, a Mac Studio, or a Strix Halo box with fast storage for the n-gram table. The interesting part for the vault is that "engram on SSD" makes a 176B-parameter model runnable on machines that could never hold it in memory. ## References - Model card: https://huggingface.co/Qwen/Qwen3.8-Flash-Next (FP8: https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8) - Blog post: https://qwen.ai/blog?id=qwen3.8-flash-next - Technical report: https://github.com/QwenLM/Qwen3.8-Flash-Next/blob/main/tech_report.pdf - Hosted version (Qwen3.8-Flash): https://www.qwencloud.com/models/qwen3.8-flash - [[Simon Willison]], 2026-08-26: https://simonwillison.net/2026/Aug/26/qwen38-flash-next/ - Unsloth quants: https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF ## Related - [[Qwen]] - [[Qwen 3.8]] - [[Qwen3.8-27B]] - [[Qwen3.6-35B-A3B]] - [[DeepSeek V4.1 Flash]] - [[DeepSeek v4]] - [[AI Mixture of Experts (MoE)]] - [[AI Sparse Attention]] - [[AI KV Cache]] - [[AI Open Weight Models]] - [[Unsloth]] - [[llama.cpp]] - [[Simon Willison]] - [[Large Language Models (LLMs)]]