Qwen3.8-Flash-Next 125B Achieves Breakthrough Performance on 6 GB VRAM
PROSTO24 reports a significant advancement in large language model optimization. Developers have successfully run the full 125-billion parameter model, Qwen3.8-Flash-Next, which serves as the architectural foundation for the upcoming Qwen4, on a single RTX 4090 graphics card with a peak VRAM consumption of just 5.95 GB. This achievement was accomplished without the use of 4-bit quantization or distillation, utilizing a complete bf16 checkpoint for standard token generation.
According to Qwen’s internal data, the Qwen3.8-Flash-Next model exhibits competitive performance against leading closed-source frontier models, such as Opus 4.6 Max, across critical domains. These areas include programming tasks, agent interaction, and general computer use, underscoring its potential as a powerful open-source solution.
KV-Cache Optimization for Extended Contexts
In the operational context of models like Qwen3.8-27B, the issue of growing KV-cache with long contexts presents a significant challenge. For instance, the 27-billion parameter model in Q4TP format occupies approximately 14.3 GB for its weights. However, for the 16 full-attention layers of this hybrid model, the FP16 KV-state expands by 65,536 bytes with each new token. This implies that for a 64K token context, roughly 4 GiB could be required solely for the K/V cache.
To address this, an experimental O(1) CMF runtime mode has been developed. This mode replaces the dynamically growing KV-cache for these 16 layers with a fixed state. This fixed state comprises four sink tokens, a precise window of the last 128 tokens, and 32 anchor representations for the earlier part of the context. In this usage profile, the GPU-state attention consumes only 44.1 MiB and does not increase with context length, marking a crucial step in efficient memory management for large models.
The Qwen3.8-Flash-Next 125B achieving bf16 inference on 6GB VRAM is a significant leap, especially without 4-bit quantization. The O(1) CMF runtime mode for KV-cache optimization, leveraging sink tokens and a fixed window, is particularly noteworthy. This approach addresses a critical bottleneck in long-context inference and could drastically alter the viability of deploying such large models on consumer-grade hardware, impacting edge AI and democratizing high-performance LLM access.