Revolutionizing Local AI Agents: Speed and Performance on Consumer Hardware
The development of local code agents, designed to operate entirely on user devices without reliance on cloud resources, faces significant challenges, primarily concerning processing speed. A major hurdle is that AI agents constantly re-read extensive contexts and require considerable ‘thinking’ time, which on local hardware can translate into minutes of waiting for each task.
Enthusiasts and developers are actively seeking optimization methods to transform standard PCs into powerful platforms for local AI models. One such experiment yielded impressive results: the Qwen3.8-27B model was significantly accelerated on a configuration featuring two RTX 3060 12GB graphics cards on an older Z97 platform. Initial processing speeds of 9 tokens/s with long contexts were boosted to approximately 40 tokens/s in real agent sessions with contexts exceeding 100K. This was achieved without reducing the model size or sacrificing the 128K context window.
Key Optimization Techniques for Enhanced Performance
To achieve these results, the following techniques were implemented:
- Tensor split without CUDA P2P: Efficiently distributing the workload across multiple GPUs without relying on direct peer-to-peer data exchange.
- Q8 KV-cache: Optimizing the key-value cache to an 8-bit representation to reduce memory consumption and accelerate access.
- Integrated MTP: Implementing optimized multi-threaded processes.
- N-max selection: Carefully tuning the maximum number of tokens for generation.
These methods not only dramatically increased operational speed but also enabled successful testing on long-context coding tasks, confirming high-quality outcomes. Unsuccessful experiments with NCCL and shared buffers also provided valuable lessons, underscoring the importance of a comprehensive approach to optimization.
I’ve been experimenting with Qwen3.8-27B locally on my RTX 4090, and the long context performance has been a real bottleneck. I’m hitting similar token/s rates to your initial findings, around 10-12 t/s with larger contexts. The Q8 KV-cache optimization sounds promising; I’ve been hesitant to quantize, but if it doesn’t sacrifice the 128K context, it’s definitely worth exploring. My practical tip would be to monitor VRAM usage closely, as exceeding it often silently tanks performance rather than crashing. I find that using `nvidia-smi` in a loop helps track it in real-time.