Managing GPU Failures in Kubernetes Clusters

In the realm of high-performance computing, particularly when training large-scale artificial intelligence models, hardware stability is paramount. A compelling report from 2024 detailed that Meta’s cluster, comprising 16,384 H100 cards, experienced 419 unexpected interruptions over 54 days during the training of the Llama 3 405B model, averaging approximately one failure every three hours. Notably, 58.7% of these incidents were attributed to GPU hardware and its memory, while central processing unit (CPU) failures accounted for only two instances during the same period.

Despite the lack of more recent public data on such a large scale, the adoption of Kubernetes for managing containerized workloads has only intensified. A 2025 CNCF survey revealed that 82% of container users operate Kubernetes in production environments, and 66% of companies utilizing generative AI models rely on it for inference tasks.

Addressing GPU Failure Handling in Kubernetes

Historically, Kubernetes has approached hardware failures with the assumption that restarts are the primary remedy. While this strategy is effective for stateless services, it proves inadequate for pods leveraging GPUs. Should a container utilizing a GPU crash due to device failure, kubelet will attempt to restart it on the same faulty hardware, leading to a perpetual cycle of failures and a persistent ‘Pending’ state.

Stas Pogorzhelsky, a technology evangelist at VK Cloud, points out that while topics such as GPU allocation via DRA and general cluster fault tolerance have been covered, there has been less focus on the post-failure scenario of a dedicated device. The forthcoming analysis will visually demonstrate a GPU failure, illustrate its detection via kubectl, and, critically, introduce mechanisms available in Kubernetes version 1.36. These new capabilities offer a solution to escape the endless ‘Pending’ state caused by a failed GPU without necessitating a complete node shutdown.