- Cloud Tracing, VK Cloud’s distributed tracing service, which enables applications to send spans via the OpenTelemetry protocol and visualize them through Jaeger...
- Initially, the Jaeger deployment page listed six potential backend options: Cassandra, Elasticsearch, Kafka, gRPC plugin, Badger, and memory.
- Consequently, the actual comparison focused on Cassandra and Elasticsearch, with ClickHouse emerging as a third candidate via the gRPC plugin.
VK Cloud Reveals Storage Choice for Cloud Tracing Service
Cloud Tracing, VK Cloud’s distributed tracing service, which enables applications to send spans via the OpenTelemetry protocol and visualize them through Jaeger UI and Grafana, faced a complex storage dilemma. For a public cloud environment handling data streams from hundreds of unrelated clients and storing tens of terabytes of telemetry, a solution was needed that ensured robust project data isolation and efficiently processed up to 1 MB/s per project, with access controlled by IAM tokens.
Navigating Jaeger Backend Options
Initially, the Jaeger deployment page listed six potential backend options: Cassandra, Elasticsearch, Kafka, gRPC plugin, Badger, and memory. However, as noted by Leonid Levin, PaaS Architect at VK Cloud, and his team, including technology evangelist Stas Pogorzhelsky, this list quickly narrowed down.
- Local storage solutions (Badger, memory) were deemed unsuitable for production environments.
- Kafka was identified as a buffer rather than a primary storage solution.
- The gRPC plugin serves to integrate systems not originally on the list.
Consequently, the actual comparison focused on Cassandra and Elasticsearch, with ClickHouse emerging as a third candidate via the gRPC plugin. VK Cloud experts conducted a detailed analysis, evaluating the performance and suitability of each solution for trace storage. Testing achieved impressive metrics, reaching up to 362,000 spans per second on four collector cores.
ClickHouse as the Optimal Tracing Solution
Particular attention was paid to how different systems handled point reads and attribute-based searches. The storage schema implemented with ClickHouse, as demonstrated by VK Cloud’s analysis, effectively met these critical requirements, offering a reliable and scalable solution for the Cloud Tracing service.
It’s interesting to see VK Cloud land on ClickHouse for Jaeger tracing. I’ve been running Jaeger with Elasticsearch for a while now, and while it works, managing the indices and shard allocation for cost-efficiency can be a headache, especially with high cardinality tags. I’ve been meaning to experiment with ClickHouse as a backend; the promise of better query performance for analytical queries is really appealing. My main concern is the operational complexity compared to a managed Elasticsearch service. For anyone else considering the switch, definitely test your most common trace queries extensively; I found some surprising performance drops with certain complex attribute searches in my initial ClickHouse POC.