LinkedIn Upgrades Service Discovery by Replacing ZooKeeper with Kafka and xDS
LinkedIn has upgraded its service discovery platform by replacing the legacy ZooKeeper system with Apache Kafka for writes and the xDS protocol for reads. This new architecture addresses past scalability issues and is designed for improved performance and availability. A 'Dual Mode' strategy ensures zero-downtime migration, while the new system can handle 40k client streams and 10k updates per second with independent operations across data centers.
Theia Market Signal Identification - AI Assisted
LinkedIn has transitioned its service discovery platform from Apache ZooKeeper to a new architecture utilizing Apache Kafka and the xDS protocol. This upgrade addresses capacity limits reached by 2025 due to scaling issues, including high latency and session timeouts from write spikes and read storms.
The new system employs an eventual consistency model, separating the write path through Kafka from the read path via an Observer service. The Observer can maintain 40k client streams and process 10k updates per second, operating independently in data centers while allowing client connections for failover.
Comments