There are many design decisions that contributed to Kafka’s performance. In this post, we’ll focus on two. We think these two carried the most weight.
The first one is Kafka’s reliance on Sequential I/O.
The second design choice that gives Kafka its performance advantage is its focus on efficiency: zero copy principle.
The diagram above illustrates how the data is transmitted between producer and consumer, and what zero-copy means.
* 2.1: The data is loaded from disk to OS cache
* 2.2 The data is copied from OS cache to Kafka application
* 2.3 Kafka application copies the data into the socket buffer
* 2.4 The data is copied from socket buffer to network card
* 2.5 The network card sends data out to the consumer
* 3.1: The data is loaded from disk to OS cache
* 3.2 OS cache directly copies the data to the network card via sendfile() command
* 3.3 The network card sends data out to the consumer
Zero copy is a shortcut to save multiple data copies between the application context and kernel context.
There are many design decisions that contributed to Kafka’s performance. In this post, we’ll focus on two. We think these two carried the most weight.
The first one is Kafka’s reliance on Sequential I/O.
The second design choice that gives Kafka its performance advantage is its focus on efficiency: zero copy principle.
The diagram above illustrates how the data is transmitted between producer and consumer, and what zero-copy means.
* 2.1: The data is loaded from disk to OS cache
* 2.2 The data is copied from OS cache to Kafka application
* 2.3 Kafka application copies the data into the socket buffer
* 2.4 The data is copied from socket buffer to network card
* 2.5 The network card sends data out to the consumer
* 3.1: The data is loaded from disk to OS cache
* 3.2 OS cache directly copies the data to the network card via sendfile() command
* 3.3 The network card sends data out to the consumer
Zero copy is a shortcut to save multiple data copies between the application context and kernel context.