Key concepts like events, topics, partitions, replicas, producers, and consumers form the foundation of Kafka's distributed system.
Understand your specific use case to decide whether to prioritize moving large volumes of data (throughput) or ensuring messages arrive quickly (latency).
Minimize end-to-end message delivery time by tuning producer parameters like linger.ms and acks, and consumer settings like fetch.min.bytes.
The sticky partitioner improves batching for messages without keys by sending them to a single partition until a batch is full, reducing latency compared to the old round-robin strategy.
Maximize the rate of data movement by increasing partition count, tuning producer batching with batch.size and linger.ms, and enabling compression.
Employ stress, spike, and soak testing methods while following best practices like system warmup, multiple runs, and analyzing percentiles to get accurate performance results.
Gain observability into your cluster's health by collecting JMX metrics from brokers and clients, alongside essential system metrics like CPU, disk, and network usage.
Leverage the out-of-the-box kafka-producer-perf-test and kafka-consumer-perf-test command-line tools for basic performance benchmarking.
Use advanced tools like Trogdor for fault injection and workload execution, or Apache JMeter with the Pepper-Box plugin for more complex and controlled load testing scenarios.
38:50Lucia Cerchie
54:29DeveloperSteve
24:53Dainius Jocas
52:15Alex Soto
30:51Soroosh Khodami
45:48Denis Washington & Olli Salonen
57:19Mary Grygleski
39:20Dan Erez
.gif?w=240&auto=compress,format)
.gif?w=240&auto=compress,format)
Jobs that call for the skills explored in this talk.
AUTO1 Group SE
Berlin, Germany
evoila Frankfurt GmbH
Mainz, Germany
smartclip Europe GmbH
Hamburg, Germany
