A Kafka topic is a named stream of events, and a partition is one ordered log within that stream. Kafka preserves order within each partition—not across all partitions in a topic. That distinction determines how records are routed, how much work a consumer group can process in parallel, and how to choose a partition count.
What is a Kafka topic?
A topic is a named stream to which producers write records and from which consumers read them. Multiple producers can write to a topic, and multiple consumers can read it independently. Kafka retains records according to the topic’s retention settings; reading a record does not, by itself, remove it. Consumers can also control their position to replay retained data. See the Apache Kafka introduction.
A topic is divided into partitions, which Kafka distributes across brokers. This provides units for distributing stored data and client work. A topic is therefore not one indivisible log: its partitions are separate logs.
What is a partition, and what does an offset mean?
A partition is an append-only, ordered log. As records are appended, each receives an offset identifying its position in that partition. Offsets are partition-local: an offset does not identify a position in a single topic-wide sequence. The Kafka documentation on topics describes this log-and-offset model.
#1 Best Overall
For example, a topic with three partitions has three ordered sequences, each with its own offsets. A record at offset 12 in one partition is not necessarily earlier or later than a record at offset 12 in another.
How does Kafka preserve ordering?
Kafka guarantees order within a partition. It does not establish a total order among records in different partitions. Apache Kafka’s current introduction explains that records with the same event key, such as a customer or vehicle ID, are written to the same partition, so consumers of that topic-partition read its events in write order.
This makes keys useful when related events need to remain ordered. For instance, an application can use a customer ID to route that customer’s events together. The key-to-partition mapping is a producer-side assignment choice, however: Kafka clients or custom partitioners may use different mapping behavior under different configurations. The protocol documentation notes that clients address a particular partition and that the broker does not define an application’s mapping semantics: Kafka protocol guide.
If the application needs one total order for every record in a topic, a single partition is the conceptual option. That also limits the topic to one partition’s worth of consumer-group parallelism, so it trades processing distribution for a single ordered sequence.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Rank #3
How partitions provide consumer parallelism
Within a consumer group, Kafka assigns partitions among the group’s consumer instances. Each distinct partition can be actively processed by at most one consumer in that group at a time. Consequently, a group cannot actively process more distinct partitions than the topic provides; extra consumer instances can remain idle rather than create new parallel work.
More partitions provide more assignment units and can allow more group members to work concurrently, subject to the workload and operational limits. Partition count alone does not guarantee balanced work: a highly active key routed to one partition can make that partition a bottleneck even when others are less busy.
How replication differs from partition count
Partition count describes how many logs make up a topic. Replication factor describes how many copies Kafka maintains for each partition; it does not add partitions or create a topic-wide order. Under Kafka’s leader/follower design, each partition has a leader and may have followers. Writes go to the leader, while followers replicate the log. Replicas are placed on brokers to support availability and fault tolerance. See the Kafka 4.1 design documentation.
Replication consumes additional storage and replication work. A replication factor alone does not guarantee survival of a particular number of arbitrary broker failures: the safety of acknowledged records depends on configuration and the actual failure conditions. Apache Kafka gives three as an example of a common production replication factor, not as a universal requirement or recommendation (Kafka introduction).
How many partitions should a Kafka topic have?
There is no universal partition count supported by the documented mechanisms. Choose based on the ordering scope, desired parallelism, key distribution, and the capacity of the brokers and consumers handling the actual workload.
- Ordering scope: Decide whether ordering is needed per entity, such as per customer or order, or across the entire stream. Per-entity ordering can use keys to keep related records together; one total topic order points to a single partition.
- Parallelism: Estimate how much independent consumer-group work should be possible. A group’s active parallelism is bounded by its assigned partitions.
- Key distribution: Consider whether keys spread traffic evenly. A very high-volume key can concentrate its records on one partition.
- Availability and resources: Select replication and broker placement separately from partition count, balancing fault-tolerance needs against storage and replication costs.
- Measured workload: Validate the design against throughput, retention, message size, broker capacity, and consumer behavior. Kafka’s documentation explains these mechanisms but does not establish a numeric formula that fits every workload.
For configuration details or behavior that may vary by release, use the documentation for the Kafka version you run.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
