To create Kafka topics dynamically from Apache NiFi, provision them explicitly before publishing—typically through a service or component that calls Kafka’s Admin API. PublishKafka sends records to its configured topic; setting that topic name, even with a NiFi parameter, is not the same as creating or configuring the topic. Broker-side auto-creation is another option, but it depends on broker policy and uses broker defaults rather than settings specified by the flow.
Does PublishKafka create a topic if it does not exist?
Do not treat PublishKafka as a topic-provisioning step. Its job is to publish FlowFile content to a configured Kafka topic. A topic property that contains a literal name, parameter reference, or dynamically supplied value tells the processor where to publish; it does not establish that the topic has been created with the settings your workflow requires. The NiFi 1.28.0 documentation for its Kafka 2.6 API component describes publishing and its processor properties, not explicit topic administration: PublishKafka component documentation.
Kafka may create a missing topic when a producer first publishes only if broker-side automatic topic creation is enabled and permitted by the cluster’s policy. That is a broker behavior, not proof that a NiFi processor explicitly created the topic. The Kafka operations guide covers both manual and automatic creation and notes that automatic creation uses broker defaults, which may need tuning: Kafka Basic Operations.
Use an explicit provisioning step when the flow must control topic settings
A runtime flow that needs predictable partitions, replication, or topic-level configuration should provision the topic before sending records. Kafka’s Admin API supports topic administration; the corresponding command-line workflow accepts explicit partition, replication-factor, and configuration arguments. Choose a purpose-built provisioning service or an approved administrative command mechanism appropriate to your NiFi installation. The cited NiFi material does not establish a built-in processor that creates topics, so verify the available extensions and behavior for the version you run.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problems- Derive and validate the topic name. Build it from trusted flow data or controlled parameters. Validate naming rules and guard against arbitrary input creating an unbounded number of topics.
- Submit a topic definition. Call the provisioning component with the intended topic name, partition count, replication factor, and any required non-default settings such as retention or cleanup configuration. Kafka documents explicit topic configuration in its operations guide.
- Route the administration result. Treat an already-existing topic as an expected, idempotent outcome only after interpreting the Admin API response. Route authorization, validation, connectivity, and broker errors to a defined failure or retry path.
- Publish only after creation is confirmed. Send the records to
PublishKafkaconfigured for that topic. NiFi parameters can make processor configuration reusable across environments, but parameter references are configuration substitution, not topic administration. See the NiFi User Guide and confirm expression-language support and lifecycle behavior for the installed processor.
Choose partitions and replication deliberately
Kafka partitions divide a topic’s log and bound the parallelism available to consumers. More partitions are not automatically better: increasing a topic’s partition count can change default key-to-partition assignments, potentially affecting ordering for keyed records. Existing records are not automatically redistributed when partitions are added. Set the count against expected workload and consumer parallelism, rather than applying an unexamined universal default.
Set the replication factor in coordination with the brokers available and the cluster’s resilience policy. Kafka’s documentation supports explicit configuration, but the appropriate values depend on the deployment. Do not assume a broker’s defaults match the requirements of every dynamically created topic.
Rank #2
Handle partial success and metadata visibility
A batch create request is not an all-or-nothing transaction: some requested topics can succeed while others fail. Track results per topic and make retries selective so a partial result does not accidentally trigger duplicate or conflicting work. Kafka’s KafkaAdminClient 4.1.2 API reference also notes that a successful create response can precede full cluster-wide metadata visibility by several seconds. Allow for that propagation window before treating a newly created topic as absent.
Separate topic-administration outcomes from producer outcomes in the flow. An invalid name or configuration, missing authorization, connectivity problem, partial batch result, temporary metadata delay, and record-production failure are different cases; assign retry limits and an operator-review or dead-letter route according to your operational policy. These routes are flow-design choices, not an automatic retry policy guaranteed by NiFi.
Recommended Free Tools
Rank #3
Choose where topic ownership belongs
| Approach | Control over settings | Operational owner | Trade-off |
|---|---|---|---|
| Pre-create topics outside NiFi | High; administrators set configuration and policy | Kafka or platform team | Simpler flow, but topic creation is a separate deployment step. |
| Provision through Kafka Admin API in the flow | High; the request can specify topic settings | Flow and platform integration owners | Enables runtime provisioning, but requires an administrative client or service, credentials, permissions, idempotency, and partial-failure handling. |
| Broker auto-creation on first publish | Generally broker defaults, unless those defaults are tuned | Kafka broker administrators | Requires less flow work, but policy may disable it and the flow has less direct control over settings. |
Pre-provision topics when platform policy requires naming review, quotas, ACL setup, or standardized retention. Use in-flow administration when runtime creation is genuinely needed and the responsible team can operate its identity and failure handling. Rely on auto-creation only when broker administrators have explicitly approved it and its defaults meet the use case.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Protect the administrative and publishing credentials
Topic creation and record production may use different components and identities; verify the permissions and secret handling of each. The NiFi 1.28.0 Kafka 2.6 PublishKafka documentation warns that a password placed in the dynamic sasl.jaas.config property is not secured and may be stored in clear text in flow.xml.gz and versioned flows. Use the sensitive-property and secret-management mechanisms supported by your deployed NiFi release, and scope the administrative identity to only the required operations and topics. Check version-specific behavior rather than assuming the cited warning applies identically to every release.
Quick Recap
Rank #4
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
