To ingest Kafka topic records into Azure Data Explorer (ADX), run the Kusto Kafka Sink connector on Kafka Connect. The worker reads the topic and queues records for ingestion through ADX’s ingestion endpoint; Event Hubs is not a required hop in this direct workflow. You will need an ADX database and matching table/mapping, a connector configuration that maps the topic to that destination, and valid ADX authentication.
How the Kafka-to-ADX data path works
The direct workflow is Kafka topic → Kafka Connect worker → Kusto Kafka Sink connector → ADX ingestion endpoint → destination table. The connector class in Microsoft’s tutorial is com.microsoft.azure.kusto.kafka.connect.sink.KustoSinkConnector. Kafka Connect hosts the connector, which reads topic data and queues it for ADX ingestion; application code is not required for this connector path. See Microsoft’s Kafka ingestion tutorial.
Microsoft lists batching and streaming sink modes, with logs, telemetry, and time-series data among the use cases. Those are supported scenarios, not a promise of a particular latency or throughput. The documentation does not establish a universal performance winner or benchmark for Kafka-to-ADX ingestion. See the ADX integrations overview.
Prerequisites
- An Azure subscription and an ADX cluster with a database.
- A Kafka topic containing records in a format you can map to the ADX destination table.
- Azure CLI, Docker, and Docker Compose for the documented self-contained lab.
- A Kafka Connect worker able to load a compatible Kusto Kafka Sink connector release. A production deployment may use a separately managed worker; verify the connector release and its configuration requirements for that environment.
- An identity with the required ADX permissions, plus a secure way to provide its credentials to the connector.
The Docker-based lab is one way to run the components together, not a requirement for production architecture. Follow the Microsoft tutorial for its environment-specific setup details.
Free tools Windows power users keep installed
One-click scans. No signup required.
#1 Best Overall
Create the ADX destination table and mapping
Before starting the connector, create the database destination table and an ingestion mapping that match the records the connector will send. A mapping associates fields in the incoming data with columns in the table; mismatched names, types, or serialization can prevent records from appearing as expected.
The connector configuration associates each Kafka topic with its ADX database, table, data format, and mapping. Treat these as a single routing definition: the topic name must be the one the worker consumes, and the database, table, format, and mapping name must identify the intended destination and agree with its schema.
The tutorial’s sample uses Kafka Connect string converters. If your topic carries structured or otherwise differently encoded records, select converters and an ADX ingestion format/mapping appropriate to that representation rather than copying the sample unchanged.
Configure the connector, endpoints, and identity
Configure the Kafka Connect sink with the connector class com.microsoft.azure.kusto.kafka.connect.sink.KustoSinkConnector, the topic-to-destination association, and both the ADX ingestion and query URIs. The sample’s default authentication uses a Microsoft Entra service principal. Microsoft also describes a managed identity option; that requires configuring the connector’s documented identity strategy and granting the identity the permissions needed for ingestion.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Rank #3
Authentication details depend on where the worker runs and which identity model you choose. Check the current documentation for the connector version deployed, especially before reusing older managed-identity examples. Keep client secrets and other credentials out of checked-in configuration; supply them through your deployment’s approved secret-management mechanism.
Use the full sample configuration and setup sequence in Microsoft’s Kafka-to-ADX tutorial, adapting its topic, database, table, format, mapping, URIs, converters, and identity values to your environment.
Start the sink and verify ingestion
- Start the Kafka Connect worker and ensure it can reach Kafka and the configured ADX endpoints.
- Submit the sink connector configuration through Kafka Connect’s REST API as described in the tutorial. Use the connector name and configuration for your deployment rather than assuming a sample name or endpoint.
- Check the connector status using Kafka Connect’s REST status endpoint. Confirm the connector and its task are running, and inspect worker or connector logs for authentication, connectivity, conversion, or mapping errors.
- Query the configured destination table in ADX and confirm that the expected records are present. A running task or accepted/queued records alone does not establish that the data is queryable in the table.
If records do not appear, verify the topic name and the configured database, table, format, and mapping; compare the table schema and ingestion mapping with the record serialization and converter settings; then check task status and logs. Re-check identity configuration and ingestion permissions if the connector reports authorization failures.
Tune batching for the workload
Batching occurs at both the sink connector and the ADX service. The connector’s flush size and ADX’s batching policy should be tuned together: larger batches can improve ingestion efficiency but can also mean records wait longer before being sent or made available. Microsoft’s tutorial provides starting-point guidance, not a universal optimal value or a throughput/latency guarantee.
Begin with the documented sample configuration only as a baseline. Observe task health, ingestion behavior, and the time between Kafka production and queryable ADX data under your own workload, then adjust connector flush settings and the ADX batching policy deliberately. Revalidate after changes rather than assuming one layer’s tuning alone determines the outcome.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.When Event Hubs belongs in the design
Event Hubs is a separate architectural choice, not a mandatory intermediary in the Kafka Connect sink path. It may be used as a Kafka-compatible endpoint for Kafka clients, or ADX may ingest continuously from an Event Hub through its Event Hubs data connection. The latter is a distinct ingestion route with its own consumer group, routing, and authentication requirements; see ADX’s Event Hubs ingestion overview.
| Design | Use it when | Key considerations |
|---|---|---|
| Kafka Connect with Kusto Kafka Sink | You have Kafka topics and want Kafka Connect to route their records directly to ADX. | Operate Kafka Connect and configure topic-to-database/table/format/mapping, ADX endpoints, identity, and batching. |
| Event Hubs Kafka endpoint | You want Kafka-compatible clients to connect to an Event Hubs namespace. | The namespace must be Standard tier or higher; the Basic tier does not support Event Hubs for Kafka. Consult the Event Hubs Kafka quickstart and Kafka developer guide for tier and client authentication details. |
| ADX Event Hubs data connection | You want ADX to continuously ingest from an Event Hub rather than use the Kafka sink connector path. | Configure the Event Hubs data connection’s own consumer group, routing, and key-based or managed-identity authentication as documented in the ADX overview. |
The appropriate route depends on broker ownership, whether a managed Kafka-compatible endpoint is needed, identity and routing needs, and which components your team will operate. The cited documentation does not establish a general cost or latency winner between these architectures.
Clean up the tutorial environment
When the lab is finished, stop and remove its Docker Compose services and other lab containers as described by the tutorial. Delete cloud resources created solely for the exercise—such as the test ADX cluster or database—only after confirming they contain no data you need and are not shared with another workload. Follow the cleanup steps in Microsoft’s tutorial.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
