Kafka

Kafka (`kafka`)

Publish events to Kafka/Redpanda clusters.

Messaging binary json raw

Minimal example

output:
  kafka:
    bootstrap-servers: ""

JSON

{
  "output": {
    "kafka": {
      "bootstrap-servers": ""
    }
  }
}

Minimal example
Fields
Reliability
Batching
Connection
Payload
Security
Performance
Partitioning
Advanced
Schema

Fields

Field	Type	Required	Description
`bootstrap-servers` Connection	`hostname` (`string`)	✅	Comma separated bootstrap servers. Examples: `example.com`, `localhost`
`retry` Reliability	`Retry`		Retry policy for failed deliveries.
`batch` Batching	`Batch`		Logical batching hints shared with other outputs.
`client-id` Connection	`string`		Optional producer client identifier.
`topic` Payload	`string`		Static topic name.
`topic-field` Payload	`field` (`string`)		Event field containing the target topic. Examples: `data_field`
`topic-pointer` Payload	`json-pointer` (`string`)		JSON pointer resolving to target topic.
`input-field` Payload	`field` (`string`)		Optional event field to treat as payload instead of entire event. Examples: `data_field`
`headers` Payload	`map` (`string`)		Message headers to attach to each record.
`payload` Payload	`Payload`		Encoding strategy for message payloads. Allowed values: `json`, `raw`, `avro-bridge`
`avro` Payload	`Avro`		Schema registry configuration required when payload=avro-bridge.
`auth` Security	`Auth`		Authentication and TLS configuration.
`acks` Reliability	`Acks`		Acknowledgement requirement. Allowed values: `none`, `leader`, `all`
`idempotent` Reliability	`boolean` (`bool`)		Enable Kafka idempotent producer mode.
`max-in-flight` Reliability	`number` (`integer`)		Maximum in-flight requests per connection (relevant for idempotency). Examples: `42`, `1.2e-10`
`linger-ms` Performance	`number` (`integer`)		Linger time before flushing a batch (ms). Examples: `42`, `1.2e-10`
`message-timeout-ms` Performance	`number` (`integer`)		Timeout waiting for delivery reports (ms). Examples: `42`, `1.2e-10`
`compression` Performance	`Compression`		Compression codec used by the producer. Allowed values: `none`, `gzip`, `snappy`, `lz4`, `zstd`
`partitioner` Partitioning	`Partitioner`		Partitioning strategy for the producer. Allowed values: `round-robin`, `hash-key`, `explicit`
`key-field` Partitioning	`field` (`string`)		Event field containing message key (used for partitioning). Examples: `data_field`
`key-pointer` Partitioning	`json-pointer` (`string`)		JSON pointer to message key.
`key-expression` Partitioning	`string`		Expression used to compute message key.
`partition-field` Partitioning	`field` (`string`)		Event field providing explicit partition. Examples: `data_field`
`partition-pointer` Partitioning	`json-pointer` (`string`)		JSON pointer providing explicit partition.
`partition-value` Partitioning	`string`		Static partition value when using explicit partitioner.
`global-options` Advanced	`map` (`string`)		Additional librdkafka producer configuration.
`topic-options` Advanced	`map` (`string`)		Topic level overrides forwarded to librdkafka.

Reliability

Show fields

Field	Type	Description
`retry`	`Retry`	Retry policy for failed deliveries.
`acks`	`Acks`	Acknowledgement requirement. Allowed values: `none`, `leader`, `all`
`idempotent`	`boolean` (`bool`)	Enable Kafka idempotent producer mode.
`max-in-flight`	`number` (`integer`)	Maximum in-flight requests per connection (relevant for idempotency). Examples: `42`, `1.2e-10`

Batching

Show fields

Field	Type	Required	Description
`batch`	`Batch`		Logical batching hints shared with other outputs.

Connection

Show fields

Field	Type	Required	Description
`bootstrap-servers`	`hostname` (`string`)	✅	Comma separated bootstrap servers. Examples: `example.com`, `localhost`
`client-id`	`string`		Optional producer client identifier.

Payload

Show fields

Field	Type	Description
`topic`	`string`	Static topic name.
`topic-field`	`field` (`string`)	Event field containing the target topic. Examples: `data_field`
`topic-pointer`	`json-pointer` (`string`)	JSON pointer resolving to target topic.
`input-field`	`field` (`string`)	Optional event field to treat as payload instead of entire event. Examples: `data_field`
`headers`	`map` (`string`)	Message headers to attach to each record.
`payload`	`Payload`	Encoding strategy for message payloads. Allowed values: `json`, `raw`, `avro-bridge`
`avro`	`Avro`	Schema registry configuration required when payload=avro-bridge.

Security

Show fields

Field	Type	Required	Description
`auth`	`Auth`		Authentication and TLS configuration.

Performance

Show fields

Field	Type	Description
`linger-ms`	`number` (`integer`)	Linger time before flushing a batch (ms). Examples: `42`, `1.2e-10`
`message-timeout-ms`	`number` (`integer`)	Timeout waiting for delivery reports (ms). Examples: `42`, `1.2e-10`
`compression`	`Compression`	Compression codec used by the producer. Allowed values: `none`, `gzip`, `snappy`, `lz4`, `zstd`

Partitioning

Show fields

Field	Type	Description
`partitioner`	`Partitioner`	Partitioning strategy for the producer. Allowed values: `round-robin`, `hash-key`, `explicit`
`key-field`	`field` (`string`)	Event field containing message key (used for partitioning). Examples: `data_field`
`key-pointer`	`json-pointer` (`string`)	JSON pointer to message key.
`key-expression`	`string`	Expression used to compute message key.
`partition-field`	`field` (`string`)	Event field providing explicit partition. Examples: `data_field`
`partition-pointer`	`json-pointer` (`string`)	JSON pointer providing explicit partition.
`partition-value`	`string`	Static partition value when using explicit partitioner.

Advanced

Show fields

Field	Type	Required	Description
`global-options`	`map` (`string`)		Additional librdkafka producer configuration.
`topic-options`	`map` (`string`)		Topic level overrides forwarded to librdkafka.

Schema

Retry Fields
Batch Fields
Avro Fields
Auth - Sasl Fields
Auth - Tls Fields
Auth Fields
Headers Table
Auth - Sasl - Extensions Table
Auth - Config Table
Global Options Table
Topic Options Table
Batch - Mode Options
Payload Options
Auth - Sasl - Mechanism Options
Acks Options
Compression Options
Partitioner Options

Retry Fields

Field	Type	Required	Description
`timeout`	`time-interval` (`string`)	✅	timeout (e.g. 500ms, 2s etc. - default is 30). Examples: `500ms`, `2h`
`retries`	`number` (`integer`)		number of retries. Examples: `42`, `1.2e-10`

Batch Fields

Field	Type	Required	Description
`fixed-size`	`number` (`integer`)		maximum number of events in an output batch. Examples: `42`, `1.2e-10`
`mode`	`Mode`	✅	If ‘document’ send on end of document generated by input. If ‘fixed’, use `fixed_size`. Allowed values: `fixed`, `document`
`timeout`	`time-interval` (`string`)	✅	interval after which the batch is sent, to keep throughput going (default 100ms). Examples: `500ms`, `2h`
`header`	`multiline-text` (`string`)		put a header line before the batch.
`footer`	`multiline-text` (`string`)		put a header line after the last line of the batch.
`use-document-marker`	`boolean` (`bool`)		Enrich the job metadata with a document marker (for document handling in batch mode).
`wrap-as-json`	`boolean` (`bool`)		Format the output batch as a JSON array.

Avro Fields

Field	Type	Required	Description
`url`	`url` (`string`)	✅	Examples: `https://example.com/path`
`username`	`string`
`password`	`string`
`access-token`	`string`
`subject-suffix`	`string`		Optional suffix appended to subject names (e.g. `-value`).
`use-latest`	`boolean` (`bool`)		When true, fetch the latest compatible schema instead of an explicit version.

Auth - Sasl Fields

Field	Type	Required	Description
`mechanism`	`Mechanism`	✅	Allowed values: `plain`, `scram-sha256`, `scram-sha512`, `o-auth-bearer`
`username`	`string`
`password`	`string`
`token`	`string`
`token-command`	`string`
`extensions`	`map` (`string`)

Auth - Tls Fields

Field	Type	Required	Description
`ca-certificate`	`string`
`client-certificate`	`string`
`client-key`	`string`
`insecure-skip-verify`	`boolean` (`bool`)