Kafka exporter
Kafka exporter configuration
Every section and key of the Kafka exporter's sqlload.toml, with defaults.
The exporter reads sqlload.toml. ${ENV:NAME} is replaced from the environment; $${ is a
literal ${. Unknown keys are refused. Sizes are a number of bytes or text such as 32MB (10⁶) or
32MiB (2²⁰); durations are text such as 500ms, 60s, 5m or 7d. The repository’s
sqlload.example.toml sets
every section.
[source]
Section titled “[source]”| Key | Default | Meaning |
|---|---|---|
kind |
required | kafka. |
brokers |
required | The bootstrap brokers. |
include |
required | Topic patterns to export. |
exclude |
none | Topic patterns to leave out. |
consumer_group |
sqlload-{stream} |
The consumer group template; see Naming topics. |
start_from |
latest |
Where a group with no committed offset starts: earliest or latest. It applies only to a group that has no committed offset: when the committed offset can’t be read, the read is retried for 4 minutes and the stream then stops with an error, rather than start from start_from, which with latest would skip records. |
discovery_interval |
5m |
How often new topics are looked for. |
shards |
1 |
How many instances split the streams between them. Each runs with its own --shard. |
shard |
0 |
This instance’s index below shards. run --shard, or SQLLOAD_SHARD, overrides it. |
prefetch_budget |
1GiB |
Kafka prefetch shared by all consumers, at most 64 MiB and at least 1 MiB each. |
startup_batch |
20 |
Streams start this many at a time; 0 starts them all at once. |
startup_interval |
500ms |
The pause between startup batches. |
[source.properties] passes any librdkafka consumer property, such as security.protocol or
sasl.mechanisms. queued.max.messages.kbytes replaces the prefetch share, and
fetch.wait.max.ms defaults to 2000.
[destination]
Section titled “[destination]”| Key | Default | Meaning |
|---|---|---|
kind |
required | snowflake, databricks (beta) or bigquery (beta). |
Snowflake:
| Key | Default | Meaning |
|---|---|---|
account, user |
required | The Snowflake account and user. |
auth |
required | { method = "pat", token = "..." } or { method = "key_pair", private_key_file = "..." }. Browser sign-in (oauth_authorization_code) works too, but it needs a person at a terminal, so it doesn’t suit a long-running exporter. |
role |
none | The role to use. |
warehouse |
none | Needed for COPY loads, plan and apply, and required with s3.auto_ingest, whose loads are confirmed from COPY_HISTORY; streaming doesn’t use it. |
database, schema |
required | Where the tables, stages and pipes go. |
Databricks:
| Key | Default | Meaning |
|---|---|---|
host |
required | The workspace URL. |
client_id, client_secret |
required | An OAuth machine-to-machine client (a service principal). |
warehouse_id |
required | The SQL warehouse for plan, apply and COPY INTO. |
catalog, schema |
required | Where the tables and volumes go; stored in lower case. |
zerobus_endpoint |
none | https://<workspace-id>.zerobus.<region>.cloud.databricks.com; needed for staging.kind = "none". |
BigQuery:
| Key | Default | Meaning |
|---|---|---|
project |
required | The project the dataset is in; load jobs and queries run there too. |
dataset |
required | Where the tables go. It must exist; apply creates tables, not datasets. |
location |
required | The dataset’s location, such as EU or europe-west2. |
service_account_key |
required | The service-account key file’s JSON text, such as "${ENV:BIGQUERY_KEY}". |
[staging]
Section titled “[staging]”| Key | Default | Meaning |
|---|---|---|
kind |
required | none (Snowpipe Streaming, Zerobus Ingest on Databricks, the Storage Write API on BigQuery), internal (files on a table stage, in a volume on Databricks, or sent with each load job on BigQuery), s3 (files in your bucket; not on BigQuery) or gcs (BigQuery only: files in your Cloud Storage bucket). See How rows reach the warehouse. |
load |
auto |
Snowflake staged files: snowpipe, copy, or auto, which picks Snowpipe with a key pair and COPY otherwise. Databricks always uses COPY INTO. |
format |
json; parquet on Databricks and BigQuery |
Staged files: json (gzip NDJSON) or parquet (zstd). Databricks and BigQuery load Parquet only. |
retain |
7d |
internal: how long dated stage or volume folders are kept. |
max_in_flight |
32 |
Uploads, and separately loads, in flight at once across all streams. |
max_pending_files |
32 |
Snowpipe loads: files per stream sent but not committed yet. A full stream pauses its partitions. |
on_load_error |
halt |
A file Snowpipe reports as failed or partly loaded: halt stops the stream without committing it; skip logs and counts it, then commits past it. |
pending_check_after |
5m |
Snowpipe loads: when a file with no report is first checked, in load history and then the stage. With auto_ingest, also the shortest time between two COPY_HISTORY queries. |
pending_recheck_every |
10m |
How often it’s checked again. Slow loads are never failed. |
With Snowpipe loads, the loading role needs OPERATE on each pipe and MONITOR to read its load
reports. Pipes that sqlload apply creates are owned by it.
[staging.s3], for kind = "s3":
| Key | Default | Meaning |
|---|---|---|
bucket, region |
required | The bucket and its region. |
prefix |
"" |
The folder under the bucket. |
stage |
required on Snowflake | The external stage on it: a name in the destination schema, or DATABASE.SCHEMA.NAME. Databricks reads the bucket through a Unity Catalog external location instead and refuses this key. |
path |
{topic}/date={date}/ |
Each file’s folder under prefix, with {topic}, {table}, {stream}, {date} (UTC YYYY-MM-DD) and {hour} (UTC HH). |
endpoint, path_style |
none, false |
For S3-compatible stores. |
auto_ingest |
false |
Snowflake: load with an auto-ingest pipe on bucket notifications. Needs destination.warehouse; see Auto-ingest. |
[staging.gcs], for kind = "gcs" (BigQuery):
| Key | Default | Meaning |
|---|---|---|
bucket |
required | The Cloud Storage bucket, in or near the dataset’s location. |
prefix |
"" |
The folder under the bucket. |
path |
{topic}/date={date}/ |
Each file’s folder under prefix, with the placeholders of [staging.s3]. |
The exporter doesn’t delete files from the bucket; set a lifecycle rule on it.
[batching]
Section titled “[batching]”A stream’s records become one batch when any limit is reached.
| Key | Default | Meaning |
|---|---|---|
max_bytes |
32 MiB | Bytes per batch. |
max_rows |
100000 |
Rows per batch. |
max_age |
60s |
Age of the oldest record in the batch. BigQuery load jobs start at most once a minute per table, so a shorter max_age there doesn’t load sooner; see BigQuery. |
max_total_bytes |
2 GiB | Across every stream; any stream flushes early when it’s reached. |
max_request_bytes |
3,500,000 bytes | Streaming: bytes per request; compressed for Snowpipe Streaming, below its 4 MiB limit, JSON for Zerobus Ingest, and Arrow for the BigQuery Storage Write API, below its 10 MB limit. |
oversized |
fail |
Streaming: a record larger than one request, fail or skip. |
[[topics]]
Section titled “[[topics]]”Settings for topics matching a glob in match. Later entries win. Each can set start_from,
max_bytes, max_rows, max_age and oversized, and unwrap: columns filled from a member of
the JSON value instead, such as { VALUE = "value" }.
[[topics]]match = "shop.*.events"start_from = "earliest"max_age = "10s"[naming]
Section titled “[naming]”How topics map to tables, stages and pipes. See Naming topics.
[envelope]
Section titled “[envelope]”Every table has the same columns, filled from each record. On Snowflake:
| Column | Field | Type |
|---|---|---|
TOPIC |
topic |
VARCHAR |
KAFKA_TIMESTAMP |
timestamp |
NUMBER(38,0) |
KAFKA_TIMESTAMP_TYPE |
timestamp_type |
NUMBER(38,0) |
KAFKA_OFFSET |
offset |
NUMBER(38,0) |
KAFKA_PARTITION |
partition |
NUMBER(38,0) |
KEY |
key |
VARCHAR |
VALUE |
value |
VARCHAR |
EXPORTED_AT |
exported_at |
TIMESTAMP_TZ |
DATE |
date |
DATE, the record’s UTC date (its timestamp, else the export time) |
On Databricks the names are lower case and the types are STRING for topic, key and value,
BIGINT for kafka_timestamp and kafka_offset, INT for kafka_timestamp_type and
kafka_partition, TIMESTAMP for exported_at and DATE for date. A VARIANT column holds the
parsed JSON value; MAP, STRUCT and ARRAY columns are read from the JSON text. The Kafka
timestamp is milliseconds since the epoch, so its column must stay BIGINT. With Zerobus Ingest, a
TIMESTAMP column taken from the record (key, value or an unwrapped member) needs RFC 3339 text
or a date and time without a zone, read as UTC; any other value stops the stream with its topic,
partition and offset.
On BigQuery the names are lower case and the types are STRING for topic, key and value,
INT64 for the Kafka timestamp, timestamp type, offset and partition, TIMESTAMP for
exported_at and DATE for date. Columns can only take these types, or another name for
INT64 such as INTEGER; a config with any other type is refused. A TIMESTAMP column can also
hold the Kafka timestamp, which
is read as milliseconds. A TIMESTAMP or DATE column taken from the record needs RFC 3339 text or
a date and time without a zone, read as UTC; any other value stops the stream, naming the column
and the value.
| Key | Default | Meaning |
|---|---|---|
columns |
the table above | A list of { name, field, type }. field is one of the fields above, headers or null. |
time_zone |
UTC |
The IANA zone whose UTC offset exported_at values carry, such as America/Los_Angeles. |
error_logging |
true |
Snowflake streaming: create tables with Snowflake’s ERROR_LOGGING. |
[metrics]
Section titled “[metrics]”| Key | Default | Meaning |
|---|---|---|
listen |
none | The address of the metrics and health endpoint, such as 0.0.0.0:9898. |