Skip to content

CLI Reference

load

Load the selected tables of a project, reload them from scratch, follow a Postgres CDC source, and print the result as JSON.

sqlload load [SELECTOR ...] [OPTIONS]

sqlload load loads the selected tables, creating any that don’t exist yet. It prints the plan’s closing line and any errors and warnings, then a line per table as it finishes. It takes the same options as plan.

$ sqlload load
✓ Plan complete 2 to load (sqlload plan shows the tree)
loaded raw.app__customers 50 rows 0.0s
loaded raw.app__orders 1,000 rows 0.0s
✓ Completed 2 loaded, 1,050 rows, 0.1s
├── slowest raw.app__orders 0.0s
└── peak memory 85.4 MiB

Each table prints created, altered, loaded or failed; a loaded line ends with the table’s new cursor position when it keeps one. A failed table doesn’t stop the others. The summary counts them, names the slowest table, gives the run’s peak memory, and after failures says what a rerun does. The command exits 1, or 75 when every failure may pass on a rerun (see Retrying failed runs). A table the plan skips, such as a files table with no new file, isn’t read.

On a terminal, a live block under these lines shows how many tables are done, each running table with its rows so far, the tables waiting, and any rate_limit in use. It’s redrawn in place and cleared at the end. See Events for --progress.

--fail-fast stops the load once a table fails: no other table starts, and tables already loading finish and keep what they committed. The summary names the tables that never started under not started, and the command exits as it would have otherwise: 1, or 75 when every table that failed, including any still loading when the first failed, may pass on a rerun. A rerun loads the tables that didn’t start as usual.

$ sqlload load --fail-fast --concurrency 2
…
loaded raw.shop__customers 400,000 rows 0.8s
loaded raw.shop__refunds 200,000 rows 0.4s
failed raw.shop__returns checks.min_rows failed: 50,000 rows staged, fewer than the 100,000 required; nothing was published
loaded raw.shop__orders 3,000,000 rows 2.4s
✗ Completed with failures 3 loaded, 1 failed, 1 not started, 3,600,000 rows, 2.5s
├── not started raw.shop__shipments (--fail-fast)
├── slowest raw.shop__orders 2.4s
├── peak memory 314 MiB
└── next the 2 unchanged tables load; the 3 that committed carry on from their positions

--fail-fast and --memory-per-table don’t combine with --follow.

Large loads commit along the way, so a failure keeps what committed and the next run carries on from there; a delete_insert, append or merge into ClickHouse read in windows or cursor order publishes once per run instead (a merge also every 10 million rows). See Batches. To read a range instead of continuing from the destination’s maximum, give --start-cursor and --end-cursor, or their typed forms --start-cursor-ts/--end-cursor-ts and --start-cursor-int/--end-cursor-int; see Backfilling a range.

--reload rebuilds tables from scratch. On a terminal it shows the plan and asks; in scripts, add --yes:

$ sqlload load --reload --yes app__customers
✓ Plan complete 1 object to create, 1 to load (sqlload plan shows the tree)
loaded raw.app__customers 50 rows 0.0s
✓ Completed 1 loaded, 50 rows, 0.1s
├── slowest raw.app__customers 0.0s
└── peak memory 75.1 MiB

--reload takes --start-cursor and --end-cursor, and rebuilds the table with only the rows in that range, as SQLBuild’s --full-refresh does with an interval. Rows outside the range are dropped, and plan warns which:

$ sqlload plan -v --reload --start-cursor 100000 --end-cursor 200000 shop__orders
…
├── reads order_id from 100000 (--start-cursor), below 200000 (--end-cursor)
…
└── commits 2 windows of 50000, read one after another into a new table, swapped in once at the end
Warnings (1)
└── shop__orders
└── --reload rebuilds the table with only the rows from 100000 (--start-cursor), below 200000 (--end-cursor): rows before 100000 and from 200000 on are dropped

Without --start-cursor, a reload starts from cursor_start, or from the first row. An integer batch_size on a source that can’t report its maximum needs --end-cursor, with or without --reload; see Batches.

A postgres_cdc table is copied again with --resnapshot instead; --reload refuses it. See Reloading. --reload is also refused on Postgres targets.

On a files source, --reload reads every file again; a merge table merges them one at a time into the new table, and after_load = { move_to = ... } reads the moved files too. See Reloading a files table.

Every load reports the most memory the process held at once: under peak memory in the summary, as peak_memory_bytes in the JSON report and in the run_finished event. It’s the peak for the whole run, all tables together: ODBC drivers, DuckDB and Python sources count too. Under the Python package, it includes the Python interpreter. It’s the peak resident set (ru_maxrss) on Linux and macOS, and the peak working set on Windows.

On Linux, --memory-per-table also measures each table. It loads one table at a time, whatever --concurrency or concurrency say, resets the process’s high-water mark before each table and reads it after:

$ sqlload load --memory-per-table
✓ Plan complete 2 to load (sqlload plan shows the tree)
loaded raw.app__customers 50 rows 0.0s peak memory 80.7 MiB
loaded raw.app__orders 1 row 0.0s peak memory 84.7 MiB
✓ Completed 2 loaded, 51 rows, 0.1s
├── slowest raw.app__orders 0.0s
└── peak memory 84.7 MiB

The JSON report and the events give the figures in bytes:

$ sqlload load --json | jq -c .summary
{"created":0,"altered":0,"recreated":0,"loaded":2,"failed":0,"skipped":0,"rows":51,"peak_memory_bytes":90210304}
$ sqlload load --memory-per-table --events jsonl 2>&1 >/dev/null | grep -E 'table_done|run_finished'
{"schema_version":0,"run_id":"2e96ce9d-…","at":"2026-11-15T02:06:38.244Z","table":"raw.app__customers","event":"table_done","rows":50,"elapsed_ms":22,"peak_memory_bytes":84353024}
{"schema_version":0,"run_id":"2e96ce9d-…","at":"2026-11-15T02:06:38.266Z","table":"raw.app__orders","event":"table_done","rows":1,"elapsed_ms":22,"peak_memory_bytes":89116672}
{"schema_version":0,"run_id":"2e96ce9d-…","at":"2026-11-15T02:06:38.267Z","event":"run_finished","loaded":2,"failed":0,"not_started":0,"rows":51,"elapsed_ms":113,"exit_code":0,"peak_memory_bytes":89116672}

A table’s figure is the most the process held while it loaded, so it includes what the process already held when the table started, such as memory an earlier table left allocated. Compare the tables of one run: the table that uses too much stands out. On other systems, plan, apply and load refuse the flag: --memory-per-table needs Linux; the run's peak is in every report. See Finding a source that uses too much memory.

A run stays under one memory budget, and plan -v shows it first, with where it came from and how it’s shared:

$ sqlload plan -v --memory-limit 1200MB
Memory 1.1 GiB budget (--memory-limit); 39.8 MiB held before loading; DuckDB 488 MiB (memory_limit 244 MiB, 2 threads); 154 MiB per table, 4 at once
Objects (2) sqlload apply
├── raw.customers create table, 4 columns
└── raw.documents create table, 6 columns
Loads (2) into DuckDB dev sqlload load
└── app postgres into raw
├── customers replace
└── documents replace
✓ Plan complete 2 objects to create, 2 to load
  • The budget is memory_limit or --memory-limit. Without either, it’s 80% of the limit of the container (the cgroup’s memory.max) or the Windows job object SQLLoad runs in, else 50% of the machine’s memory, which other programs likely share. Where the memory can’t be read, the line says no budget and why, and the run has no budget.
  • Held before loading is what the process holds once it has connected, which comes off first.
  • DuckDB (and MotherDuck) targets take 30% of the rest together. Each one’s memory_limit is half its part, because DuckDB holds up to about twice its memory_limit, and never under 256MB. It runs a statement on one thread per 128 MiB of its memory_limit, between two and the machine’s cores.
  • Per table: the tables share what remains, as many at once as get at least 64 MiB each, up to concurrency. When fewer fit, the line says so: 1 at once (concurrency 8 doesn't fit). A table’s sources keep a quarter of its share each for what they read ahead: an ODBC fetch buffer, the pages an sl.http job requests ahead (read_ahead sends fewer at once when its pages are large), and a Postgres source’s batches.

memory_limit = "off" (or --memory-limit off) turns the budget off. The run then loads with no watchdog, no pieces (each commit is whole), DuckDB with its own memory_limit (80% of the machine’s memory) and threads and its own checkpoints, and an ODBC fetch buffer of 512 MiB per table. Only the row-group check stays, against DuckDB’s own memory_limit, since a checkpoint that can’t rewrite a row group fails whatever the budget. plan always shows the line, and load still reports its peak memory:

Memory off (memory_limit): no budget or watchdog, and DuckDB keeps its own memory_limit and threads

plan fails when the budget can’t hold what SQLLoad holds already, or leaves no table 64 MiB, naming the budget the run needs, and load stops there, before it creates or loads anything:

the 600 MiB memory budget leaves the tables 11.7 MiB after DuckDB's 488 MiB, under the 64 MiB one table loads in; raise memory_limit to at least 653MiB

DuckDB keeps values of about 85 KiB and more in memory from their insert until a checkpoint writes them to the file. So when the large values of a table’s open commit reach a quarter of DuckDB’s memory_limit, SQLLoad commits what it has written as a piece and carries on. Rows that go through a staging table first, as a merge or delete_insert does, count three times. Under send = "at_commit", the default on MotherDuck, the rows held for a commit are sent a piece at a time as well. How the pieces commit depends on the table:

  • A table the load builds and swaps in at the end (a first load, replace, --reload) commits its pieces into the build table, so nothing changes for readers.
  • append and merge commit each piece into the table. A delete_insert cuts its pieces only between values of its replace_key, or of its cursor for a window, so each piece replaces whole keys or a whole part of the window; its deletes keep the rows this load committed in earlier pieces. Until the last piece, a reader may see part of the load, as with commits every batch_size.
  • An append that stops between its pieces, even by a crash, leaves a marker; the next load of that table removes that load’s rows before it loads, so it reads them again. See Unfinished appends.

SQLLoad turns DuckDB’s automatic checkpoint off during a load, since one that runs out of memory leaves the database unusable. Instead it checkpoints after a piece as needed, and after each table that held large values.

Some rows can’t be cut small enough. Then only their table fails, naming the fix, while the other tables load:

  • one replace_key value, cursor value or partition whose rows are larger than a piece:

    failed raw.attachments attachments: the rows of one k value hold 300 MiB of values over 85 KiB, which a delete_insert replaces in one commit, past the 61 MiB of them DuckDB's 244 MiB memory_limit commits at once (through a staging table, they count three times); raise memory_limit to at least 23.5GiB
  • a table whose open row group, the rows past its last full 122,880, holds more large values than DuckDB can rewrite at a checkpoint. The message also gives what a full row group of such rows needs. This doesn’t apply to MotherDuck, which checkpoints on its own servers:

    failed raw.documents documents: its last row group would hold 366 MiB of strings, 366 MiB of them values over 85 KiB from this load (rows of about 1 MiB each), more than DuckDB rewrites at a checkpoint within its 244 MiB memory_limit; raise memory_limit to at least 2GiB (a row group fills at 122,880 rows, and rows this wide need a memory_limit of at least 640.1GiB by then)

Unnested child tables, Postgres CDC changes, the key scan of --sync-deletes, a merge that refuses repeated keys (on_duplicate_keys = "fail") and a table with checks that isn’t rebuilt aren’t cut into pieces: a refusing merge and checks must see every row of a commit before any of it is written. They fail the same way when one commit is too large, writing nothing of it. A CDC table then stops at that commit until memory_limit is raised, and its replication slot keeps the source’s write-ahead log meanwhile.

While a run loads, SQLLoad reads the memory of the whole process, and of a MotherDuck target’s bridge process, every 10 ms. From 80% of the budget, no table starts until it falls again, and pieces are cut four times smaller. From 90%, the table holding the most large values in memory stops: its running DuckDB statement is interrupted and rolled back, and it fails while the other tables carry on. A table holding none isn’t stopped.

failed raw.attachments raw.attachments stopped: the load held 1 GiB of its 1.1 GiB memory budget (90%); from 90% the table holding the most large values in memory stops, here this one with 39 MiB; raise memory_limit, or lower batch_size

From 95%, when no table holds large values to stop (memory taken by a Python function building a list, or an ODBC driver’s own buffers), or the table stopped at 90% didn’t bring memory back under 90% within half a second, the whole run stops. Every running DuckDB and MotherDuck statement is interrupted and rolled back, the running tables fail with the reason below, and the others don’t start. Tables that committed before stay committed, and an append stopped between its pieces is cleaned up by the next load. SQLLoad then writes its summary, the --json report and the run_finished event, and exits with 1 (retryable: false, since the same run would likely run out again) without waiting for a table busy in its source, such as a Python function that hasn’t yielded. A MotherDuck bridge ends with the process.

failed raw.events stopped: the run held 3.9 GiB of its 4 GiB memory budget (97%), and from 95% it stops as no table held large values to stop; raise memory_limit, or lower concurrency

The watchdog reads memory every 10 ms, so an allocation that grows past the limit the OS or a service manager sets within one reading is only caught by that limit.

sqlload load SOURCE --follow keeps streaming one postgres_cdc source after it catches up, writing a batch every batch_interval, until --max-duration passes or it gets Ctrl-C or SIGTERM. A second Ctrl-C or SIGTERM stops it at once, with exit code 130. See Keep streaming.

sqlload load app --follow --max-duration 10m

It prints the same plan line, table lines and summary as load, with a written line for each table in each batch, giving its rows and how far it is behind the server. On a terminal, a live block shows the rows and batches written so far.

apply --json and load --json print one JSON document on standard output when they finish, for scripts and orchestrators. Logs stay on standard error. The exit code is the same as without --json.

$ sqlload load --json
{
"schema_version": 1,
"command": "load",
"ok": false,
"retryable": true,
"started_at": "2026-11-15T02:24:02.691Z",
"finished_at": "2026-11-15T02:24:04.729Z",
"duration_seconds": 2.039,
"summary": {
"created": 1,
"altered": 0,
"recreated": 0,
"loaded": 1,
"failed": 1,
"skipped": 0,
"rows": 1200,
"peak_memory_bytes": 101089280
},
"plan": { "ok": true, "summary": "2 selected, 2 objects to create, 2 to load", … },
"objects": [
{ "action": "created", "name": "raw.shop__orders", "detail": "4 columns" }
],
"tables": [
{
"table": "raw.shop__orders",
"source": "shop",
"target": "dev",
"write_strategy": "replace",
"status": "loaded",
"rows": 1200,
"duration_seconds": 0.043,
"note": null,
"error": null,
"retryable": false
},
{
"table": "raw.shop__refunds",
"source": "shop",
"target": "dev",
"write_strategy": "replace",
"status": "failed",
"rows": null,
"duration_seconds": 1.999,
"note": null,
"error": "GET http://api.example.com/v1/refunds?page=3: HTTP 503: unavailable (after 4 attempts)",
"retryable": true
}
]
}
Field Meaning
schema_version 1. New fields can appear without a change; it changes only when a field changes meaning or goes away.
command apply or load.
started_at, finished_at, duration_seconds When the command started and finished, as RFC 3339 UTC timestamps, and the seconds between.
ok Whether every selected table planned and loaded: true exactly when the exit code is 0.
retryable Whether running the command again unchanged may fix every failure: true exactly when the exit code is 75. It’s false when ok is true, whenever a table failed for good, and whenever a plan error other than an unreachable source left one out. See Retrying failed runs.
summary How many objects were created, altered (each table once, however many changes it had) and recreated; tables loaded and failed, and skipped; rows loaded. peak_memory_bytes: the most memory the whole process held at once during the run, in bytes; see Peak memory.
plan The same document plan --json prints, with its own ok and retryable. Tables a plan error left out are in its errors, and tables with nothing to load in skipped. Its phases give each connection’s and discovery’s time: see Connecting and discovering. Its memory is the plan’s memory line, such as "4 GiB budget (memory_limit); …", or "off (memory_limit): …" with the budget off; see Memory budget.
objects One entry per change to a table or stage, by apply or during a load: its action (created, altered or recreated), its name and the detail. A table that gains two columns has two altered entries.
tables Each table loaded: its status (loaded or failed, or skipped for a table --fail-fast didn’t start, its error naming the table that stopped the run and retryable true; also for a table the watchdog’s run stop didn’t start, with retryable false), rows, duration_seconds, a note (the cursor window it read, or that its table waits for first data), and the error when it failed. retryable says whether that failure may pass on a rerun. source, target and write_strategy are null only for a load --follow that failed before any table. A files source with after_load also has after_load: its action (delete or move), status (done, or failed when any file stayed), files handled, and failed, each file left with its error. A table that keeps its files has no such field. A delete_insert table with a replace_key also has replaced_keys: its columns, the values its commits replaced (sorted, each as its columns’ text joined by , , NULL for a NULL, at most 1,000) and truncated, true when there were more. Under --memory-per-table each table has peak_memory_bytes, the most memory the process held while it loaded. A table with a transform also has transforms: for each file or range, its file, the milliseconds the function took, the rows it was given and dicts, whether it returned dicts. A table with a column rename named or numbering numbered also has names: a line for each, as plan shows them, such as "Qty (column 4) -> qty_1 (numbered)". A table whose load warned, such as of rows outside a cursor window, has warnings, one line each. apply loads nothing, so its list is empty.

When the run can’t start (the configuration doesn’t parse, a connection fails), standard output holds one error document instead, the same as plan --json’s: {"error": {"code": "config", "message": "…", "help": "…", "retryable": false}}. code is config for the configuration, else the command’s name. retryable is true, and the exit code 75, when the run couldn’t start for a reason a rerun may fix, such as a target that refused the connection or timed out.

--reload and --resnapshot need --yes with --json, as there is no prompt to answer.

Under --json, standard output holds only the document. Everything a project with Python sources writes to standard output while it runs goes to standard error instead: a source’s print(), a child process that inherits standard output, and os.write(1, …).

--events jsonl writes each load event to standard error as one line of JSON, next to the logs, while it happens. --events-file PATH writes them to a file instead. Neither combines with --follow. Prefect and Dagster shows how to send them to an orchestrator’s log.

$ sqlload load --events jsonl 2>&1 >/dev/null
{"schema_version":0,"run_id":"6c3e27d9-…","at":"2026-11-15T02:24:32.303Z","event":"run_started","selected":2,"to_load":2}
{"schema_version":0,"run_id":"6c3e27d9-…","at":"2026-11-15T02:24:32.307Z","table":"raw.items","event":"table_started","strategy":"replace"}
…
{"schema_version":0,"run_id":"6c3e27d9-…","at":"2026-11-15T02:24:32.326Z","table":"raw.items","event":"table_failed","reason":"checks.min_rows failed: 1 row staged, fewer than the 1,000 required; nothing was published","retryable":false,"elapsed_ms":18}
{"schema_version":0,"run_id":"6c3e27d9-…","at":"2026-11-15T02:24:32.326Z","table":"raw.orders","event":"table_done","rows":3,"elapsed_ms":18}
{"schema_version":0,"run_id":"6c3e27d9-…","at":"2026-11-15T02:24:32.326Z","event":"run_finished","loaded":1,"failed":1,"not_started":0,"rows":3,"elapsed_ms":71,"exit_code":1,"peak_memory_bytes":86921216}

The events are version 0: names and fields can still change in a minor release until schema_version is 1. Every event has schema_version, run_id, at (RFC 3339, UTC) and event, and a table’s events have table, named as plan names it.

event Fields
connected Before run_started: a target’s or source’s connection opened, or failed. role (target or source), name, label (what it reached, such as OPENEDGE through ODBC), ok and elapsed_ms. No table.
discovered Before run_started: a source’s tables were found, or that failed. source, tables (how many), ok and elapsed_ms, timed from its connection when it has one. No table.
run_started selected, to_load: the tables selected and the tables the load starts.
table_started strategy: the write strategy, as plan shows it.
table_progress unit (page, batch, window or file), count of them read so far, rows so far.
rate_limited connection, its rate_limit as rate, and wait_ms, how long a request waited. No table.
table_committing rows: the table’s rows are read and its last commit starts.
table_done rows, elapsed_ms, and position, the cursor value it reached, when it keeps one. changes lists the created and altered lines of a table that made or widened its columns while loading. transforms lists each file or range a transform ran on, as in the JSON report, and under --memory-per-table peak_memory_bytes gives the table’s peak memory. names gives a line per column rename named or numbering numbered, as in the JSON report, and warnings what the load warns of, such as rows outside a cursor window.
table_failed reason, retryable (whether a rerun may pass), elapsed_ms, and peak_memory_bytes under --memory-per-table.
table_not_started reason, such as the table that stopped a --fail-fast load.
run_finished loaded, failed, not_started, rows, elapsed_ms and exit_code; peak_memory_bytes, the run’s peak memory.

On a terminal, load also draws a live block of running tables on standard error. The default, --progress auto, leaves it out under --json or --log-format json; --progress always draws it on a terminal anyway, and --progress never leaves it out. Output that isn’t a terminal never has the block or colours.