Skip to content

Sources

Sources

Postgres tables, Postgres changes, Snowflake tables and queries, ODBC databases, BigQuery, Databricks and MotherDuck tables and queries (beta), DuckDB files, files in S3 or a folder, Google Sheets, and Python sources: the kinds of source and the settings they share.

A source is where data comes from. SQLLoad has eleven kinds:

kind Reads Tables Declared in
postgres Tables of a Postgres database, with binary COPY One per matching source table sqlload.toml
postgres_cdc A Postgres database’s inserts, updates and deletes, from a logical replication slot, after a first snapshot One per matching source table sqlload.toml
snowflake Tables and views of a Snowflake database, and query results, unloaded as Parquet One per matching table, and one per query sqlload.toml
odbc Tables, views and query results of any database with an ODBC driver, read as Arrow One per matching table, and one per query sqlload.toml
bigquery (beta) Tables, views and query results of BigQuery, read as Arrow through the Storage Read API One per matching table, and one per query sqlload.toml
databricks (beta) Tables, views and query results of a Databricks SQL warehouse, read as Arrow through the Statement Execution API One per matching table, and one per query sqlload.toml
duckdb Tables, views and query results of a DuckDB database file, read as Arrow One per matching table, and one per query sqlload.toml
motherduck (beta) Tables, views and query results of a MotherDuck database, read as Arrow through a separate DuckDB 1.5 process One per matching table, and one per query sqlload.toml
files Parquet, CSV, JSON or NDJSON files in S3 or a local folder One per source sqlload.toml
google_sheets Tabs or ranges of a Google Sheets spreadsheet, typed One per tab or range sqlload.toml
Python Whatever a Python function yields: dicts, Arrow data, JSON bytes, HTTP jobs One per decorated function Python, under python_paths

Every source loads into a target: its destination, or the project’s default_target. Settings that shape tables (write_strategy, unique_key, cursor_column and the rest) can be set on the source as defaults for all its tables, and per table. See sqlload.toml for the full list.

Each kind has its own page:

  • Postgres: choosing tables and columns, masking, and how tables are read.
  • Postgres changes: streaming changes from a replication slot into DuckDB, Snowflake, Databricks or Parquet files, and what to watch on the server. To set one up, on a schedule or continuously, see Capture Postgres changes.
  • Snowflake: choosing tables, query tables, read windows around the time of the run, and where unloaded files go.
  • ODBC: connections, drivers, choosing tables, query tables and value sizes.
  • BigQuery (beta): choosing tables, query tables, types, cost and permissions.
  • Databricks (beta): choosing tables, query tables, types, cost and permissions.
  • DuckDB: choosing tables, query tables, types, and reading a file another process writes.
  • MotherDuck (beta): installing the separate DuckDB 1.5 process, and what reads as for a DuckDB source.
  • Files: formats, partitions, which files load, and files the destination reads itself.
  • Google Sheets: tabs and ranges, headers, and how cells are typed.
  • Python: any API, SaaS or custom source as a table, including REST APIs that run entirely in Rust. Every function is in the Python API.