Skip to content

Connectors

Connector matrix

What SQLLoad reads and writes, by which route, and how each is tested.

Source Formats Read by
Postgres Tables, by include patterns Binary COPY into Arrow, over TLS when the connection’s sslmode asks
Postgres changes (postgres_cdc) Inserts, updates and deletes of tables, by include patterns A snapshot, then a logical replication slot (pgoutput), in whole transactions. See Postgres changes.
Snowflake Tables and views, by include patterns, and query results Parquet unloaded with COPY INTO to the user stage or an S3 stage, then read by SQLLoad, or by DuckDB, Databricks or ClickHouse themselves from an S3 stage. A Snowflake target selects or copies the rows inside Snowflake when it writes through the source’s connection, or its external stage covers the unload; see Snowflake loads the rows itself. See Snowflake.
Databases with an ODBC driver: SQL Server, MySQL, MariaDB, OpenEdge and others Tables and views, by include patterns, and query results The driver, as Arrow. Tested with psqlODBC, Microsoft ODBC Driver 18 for SQL Server and MySQL Connector/ODBC; MariaDB Connector/ODBC and OpenEdge are checked by hand. See ODBC.
BigQuery (beta) Tables and views, by include patterns, and query results The Storage Read API, as Arrow in parallel streams; views and queries through a query job. A BigQuery target on the source’s connection selects the rows itself; see BigQuery loads the rows itself. See BigQuery.
Databricks (beta) Tables and views, by include patterns, and query results One SELECT on the SQL warehouse, its result downloaded as Arrow files in parallel through the Statement Execution API’s links. See Databricks.
DuckDB Tables and views of a database file, by include patterns, and query results One SELECT on a read-only connection, streamed as Arrow. See DuckDB.
MotherDuck (beta) Tables and views of one database, by include patterns, and query results One SELECT run by a separate DuckDB 1.5 process (sqlload[motherduck]), its Arrow batches handed over one at a time. See MotherDuck.
Files in S3 Parquet, CSV, JSON, NDJSON, Excel (xlsx) and XML; gzip or zstd for text formats; ZIP archives of any of them; any file name through pattern. See Files. The destination itself where it can load them exactly, otherwise SQLLoad’s reader
Files in Google Cloud Storage As for S3 As for S3; DuckDB and MotherDuck read Parquet themselves with an HMAC key, and BigQuery with its own service account. See gcs.
Files in a local folder As for S3 As for S3
Files on an SFTP server As for S3 SQLLoad’s reader, over SFTP. See Files.
Google Sheets Tabs or A1 ranges, values typed by their cells The Sheets API with a service-account key, into Arrow. See Google Sheets.
Python Dicts, Arrow data (pyarrow, polars, pandas), JSON and NDJSON bytes The sqlload package, in process
HTTP APIs JSON, through sl.http in a Python source Rust
Kafka Topics, through the separate exporter, into Snowflake, or (beta) Databricks or BigQuery Rust
Destination Write strategies How data arrives
Snowflake append, merge, delete_insert, replace Parquet staged on an external, user, table or named stage, then COPY; or Snowpipe Streaming, or INSERT, by load_method. Files under an external stage load with COPY directly.
BigQuery (beta) append, merge, delete_insert, replace Arrow streamed through the Storage Write API into a staging table, then one query job per write; Parquet in GCS, on a table that excludes _sqlload_source_row, queried by BigQuery where it lies. See BigQuery.
Databricks (beta) append, merge, delete_insert, replace Parquet in S3 copied where it lies through a Unity Catalog external location; everything else staged as Parquet in a volume, then COPY INTO; one statement per write.
ClickHouse, one self-hosted server append, merge, delete_insert, replace Arrow over HTTP into a staging table, published by swapping in a whole table or whole partitions; Parquet in S3 read by ClickHouse with s3(). No clusters, replicated or distributed tables; ClickHouse Cloud is untested. See ClickHouse.
Postgres append, merge, delete_insert, replace Binary COPY into a temporary staging table, then one transaction per commit. See Postgres.
DuckDB append, merge, delete_insert, replace Arrow appends; Parquet files read by DuckDB with read_parquet.
MotherDuck (beta) append, merge, delete_insert, replace Batches inserted by a separate DuckDB 1.5 process (sqlload[motherduck]), as MotherDuck doesn’t accept DuckDB 2.0 clients yet; Parquet in S3, and in GCS with an HMAC key, is read on MotherDuck’s servers, other files by SQLLoad. See MotherDuck.
Parquet or CSV files in S3 append, delete_insert, replace Parquet written by SQLLoad, zstd-compressed; or CSV: plain, gzip or zstd.
Parquet or CSV files in Google Cloud Storage As for S3 As for S3
Parquet or CSV files in a local folder As for S3; replace swaps the folder in one rename As for S3
From \ Into Snowflake Databricks (beta) BigQuery (beta) ClickHouse Postgres DuckDB Parquet files
Postgres Yes Yes Yes Yes Yes Yes Yes
Postgres changes (postgres_cdc) Yes Yes No No No Yes Yes, as a changelog
Snowflake Yes Yes Yes Yes Yes Yes Yes
ODBC Yes, untested Yes, tested live Yes, untested Yes, untested Yes, untested Yes Yes
BigQuery (beta) Yes, untested Yes, tested live Yes, tested live Yes, untested Yes, untested Yes Yes, untested
Databricks (beta) Yes, untested Yes, untested Yes, untested Yes, tested live Yes, tested live Yes, tested live Yes, tested live
DuckDB Yes, untested Yes, untested Yes, untested Yes, untested Yes, untested Yes Yes, untested
MotherDuck (beta) Yes, untested Yes, untested Yes, untested Yes, untested Yes, untested Yes, tested live Yes, untested
Files Yes, read by Snowflake from an external stage Yes, staged by SQLLoad in the volume Yes, read by SQLLoad and streamed Yes; Parquet in S3 read by ClickHouse Yes, read by SQLLoad Yes; Parquet read by DuckDB Read and rewritten by SQLLoad
Files on an SFTP server Yes, read by SQLLoad, tested live Yes, read by SQLLoad, tested live Yes, untested Yes, untested Yes, untested Yes Yes, untested
Google Sheets Yes Yes Yes Yes Yes Yes Yes
Python Yes Yes Yes Yes Yes Yes Yes

A MotherDuck target (beta) takes every source a DuckDB target takes, untested, through SQLLoad, except a files source’s Parquet in S3, which MotherDuck’s servers read themselves.

A files source loads into a files target, read and rewritten by SQLLoad. merge isn’t available for files sources or files targets. Google Sheets tables reach every target through the same Arrow path; the tests load them into DuckDB.

System How it’s tested
Postgres Real Postgres in containers, as a source and as a target. postgres_cdc on Postgres 15, 16 and 17, plus live loads into DuckDB, Snowflake and Databricks. See Postgres.
ClickHouse Real ClickHouse 25.3, 25.8 and 26.10 in containers, with an S3-compatible store for the files it reads itself. See ClickHouse.
DuckDB Real, in process, as a source and a target.
MotherDuck Live against a MotherDuck account on DuckDB 1.5.6: every write strategy, an incremental load and rerun, a new column and every portable type. Offline, every change loads through the separate process into a stand-in for MotherDuck and compares with DuckDB. As a source, live and offline: tables, a query table, where, exclude_columns and a cursor merge into DuckDB. See MotherDuck.
ODBC psqlODBC against real Postgres in containers, into DuckDB, plus a live round from SQL Server and MySQL into Databricks.
Databricks source A fake Statement Execution API in tests (catalog discovery, result files in several chunks, an expired link fetched again, cursor reads, no token on downloads), plus a live round from a real SQL warehouse into DuckDB, Parquet files, Postgres and ClickHouse: tables, a view, a windowed query table, merges and every column shape. See Databricks.
BigQuery source A fake Storage Read API in tests (parallel streams, a broken stream resumed, cursor filters), and live loads from a sandbox project into DuckDB: tables, views, query tables, and integer, timestamp, text, NUMERIC and BIGNUMERIC cursors, plus a live round into Databricks, and live loads into BigQuery on the same connection (append by cursor, merge, a view replaced) whose tables match SQLLoad’s reader row for row.
S3 A real S3-compatible store in containers; benchmarks against AWS S3.
Local files Real files in temporary folders.
SFTP OpenSSH’s SFTP server in a container, into DuckDB: new and rewritten files, gzip CSV and Parquet, key and password sign-in, and unknown or changed host keys refused, plus a live round into Databricks.
Kafka Redpanda in containers. The exporter into Databricks was tested live on all three routes (Zerobus Ingest, a volume and S3), with a restart mid-run.
Google Sheets A local fake Sheets API (OAuth, tabs and A1 ranges, refused tokens), plus a live smoke test on a real spreadsheet with generic contents.
Snowflake A local fake for protocol and failure tests, plus live tests on a real Snowflake account.
BigQuery A local fake (OAuth, query jobs, table metadata, the Storage Write API) for protocol and failure tests, plus live tests on a real BigQuery project. See BigQuery.
Databricks A local fake workspace (OAuth, statements, the volume) for protocol and failure tests, plus live tests of every pathway on a real SQL warehouse. See Databricks.

The sqlload and sqlload-cli packages are released for Linux (x86_64 and aarch64, glibc 2.28 or newer), macOS 11 or newer (arm64 and Intel x86_64) and Windows 10 or Windows Server 2019 or newer (x86_64). The sqlload package needs CPython 3.10 or newer. See Install.