Connectors
Connector matrix
What SQLLoad reads and writes, by which route, and how each is tested.
Sources
Section titled “Sources”| Source | Formats | Read by |
|---|---|---|
| Postgres | Tables, by include patterns |
Binary COPY into Arrow, over TLS when the connection’s sslmode asks |
Postgres changes (postgres_cdc) |
Inserts, updates and deletes of tables, by include patterns |
A snapshot, then a logical replication slot (pgoutput), in whole transactions. See Postgres changes. |
| Snowflake | Tables and views, by include patterns, and query results |
Parquet unloaded with COPY INTO to the user stage or an S3 stage, then read by SQLLoad, or by DuckDB, Databricks or ClickHouse themselves from an S3 stage. A Snowflake target selects or copies the rows inside Snowflake when it writes through the source’s connection, or its external stage covers the unload; see Snowflake loads the rows itself. See Snowflake. |
| Databases with an ODBC driver: SQL Server, MySQL, MariaDB, OpenEdge and others | Tables and views, by include patterns, and query results |
The driver, as Arrow. Tested with psqlODBC, Microsoft ODBC Driver 18 for SQL Server and MySQL Connector/ODBC; MariaDB Connector/ODBC and OpenEdge are checked by hand. See ODBC. |
| BigQuery (beta) | Tables and views, by include patterns, and query results |
The Storage Read API, as Arrow in parallel streams; views and queries through a query job. A BigQuery target on the source’s connection selects the rows itself; see BigQuery loads the rows itself. See BigQuery. |
| Databricks (beta) | Tables and views, by include patterns, and query results |
One SELECT on the SQL warehouse, its result downloaded as Arrow files in parallel through the Statement Execution API’s links. See Databricks. |
| DuckDB | Tables and views of a database file, by include patterns, and query results |
One SELECT on a read-only connection, streamed as Arrow. See DuckDB. |
| MotherDuck (beta) | Tables and views of one database, by include patterns, and query results |
One SELECT run by a separate DuckDB 1.5 process (sqlload[motherduck]), its Arrow batches handed over one at a time. See MotherDuck. |
| Files in S3 | Parquet, CSV, JSON, NDJSON, Excel (xlsx) and XML; gzip or zstd for text formats; ZIP archives of any of them; any file name through pattern. See Files. |
The destination itself where it can load them exactly, otherwise SQLLoad’s reader |
| Files in Google Cloud Storage | As for S3 | As for S3; DuckDB and MotherDuck read Parquet themselves with an HMAC key, and BigQuery with its own service account. See gcs. |
| Files in a local folder | As for S3 | As for S3 |
| Files on an SFTP server | As for S3 | SQLLoad’s reader, over SFTP. See Files. |
| Google Sheets | Tabs or A1 ranges, values typed by their cells | The Sheets API with a service-account key, into Arrow. See Google Sheets. |
| Python | Dicts, Arrow data (pyarrow, polars, pandas), JSON and NDJSON bytes | The sqlload package, in process |
| HTTP APIs | JSON, through sl.http in a Python source |
Rust |
| Kafka | Topics, through the separate exporter, into Snowflake, or (beta) Databricks or BigQuery | Rust |
Destinations
Section titled “Destinations”| Destination | Write strategies | How data arrives |
|---|---|---|
| Snowflake | append, merge, delete_insert, replace |
Parquet staged on an external, user, table or named stage, then COPY; or Snowpipe Streaming, or INSERT, by load_method. Files under an external stage load with COPY directly. |
| BigQuery (beta) | append, merge, delete_insert, replace |
Arrow streamed through the Storage Write API into a staging table, then one query job per write; Parquet in GCS, on a table that excludes _sqlload_source_row, queried by BigQuery where it lies. See BigQuery. |
| Databricks (beta) | append, merge, delete_insert, replace |
Parquet in S3 copied where it lies through a Unity Catalog external location; everything else staged as Parquet in a volume, then COPY INTO; one statement per write. |
| ClickHouse, one self-hosted server | append, merge, delete_insert, replace |
Arrow over HTTP into a staging table, published by swapping in a whole table or whole partitions; Parquet in S3 read by ClickHouse with s3(). No clusters, replicated or distributed tables; ClickHouse Cloud is untested. See ClickHouse. |
| Postgres | append, merge, delete_insert, replace |
Binary COPY into a temporary staging table, then one transaction per commit. See Postgres. |
| DuckDB | append, merge, delete_insert, replace |
Arrow appends; Parquet files read by DuckDB with read_parquet. |
| MotherDuck (beta) | append, merge, delete_insert, replace |
Batches inserted by a separate DuckDB 1.5 process (sqlload[motherduck]), as MotherDuck doesn’t accept DuckDB 2.0 clients yet; Parquet in S3, and in GCS with an HMAC key, is read on MotherDuck’s servers, other files by SQLLoad. See MotherDuck. |
| Parquet or CSV files in S3 | append, delete_insert, replace |
Parquet written by SQLLoad, zstd-compressed; or CSV: plain, gzip or zstd. |
| Parquet or CSV files in Google Cloud Storage | As for S3 | As for S3 |
| Parquet or CSV files in a local folder | As for S3; replace swaps the folder in one rename |
As for S3 |
Source and destination combinations
Section titled “Source and destination combinations”| From \ Into | Snowflake | Databricks (beta) | BigQuery (beta) | ClickHouse | Postgres | DuckDB | Parquet files |
|---|---|---|---|---|---|---|---|
| Postgres | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
Postgres changes (postgres_cdc) |
Yes | Yes | No | No | No | Yes | Yes, as a changelog |
| Snowflake | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| ODBC | Yes, untested | Yes, tested live | Yes, untested | Yes, untested | Yes, untested | Yes | Yes |
| BigQuery (beta) | Yes, untested | Yes, tested live | Yes, tested live | Yes, untested | Yes, untested | Yes | Yes, untested |
| Databricks (beta) | Yes, untested | Yes, untested | Yes, untested | Yes, tested live | Yes, tested live | Yes, tested live | Yes, tested live |
| DuckDB | Yes, untested | Yes, untested | Yes, untested | Yes, untested | Yes, untested | Yes | Yes, untested |
| MotherDuck (beta) | Yes, untested | Yes, untested | Yes, untested | Yes, untested | Yes, untested | Yes, tested live | Yes, untested |
| Files | Yes, read by Snowflake from an external stage | Yes, staged by SQLLoad in the volume | Yes, read by SQLLoad and streamed | Yes; Parquet in S3 read by ClickHouse | Yes, read by SQLLoad | Yes; Parquet read by DuckDB | Read and rewritten by SQLLoad |
| Files on an SFTP server | Yes, read by SQLLoad, tested live | Yes, read by SQLLoad, tested live | Yes, untested | Yes, untested | Yes, untested | Yes | Yes, untested |
| Google Sheets | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| Python | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
A MotherDuck target (beta) takes every source a DuckDB target takes, untested, through SQLLoad, except a files source’s Parquet in S3, which MotherDuck’s servers read themselves.
A files source loads into a files target, read and rewritten
by SQLLoad. merge isn’t available for files sources or files
targets. Google Sheets tables reach every target through the same Arrow path; the tests load
them into DuckDB.
Testing
Section titled “Testing”| System | How it’s tested |
|---|---|
| Postgres | Real Postgres in containers, as a source and as a target. postgres_cdc on Postgres 15, 16 and 17, plus live loads into DuckDB, Snowflake and Databricks. See Postgres. |
| ClickHouse | Real ClickHouse 25.3, 25.8 and 26.10 in containers, with an S3-compatible store for the files it reads itself. See ClickHouse. |
| DuckDB | Real, in process, as a source and a target. |
| MotherDuck | Live against a MotherDuck account on DuckDB 1.5.6: every write strategy, an incremental load and rerun, a new column and every portable type. Offline, every change loads through the separate process into a stand-in for MotherDuck and compares with DuckDB. As a source, live and offline: tables, a query table, where, exclude_columns and a cursor merge into DuckDB. See MotherDuck. |
| ODBC | psqlODBC against real Postgres in containers, into DuckDB, plus a live round from SQL Server and MySQL into Databricks. |
| Databricks source | A fake Statement Execution API in tests (catalog discovery, result files in several chunks, an expired link fetched again, cursor reads, no token on downloads), plus a live round from a real SQL warehouse into DuckDB, Parquet files, Postgres and ClickHouse: tables, a view, a windowed query table, merges and every column shape. See Databricks. |
| BigQuery source | A fake Storage Read API in tests (parallel streams, a broken stream resumed, cursor filters), and live loads from a sandbox project into DuckDB: tables, views, query tables, and integer, timestamp, text, NUMERIC and BIGNUMERIC cursors, plus a live round into Databricks, and live loads into BigQuery on the same connection (append by cursor, merge, a view replaced) whose tables match SQLLoad’s reader row for row. |
| S3 | A real S3-compatible store in containers; benchmarks against AWS S3. |
| Local files | Real files in temporary folders. |
| SFTP | OpenSSH’s SFTP server in a container, into DuckDB: new and rewritten files, gzip CSV and Parquet, key and password sign-in, and unknown or changed host keys refused, plus a live round into Databricks. |
| Kafka | Redpanda in containers. The exporter into Databricks was tested live on all three routes (Zerobus Ingest, a volume and S3), with a restart mid-run. |
| Google Sheets | A local fake Sheets API (OAuth, tabs and A1 ranges, refused tokens), plus a live smoke test on a real spreadsheet with generic contents. |
| Snowflake | A local fake for protocol and failure tests, plus live tests on a real Snowflake account. |
| BigQuery | A local fake (OAuth, query jobs, table metadata, the Storage Write API) for protocol and failure tests, plus live tests on a real BigQuery project. See BigQuery. |
| Databricks | A local fake workspace (OAuth, statements, the volume) for protocol and failure tests, plus live tests of every pathway on a real SQL warehouse. See Databricks. |
Platforms
Section titled “Platforms”The sqlload and sqlload-cli packages are released for Linux (x86_64 and aarch64, glibc 2.28 or
newer), macOS 11 or newer (arm64 and Intel x86_64) and Windows 10 or Windows Server 2019 or newer
(x86_64). The sqlload package needs CPython 3.10 or newer. See
Install.