Skip to content

Load data

Connections give a warehouse access to your buckets, Hosted Queues Kafka clusters and databases, so one SQL statement loads from them with no keys to paste.

Updated View as Markdown

A connection links a warehouse to something of yours: an Object Storage bucket, a Hosted Queues Kafka cluster or a Databases database. EZGH creates the credential it needs and gives it to ClickHouse as a named collection called by the connection’s name, so your SQL names the connection and never holds a key. Users can use a connection, but can’t read its credential.

Creating a connection needs warehouse.connections.create and the target’s own permission to create the credential: on a bucket, to change its settings; on a Kafka cluster or a database, to create a user (and secrets.secrets.create for that user’s Secret). Connection names match ^[a-z][a-z0-9_]{0,31}$.

Only users with the admin preset can use connections. Deleting a connection deletes the credential it created, so it needs warehouse.connections.delete and the permission to delete that credential: objects.buckets.update on a bucket, hostedqueues.users.delete or databases.users.delete on a Kafka cluster or a database, and secrets.secrets.delete for its Secret.

If a Kafka cluster or database is deleted, its connection is broken until you delete it.

From a bucket

ezgh warehouse connections create whs_4k2m9x0a7qpe lake --type bucket --bucket bkt_7hq2m0x9ka3c \
  --access read --wait

--access readWrite also lets the warehouse write to the bucket. The bucket’s settings list the warehouse and its access. A warehouse has one connection to each bucket: a second one to the same bucket is refused. The warehouse reaches the bucket without a Network Gateway.

On a warehouse with more than one shard, load into and export from a Distributed table, such as page_views_all in Sharding, so rows spread across the shards.

-- Read Parquet files
SELECT count() FROM s3(lake, filename = 'events/2026/*.parquet');

-- Load them into a table
INSERT INTO events.page_views_all SELECT ts, user_id, path FROM s3(lake, filename = 'events/2026/*.parquet');

-- Read an Iceberg table
SELECT * FROM icebergS3(lake, filename = 'warehouse/orders') LIMIT 10;

-- Write (readWrite connections)
INSERT INTO FUNCTION s3(lake, filename = 'exports/page_views.parquet') SELECT * FROM events.page_views_all;

From a Kafka topic

ezgh warehouse connections create whs_4k2m9x0a7qpe orders_kafka --type kafka --cluster hqc_4k2m9x0a7qpe \
  --topic-prefix orders --wait

EZGH creates a Kafka user on the cluster that can consume, with its password in a Secret. With a prefix, it consumes only topics starting with the prefix, in consumer groups whose names start with its username (the Secret’s username), so set kafka_group_name to such a name. Read a topic with the Kafka table engine and a materialized view into a table:

CREATE TABLE events.orders_queue
(
    order_id UInt64,
    amount Decimal(18, 2),
    ts DateTime
)
ENGINE = Kafka(orders_kafka)
SETTINGS kafka_topic_list = 'orders', kafka_group_name = 'wh_4k2m9x0a7qpe_3f9a1c2e_orders',
         kafka_format = 'JSONEachRow';

CREATE TABLE events.orders (order_id UInt64, amount Decimal(18, 2), ts DateTime)
ENGINE = ReplicatedMergeTree() ORDER BY (ts, order_id);

CREATE MATERIALIZED VIEW events.orders_mv TO events.orders AS SELECT * FROM events.orders_queue;

The cluster has to be in the warehouse’s VPC, with the warehouse’s subnet in its allowed sources. The warehouse connects over TLS and checks the cluster’s certificate against the EZGH CA. The connection sets the brokers, credentials and TLS, which a table can’t change; the table’s SETTINGS choose the topics, consumer group and format.

From a database

ezgh warehouse connections create whs_4k2m9x0a7qpe shop --type database \
  --database-instance dbi_4k2m9x0a7qpe --database shop --wait

EZGH creates a read-only user on the database, with its password in a Secret. Use postgresql() for a PostgreSQL database and mysql() for Percona Server or MariaDB:

-- Read
SELECT count() FROM postgresql(shop, table = 'orders');

-- Copy into the warehouse
INSERT INTO events.orders SELECT id, total, created_at FROM postgresql(shop, table = 'orders');

The database has to be in the warehouse’s VPC, and allow connections from the warehouse’s subnet.

Kafka clusters and databases in the VPC

A Hosted Queues Kafka cluster or a Databases database in the warehouse’s VPC is reached at its private addresses. The VPC needs no NAT Gateway, Network Gateway or DNS resolver for these connections, and the traffic stays inside the VPC.

List and delete

ezgh warehouse connections list whs_4k2m9x0a7qpe
ezgh warehouse connections delete whs_4k2m9x0a7qpe shop --wait

The console’s Connections tab shows each connection’s target and the SQL to use it.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close