Skip to content

Connect Celery

Run Celery 5.5 and later, and apps built on it such as Airflow and Superset, on a RabbitMQ cluster with quorum queues.

Updated View as Markdown

Celery apps, including Airflow’s Celery executor and Superset’s workers, use a RabbitMQ cluster as their broker over AMQPS.

Queue types

New queues on a RabbitMQ cluster are quorum queues unless the app declares another type: each vhost’s defaultQueueType is quorum. Celery 5.5 and later work with quorum queues when they’re told to use them:

task_default_queue_type = "quorum"
task_default_exchange_type = "topic"
broker_transport_options = {"confirm_publish": True}

worker_detect_quorum_queues is on by default, so workers use the quorum-queue settings they need. With task_default_exchange_type = "topic", Celery delivers countdown and ETA tasks through its celery_delayed_* quorum queues; with a direct default exchange it keeps those tasks in the worker until they’re due. These settings are tested with Celery 5.5.3 and Kombu 5.5.4.

An app that can’t use quorum queues can run on a vhost whose defaultQueueType is classic, where Celery needs none of these settings. See Vhosts and queues.

Broker URL and TLS

import ssl

broker_url = uri  # the user's Secret's uri field: amqps://app:<password>@mq-….us-west-1.ezghcompute.com:5671/%2F
broker_use_ssl = {
    "ca_certs": "ca-bundle.pem",
    "cert_reqs": ssl.CERT_REQUIRED,
}

Get ca-bundle.pem from Connect your apps. Read uri from the user’s Secret when the app starts, rather than writing the password into the file.

Airflow and Superset

Airflow takes the same settings under [celery] (broker_url, and ssl_active, ssl_cacert for TLS), or as AIRFLOW__CELERY__BROKER_URL and the like. Superset reads them from its CeleryConfig class in superset_config.py. Add the three queue settings above in the same place: Superset as attributes of CeleryConfig, Airflow through [celery] celery_config_options, pointing at a dict that adds them to Airflow’s default Celery config.

Worker queues

Celery workers declare short-lived reply and broadcast queues (exclusive or auto-delete) as they start and stop. They’re listed on the cluster’s Queues tab only when you show them.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close