Celery apps, including Airflow’s Celery executor and Superset’s workers, use a RabbitMQ cluster as their broker over AMQPS.
Queue types
New queues on a RabbitMQ cluster are quorum queues unless the app declares another type: each vhost’s
defaultQueueType is quorum. Celery 5.5 and later work with quorum queues when they’re told to use
them:
task_default_queue_type = "quorum"
task_default_exchange_type = "topic"
broker_transport_options = {"confirm_publish": True}worker_detect_quorum_queues is on by default, so workers use the quorum-queue settings they need.
With task_default_exchange_type = "topic", Celery delivers countdown and ETA tasks through its
celery_delayed_* quorum queues; with a direct default exchange it keeps those tasks in the worker
until they’re due. These settings are tested with Celery 5.5.3 and Kombu 5.5.4.
An app that can’t use quorum queues can run on a vhost whose defaultQueueType is classic, where
Celery needs none of these settings. See Vhosts and queues.
Broker URL and TLS
import ssl
broker_url = uri # the user's Secret's uri field: amqps://app:<password>@mq-….us-west-1.ezghcompute.com:5671/%2F
broker_use_ssl = {
"ca_certs": "ca-bundle.pem",
"cert_reqs": ssl.CERT_REQUIRED,
}Get ca-bundle.pem from Connect your apps. Read uri from the user’s
Secret when the app starts, rather than writing the
password into the file.
Airflow and Superset
Airflow takes the same settings under [celery] (broker_url, and ssl_active, ssl_cacert for TLS),
or as AIRFLOW__CELERY__BROKER_URL and the like. Superset reads them from its CeleryConfig class in
superset_config.py. Add the three queue settings above in the same place: Superset as attributes of CeleryConfig,
Airflow through [celery] celery_config_options, pointing at a dict that adds them to Airflow’s
default Celery config.
Worker queues
Celery workers declare short-lived reply and broadcast queues (exclusive or auto-delete) as they start and stop. They’re listed on the cluster’s Queues tab only when you show them.