Skip to content

Timeouts and reconnects

What clients see during maintenance, size changes and upgrades, and the client settings that keep apps connected.

Updated View as Markdown

During maintenance, a size change or an upgrade, a replicated cluster takes one node out of service at a time. Before a node goes, its work moves to the other nodes: Kafka moves partition leadership, and RabbitMQ moves quorum queue leaders and takes the node out of the cluster’s name. Connections to that node then close. Clients reconnect, and carry on. A Kafka broker’s own name (b0-kafka-…) points at it again when it’s back.

Kafka

Kafka clients reconnect and find the new leaders by themselves. Keep their defaults, and:

  • Producers: use acks=all (the default in current clients) and leave retries on, so a write in flight while leadership moves is retried, not lost.
  • Consumers: commit offsets after processing. A consumer group rebalances when a member’s connection drops and comes back.

RabbitMQ

An AMQP connection ends when its node leaves; the client has to open a new one to the cluster’s name.

  • Turn on automatic recovery where the client has it: the Java and .NET clients (AutomaticRecoveryEnabled, on by default), MassTransit and Celery reconnect by themselves. With pika or amqplib, reconnect in your code when the connection closes.
  • Use publisher confirms and resend what wasn’t confirmed when the connection closed.
  • Keep heartbeats on (RabbitMQ’s default is 60 seconds), so a client notices a dead connection.

Connect timeouts

Set a connect timeout of a few seconds, so a client that reaches a node as it leaves gives up quickly and tries again.

DNS caching

Names have a 5-second TTL. Set clients to cache them for no longer:

  • Java (the Kafka Java client, Spring and others): a JVM caches names for 30 seconds by default. Set networkaddress.cache.ttl=5, in the JDK’s conf/security/java.security or with java.security.Security.setProperty("networkaddress.cache.ttl", "5") before the first connection.
  • Your own caches: a resolver cache on the instance, or an address in place of a name, has the same effect. Use the names, and honor their TTL.
Navigation

Type to search…

↑↓ navigate↵ selectEsc close