During maintenance, a size change or an
upgrade, a replicated cluster takes one node out of service at a time.
Before a node goes, its work moves to the other nodes: Kafka moves partition leadership, and RabbitMQ
moves quorum queue leaders and takes the node out of the cluster’s name. Connections to that node then
close. Clients reconnect, and carry on. A Kafka broker’s own name (b0-kafka-…) points at it again when
it’s back.
Kafka
Kafka clients reconnect and find the new leaders by themselves. Keep their defaults, and:
- Producers: use
acks=all(the default in current clients) and leave retries on, so a write in flight while leadership moves is retried, not lost. - Consumers: commit offsets after processing. A consumer group rebalances when a member’s connection drops and comes back.
RabbitMQ
An AMQP connection ends when its node leaves; the client has to open a new one to the cluster’s name.
- Turn on automatic recovery where the client has it: the Java and .NET clients
(
AutomaticRecoveryEnabled, on by default), MassTransit and Celery reconnect by themselves. Withpikaoramqplib, reconnect in your code when the connection closes. - Use publisher confirms and resend what wasn’t confirmed when the connection closed.
- Keep heartbeats on (RabbitMQ’s default is 60 seconds), so a client notices a dead connection.
Connect timeouts
Set a connect timeout of a few seconds, so a client that reaches a node as it leaves gives up quickly and tries again.
DNS caching
Names have a 5-second TTL. Set clients to cache them for no longer:
- Java (the Kafka Java client, Spring and others): a JVM caches names for 30 seconds by default.
Set
networkaddress.cache.ttl=5, in the JDK’sconf/security/java.securityor withjava.security.Security.setProperty("networkaddress.cache.ttl", "5")before the first connection. - Your own caches: a resolver cache on the instance, or an address in place of a name, has the same effect. Use the names, and honor their TTL.