FIFO queues

A FIFO queue delivers the messages of each message group in the order they were sent, one batch of a group at a time, and doesn’t store a send it has already seen. Groups are independent: a slow or failing group doesn’t hold up the others. Create one with "type": "fifo"; the type is fixed.

curl -sS -X POST "$BASE/queues" \
  -H "Authorization: Bearer $EZGH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "name": "ledger-events", "type": "fifo", "contentBasedDedup": true }'

Sending

curl -sS -X POST "$BASE/queues/$QUEUE_ID/messages" \
  -H "Authorization: Bearer $EZGH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "messages": [
        { "body": "opened", "groupId": "acct-9", "dedupId": "acct-9-open" },
        { "body": "deposited 100", "groupId": "acct-9", "dedupId": "acct-9-dep-1" }
      ] }'

{
  "messages": [
    { "messageId": "01999f50-1a2b-7c3d-8e4f-5a6b7c8d9e0f", "seq": "18446" },
    { "messageId": "01999f50-1a2b-7c3d-8e4f-5a6b7c8d9e10", "seq": "18447" }
  ]
}

• groupId is required on every message: 1 to 128 letters, digits and _ . : -. Use what must stay in order, such as an account or order ID.
• dedupId (same characters) is required unless the queue has contentBasedDedup, which uses the SHA-256 of the body instead. Attributes don’t count toward it.
• seq is the message’s sequence number, a string of digits. Within a group it increases in the order messages were stored.
• delaySeconds on a message is refused (400). The queue’s delaySeconds applies to every message.
• The messages of one request are stored in order, and all of them or none.

With the CLI: ezgh queues create ledger --fifo [--content-dedup], then ezgh queues send ledger --body "opened" --group acct-9 --dedup acct-9-open.

Order is the order Queues stores the sends. To keep a group’s order, send its messages from one producer, one request after another, or several in one request.

Deduplication

A send whose dedupId (or body, with contentBasedDedup) was sent to the queue in the last 5 minutes is accepted but not stored again: its result is the original’s messageId and seq, with "duplicate": true. This holds within one request too. So retrying a send after a timeout or a network error stores it once, as long as the retry comes within 5 minutes.

Receiving

A receive takes groups that are ready, meaning none of their messages are in flight and their first message is visible, and returns consecutive messages from each, in order, up to maxMessages. One receive can return several groups, and several messages of a group.

• While any message of a group is in flight, no other receive gets that group. The group is free again when its in-flight messages are acknowledged, or when their visibility timeout ends.
• A group whose visibility timeout ends comes back with its unacknowledged messages first, in order.
• Releasing one message ("visibilityTimeoutSeconds": 0) while others of its group are in flight keeps the group held until those are acknowledged or released too; then the released message comes first.
• A message that reaches maxReceiveCount moves to the dead-letter queue, which must be a FIFO queue too, and its group continues with the next message.

Handle each group’s messages in the order they arrive, and acknowledge them before receiving the group again. Several consumers can share a FIFO queue: each group goes to one at a time.

Push subscriptions

A delivery to a push subscription can hold several groups, each in order, with groupId and seq on every message. A failed message also fails the later messages of its group in that batch, so they’re retried after it. Stop handling a group once one of its messages fails, and list the rest as failed:

const failed = []
const failedGroups = new Set()
for (const message of delivery.messages) {
  if (failedGroups.has(message.groupId)) {
    failed.push(message.messageId) // keep the group's order: retry after the failed one
    continue
  }
  try {
    await handle(message)
  } catch {
    failed.push(message.messageId)
    failedGroups.add(message.groupId)
  }
}

Throughput

Each group’s messages are handled one batch at a time, so a group’s throughput is that of its consumer. Spread work across many groups to handle it in parallel. A FIFO queue takes 300 send and 300 receive requests a second by default (Limits), and Queues spreads busy queues over more partitions without splitting any group.