Architecture: consistency and evolution · 3 MIN READ

Kafka-style consumers: commit only a contiguous completed prefix

Separate message delivery, durable effects and partition-offset ownership.

A broker can redeliver messages after a consumer crash or partition reassignment. Processing records concurrently makes offset handling more subtle: finishing a later record does not prove that earlier records are durable. Track completion per partition and advance only across the contiguous successful prefix. The business effect needs its own deduplication or transaction protocol.

system-design · A MAP OF THE IDEA

Partition owns order
Bound concurrent work
Persist idempotent effects
Commit contiguous prefix

Useful when

Separate message delivery, durable effects and partition-offset ownership.
Makes the owner and the failure boundary explicit.

Watch out

Committing offset 14 while record 12 is unfinished can lose that work after a crash.
Run native service and failure tests in your own development environment.

Implementation & operation cost: Additional state and tests must be justified by the demonstrated failure; this lesson is not a requirement for every Go program.

The contract and the failure

Work through the example

Define the key used for partitioning, the scope of ordering and what counts as durable processing. Commit after the required effect, and retain a stable event identity for idempotent storage changes. Bound queued and in-flight records, observe lag and define a poison-record policy. Libraries such as franz-go provide Kafka protocol clients, not a transaction spanning an arbitrary database and external API.

package main

import "fmt"

func nextOffset(start int64, done map[int64]bool) int64 {
	for done[start] {
		start++
	}
	return start
}
func main() {
	done := map[int64]bool{11: true, 13: true}
	fmt.Println(nextOffset(11, done)) // 12: cannot skip unfinished 12
	done[12] = true
	fmt.Println(nextOffset(11, done)) // 14: resume after the completed prefix
}

Read the code

Offsets 11 and 13 are complete, but the next safe offset remains 12. Once 12 completes, the contiguous prefix ends before 14, so 14 is the next record to resume. This pure function models commit arithmetic; actual Go clients have specific commit, rebalance and transaction APIs that must be checked for the pinned version.

Production decisions

Committing offset 14 while record 12 is unfinished can lose that work after a crash. Committing too late can cause duplicates, which is safer only when the effect is replay-safe. During rebalance, an old worker can finish after ownership moved; fence or reconcile its business effects and follow the client’s partition-revocation contract. Test out-of-order completion, duplicates, crash, rebalance and broker timeout.

Explain it in an interview

Records 11 and 13 finished but 12 did not. May the consumer commit a resume offset of 14? No. Commit only the next offset after the contiguous durable prefix; otherwise a restart can skip the unfinished record. Describe the state before the operation, its owner, the failure between steps and the evidence you would test. Distinguish what this example demonstrates from the infrastructure a production service still needs.

TAKE THIS WITH YOU

Separate message delivery, durable effects and partition-offset ownership. No. Commit only the next offset after the contiguous durable prefix; otherwise a restart can skip the unfinished record.

Check your understanding

Records 11 and 13 finished but 12 did not. May the consumer commit a resume offset of 14?

Go to the source

Progress stays in this browser. No account needed.