Senior · Docker and Kubernetes
Out-of-order work and a dangerous offset commit
A partition consumer finishes offsets 11 and 13 while 12 is still running. It commits resume offset 14. What can happen after restart?
Try explaining this aloud before opening the answer. State your assumptions and trace a boundary case.
package main
import "fmt"
func nextOffset(start int64, done map[int64]bool) int64 {
for done[start] {
start++
}
return start
}
func main() {
done := map[int64]bool{11: true, 13: true}
fmt.Println(nextOffset(11, done)) // 12: cannot skip unfinished 12
done[12] = true
fmt.Println(nextOffset(11, done)) // 14: resume after the completed prefix
}Reveal a strong answer
A broker can redeliver messages after a consumer crash or partition reassignment. Processing records concurrently makes offset handling more subtle: finishing a later record does not prove that earlier records are durable. Track completion per partition and advance only across the contiguous successful prefix. The business effect needs its own deduplication or transaction protocol.
Define the key used for partitioning, the scope of ordering and what counts as durable processing. Commit after the required effect, and retain a stable event identity for idempotent storage changes. Bound queued and in-flight records, observe lag and define a poison-record policy. Libraries such as franz-go provide Kafka protocol clients, not a transaction spanning an arbitrary database and external API. Offsets 11 and 13 are complete, but the next safe offset remains 12. Once 12 completes, the contiguous prefix ends before 14, so 14 is the next record to resume. This pure function models commit arithmetic; actual Go clients have specific commit, rebalance and transaction APIs that must be checked for the pinned version.
Committing offset 14 while record 12 is unfinished can lose that work after a crash. Committing too late can cause duplicates, which is safer only when the effect is replay-safe. During rebalance, an old worker can finish after ownership moved; fence or reconcile its business effects and follow the client’s partition-revocation contract. Test out-of-order completion, duplicates, crash, rebalance and broker timeout.
Common trap
The largest completed offset is not the same as the largest safely committed contiguous prefix.
Try it
Implement contiguous-prefix accounting and explain durable effects, deduplication and rebalance ownership.
Worked solution
The restart can skip unfinished record 12. Advance only after the contiguous durable prefix is complete and use the client’s documented next-offset semantics. A database effect needs atomic deduplication or another replay-safe design; broker transaction features do not atomically cover arbitrary external services. Bound parallel work and queued records, handle poison messages deliberately and fence or reconcile work that outlives partition ownership during rebalance.
Change the constraint
How would a worker that completes after its partition was revoked be prevented from producing a duplicate effect?
How did you do?
Self-assessment, not a compiler pass or interview score. Explain the reasoning unaided before marking understood.
Not marked. Progress stays on this browser, separate from course completion.
Primary references
Checked 2026-10-28. Use documentation for your pinned runtime, driver and tool version.