Senior · Architecture and systems
The outbox relay crashes after broker acknowledgement
An order and its outbox row committed. The broker accepted the event, but the relay crashed before recording delivery. What must recovery do?
Try explaining this aloud before opening the answer. State your assumptions and trace a boundary case.
package main
import "fmt"
type Event struct{ ID, Order string }
type Inbox struct {
seen map[string]bool
total int
}
func (i *Inbox) Apply(e Event) bool {
if i.seen[e.ID] {
return false
}
// Teaching model: one owner. Persist both changes atomically in production.
i.total++
i.seen[e.ID] = true
return true
}
func main() {
i := Inbox{seen: map[string]bool{}}
e := Event{"evt-1", "order-7"}
fmt.Println(i.Apply(e), i.Apply(e), i.total) // true false 1
}Reveal a strong answer
Writing an order and publishing its event are two different side effects. A crash between them can leave a missing event; publishing first can announce a change that never commits. An outbox stores publication intent beside the business write in the same durable transaction. A separate relay retries it, and the consumer expects duplicates rather than claiming end-to-end exactly once.
Give each outbox record an immutable event ID, aggregate ID, payload/schema version and publication state. Commit it with the order, then have a bounded relay claim pending rows. The relay marks delivery only after the broker acknowledges. A consumer uses a unique event ID constraint and updates its own state in the same local transaction. The stable event ID, not a newly generated delivery ID, identifies the duplicate. The model applies the first event once and rejects its replay. A production inbox must persist the deduplication record and business change in one transaction; this map intentionally demonstrates neither crash safety nor distributed atomicity.
A relay can crash after broker acknowledgement but before marking the row sent, so duplicates are normal. Marking sent first risks permanent loss. State the ordering requirement per aggregate, retention period and poison-event policy. Test crashes on both sides of acknowledgement and a consumer crash between its effect and deduplication write; a process-local map does not pass those tests.
Common trap
Calling the broker exactly once does not provide an end-to-end exactly-once database and external-API transaction.
Try it
List both crash windows and sketch the consumer transaction that handles the duplicate without repeating its business effect.
Worked solution
Recovery may publish the same event again using its stable identity. The consumer atomically records that identity and applies its durable business effect, so a replay is recognized. Marking sent before acknowledgement risks loss; marking afterward admits duplicates. The relay needs bounded claiming/retries and ordering policy, while external effects outside the consumer database need an additional idempotency or reconciliation boundary.
Change the constraint
How do retention, aggregate ordering and a poison event affect the design?
How did you do?
Self-assessment, not a compiler pass or interview score. Explain the reasoning unaided before marking understood.
Not marked. Progress stays on this browser, separate from course completion.
Primary references
Checked 2026-10-28. Use documentation for your pinned runtime, driver and tool version.