Go backpressure: bound concurrency, queues and request deadlines

A checkout service usually answers quickly. Then its pricing dependency slows down. Requests keep arriving, so an engineer moves a lookup into a goroutine and adds a larger channel buffer. A short test looks better: the handler accepts more work. Ten minutes later, memory climbs, useful throughput falls, and users retry requests that are still running somewhere inside the process. The program has become very good at accepting obligations it cannot finish.

Backpressure in Go means making downstream capacity visible to the producer of work. In an HTTP service, that often means refusing excess work promptly. In a streaming pipeline, it may mean slowing the source. In a durable job system, it means controlling delivery and acknowledging only completed effects. These are different contracts. Choosing one explicitly is more valuable than adding another concurrency primitive.

This guide develops a small admission limiter, then follows its consequences through deadlines, worker pools, HTTP connections, databases, retries, shutdown and testing. The example uses the standard library and was tested with Go 1.24.2; it does not depend on an experimental runtime feature. All numerical capacities below are worked assumptions, not benchmark results or production defaults.

The short version: bound admitted work; keep waiting finite; propagate the caller’s budget; release capacity when work actually ends; measure useful completions and rejected demand together. The bounded-work lesson introduces the same problem through a smaller exercise.

A rate limit answers how many operations may begin over an interval. A concurrency limit answers how many may be active together. A queue limit answers how much admitted work may wait. A byte limit answers how much data that work may retain. None of these quantities automatically bounds the others.

Imagine a dependency that normally takes 20 milliseconds but occasionally takes two seconds. A fixed arrival rate can produce a much larger active population during the slow period. Conversely, a concurrency cap can permit a high request rate when every operation completes cheaply. A channel with room for 100 elements says nothing about memory if each element retains a ten-megabyte byte slice.

Write the resource contract before choosing the mechanism. For a price lookup, it could be: at most 16 active lookups per process; no application waiting queue; at most 300 milliseconds of handler work; a bounded result; an explicit busy response when capacity is occupied. Another endpoint may need different limits because its dependency, payload and customer expectation differ.

The boundary also needs an owner. Construct the limiter once and share it across requests. Constructing a fresh limiter inside every handler invocation gives each request a private allowance and removes the intended aggregate bound. Similarly, giving every tenant an independent allowance without an overall ceiling can still exhaust the process.

Before changing code, trace a request from the load balancer to its final dependency. Possible waiting locations include an edge queue, an HTTP handler, an application channel, a semaphore, connection acquisition, a database lock, a remote server and response delivery to a slow client. A visible channel is only one part of that route.

For each location, record four things: who admits work, what limits it, which deadline applies, and how waiting ends. A useful review question is: if the dependency stops responding, which population can still grow? Do not accept “the pool has a limit” as a complete answer. A connection pool may limit connections while a larger population waits for them.

The first implementation in this article intentionally has no application wait queue. When all slots are held, it rejects the next request. That makes overload visible immediately and avoids storing promises that are unlikely to meet the caller’s deadline. It does not eliminate queues in the network, proxy or transport, and it does not make the process immune to abusive traffic.

For a pipeline, inspect both handoffs and cleanup. The official Go pipeline article illustrates cancellation-aware channel communication. Our cancellation-and-joining lesson extends the review into a practical question: after requesting stop, what observation proves the work has exited?

Start with a hypothetical route receiving 400 requests per second. Suppose its lookup stage spends an average of 40 milliseconds doing useful work under a stable representative workload. Multiplying arrival rate by time gives an average population of about 16 operations in that stage: 400 × 0.040. This arithmetic is a sizing hypothesis under the stated steady-workload assumptions. It does not predict tail latency or survive every change in traffic mix.

Now suppose the same work takes 200 milliseconds. Serving 400 requests per second would require an average population around 80. If the dependency remains healthy only below 24 concurrent operations, launching 80 goroutines cannot create the missing capacity. A useful response is to reduce admitted work, reduce cost, degrade an optional feature, or add independently usable capacity after measurement.

A queue adds waiting time. Under a simplified constant-service model, 30 waiting requests ahead of a new arrival at 300 completions per second represent roughly 100 milliseconds before its own service begins. Real queues have variable service times and scheduling effects; use this estimate to challenge a proposed buffer size, then verify with a workload test.

Reserve budget for serialization and response delivery. If the caller allows 300 milliseconds and the dependency already consumes 280, a large waiting queue has no plausible purpose for that request. The goal is a capacity decision that can be falsified, not a configuration number that merely looks conservative.

The following limiter.go is a complete source file. Its buffered channel represents occupied execution slots. The send uses a default branch, so a full gate refuses work immediately instead of parking another handler. A deferred receive returns the slot after the protected handler exits.

package backpressure

import (
	"context"
	"net/http"
	"time"
)

// Limit bounds simultaneous calls to next. It creates no application wait queue.
// Construct it once, then reuse the returned handler for all requests.
func Limit(max int, next http.Handler) http.Handler {
	if max < 1 || next == nil {
		panic("Limit needs a positive capacity and a handler")
	}
	slots := make(chan struct{}, max)
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		if err := r.Context().Err(); err != nil {
			if err == context.DeadlineExceeded {
				http.Error(w, "work budget exhausted", http.StatusGatewayTimeout)
			}
			return
		}
		select {
		case slots <- struct{}{}:
			defer func() { <-slots }()
		default:
			w.Header().Set("Retry-After", "1")
			http.Error(w, "service busy", http.StatusServiceUnavailable)
			return
		}
		next.ServeHTTP(w, r)
	})
}

// WithBudget propagates one handler-work budget. It does not interrupt a
// non-cooperating handler or impose a socket write deadline.
func WithBudget(budget time.Duration, next http.Handler) http.Handler {
	if budget <= 0 || next == nil {
		panic("WithBudget needs a positive duration and a handler")
	}
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		ctx, cancel := context.WithTimeout(r.Context(), budget)
		defer cancel()
		next.ServeHTTP(w, r.WithContext(ctx))
	})
}

Construct Limit during service setup. The gate stays held through the call to next.ServeHTTP, including response encoding. This is deliberately a conservative boundary. If slow response delivery should have a separate budget, introduce that boundary explicitly rather than releasing the dependency allowance while detached work still uses it.

Here, 503 means the service cannot currently admit the operation. A per-customer quota might use 429 instead. The Retry-After value is a hint for cooperative clients; it neither schedules a retry nor makes repeated writes safe. Authentication, route selection, cheap validation and edge protection should be arranged for your application so rejected traffic cannot consume an unlimited amount of work outside the gate.

The early context check avoids knowingly starting an already-canceled request. It is not an atomic guarantee against cancellation arriving one instruction later. Protected work must still honor its context. A zero capacity is rejected as configuration failure rather than producing a permanently unusable service.

WithBudget derives a context before entering the limiter. A shorter parent deadline remains shorter; each layer must not restart a fresh full allowance. The context is a cancellation signal for cooperating operations, not an instruction that can forcibly stop arbitrary CPU work or an uninterruptible library call. This distinction is part of the official context contract.

The next complete file, quote.go, accepts an injected lookup function. That keeps the business boundary testable without pretending a sleep is a real database. It validates a small identifier, classifies cancellation separately from ordinary dependency failure, and returns a small JSON result.

package backpressure

import (
	"context"
	"encoding/json"
	"errors"
	"net/http"
)

type Quote struct {
	ID    string `json:"id"`
	Cents int64  `json:"cents"`
}

type Lookup func(context.Context, string) (Quote, error)

// QuoteHandler is deliberately small: a production service should also log
// classified dependency errors and response-write errors without leaking them.
func QuoteHandler(lookup Lookup) http.Handler {
	if lookup == nil {
		panic("QuoteHandler needs a lookup function")
	}
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		if r.Method != http.MethodGet {
			w.Header().Set("Allow", http.MethodGet)
			http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
			return
		}
		id := r.URL.Query().Get("id")
		if len(id) == 0 || len(id) > 64 {
			http.Error(w, "id must contain 1 to 64 bytes", http.StatusBadRequest)
			return
		}
		quote, err := lookup(r.Context(), id)
		ctxErr := r.Context().Err()
		switch {
		case errors.Is(ctxErr, context.Canceled):
			return // The caller went away; do not create replacement work.
		case errors.Is(ctxErr, context.DeadlineExceeded):
			http.Error(w, "work budget exhausted", http.StatusGatewayTimeout)
			return
		case err != nil:
			http.Error(w, "lookup failed", http.StatusBadGateway)
			return
		}
		w.Header().Set("Content-Type", "application/json")
		_ = json.NewEncoder(w).Encode(quote)
	})
}

The handler checks its request context after the lookup returns, including the case where a dependency returns a value after cancellation. It does not disclose an internal error string to the caller. A production adapter should record a useful classified error and response-write failures in its observability path. The deliberately compact example does not implement authentication, business-specific missing-item semantics, or a universal error envelope.

The illustration here depicts a shared budget across stages. A work deadline still does not replace socket timeouts, body-size limits or a protocol for partially completed side effects. Review the HTTP request lifecycle when connecting this handler to a real client or database driver.

The repository contains the complete package, four tests, and a runnable demonstration at docs/examples/go-backpressure. The demo’s lookup waits on a timer or cancellation and returns a small fictional quote. It exists to make admission observable, not to predict a real service’s capacity.

The central route wiring is short:

mux.Handle("/quote", WithBudget(300*time.Millisecond,
    Limit(16, QuoteHandler(lookup))))

Run the example in one terminal, then send overlapping requests from another:

cd docs/examples/go-backpressure
go test -race ./...
go run ./cmd/demo

# In a second terminal, send 80 requests with up to 40 curl processes.
seq 1 80 | xargs -P 40 -I '{}' curl --silent --show-error \
  --max-time 2 --output /dev/null --write-out '%{http_code}\n' \
  'http://127.0.0.1:8080/quote?id=demo'

Depending on scheduling, this burst should expose a mixture of accepted and busy responses. There is no promised exact count. Repeat with a lower concurrency, then change the demo’s lookup delay. Watch the relationship between offered work and completed work rather than celebrating a high number of requests sent by the client.

The demo binds to loopback, configures header, write and idle timeouts, and uses a bounded shutdown period. It is a local learning harness. A sustained load test needs controlled arrival scheduling, representative payloads, explicit error accounting and measurements from both sides of the network.

Some operations are worth waiting for. A batch import may tolerate seconds of delay; an interactive autocomplete request often does not. If you introduce a queue, define an admission limit, an item-size limit and an expiration policy. A queue is a resource with a lifecycle, not a way to postpone a capacity decision.

Queue age can matter more than queue length. Ten stale jobs may be less valuable than one new job. Before starting queued work, check whether its deadline has expired and whether its result still has a consumer. Returning promptly from the caller does not remove an item already retained in a channel.

An in-memory queue also needs an ownership rule. Avoid putting http.ResponseWriter into a job that might outlive its handler. For durable asynchronous work, persist a job identifier and an explicit acceptance contract, then let a separate worker own retries and completion. Decide what happens if the process dies after accepting the request but before recording the job.

For data streams, backpressure can be a blocking handoff to a slower consumer. That only works if the producer can actually slow down and every blocking handoff has an exit path. If an upstream source cannot pause, the system needs a deliberate drop, spill or durable-storage policy. The worker-pool lesson is a useful place to compare these contracts.

A single admitted HTTP request may start several downstream calls. If 16 admitted requests each create 20 simultaneous lookups, the dependency sees up to 320 operations. The outer admission limit has kept its promise while the system still exceeds its useful downstream capacity.

Use a dependency-specific allowance shared across requests when protecting a common remote resource. Keep the request-level and dependency-level limits conceptually separate. Acquire scarce capacity before creating an unbounded population of waiting goroutines; otherwise the visible worker count can look bounded while blocked goroutines accumulate outside it.

errgroup.SetLimit bounds active functions in a group, and Group.Go can block while waiting to add another. TryGo provides a nonblocking decision. Choose what the caller should do when no slot exists, and do not treat the group limit as a complete service admission policy. These behaviors are documented in the errgroup reference.

For differently weighted jobs, the weighted semaphore package supplies context-aware acquisition and a nonblocking acquisition operation. It still needs sensible weights and an external policy for rejected demand. When several resources are required, avoid holding one scarce resource while waiting indefinitely for another; document acquisition order and test the failure path.

A connection is not the same unit as a request. HTTP/2 can multiplex streams over a connection, so a connection cap alone is not an application concurrency cap. Shared clients and transports avoid rebuilding connection machinery per request. Configure them once, then use the request context on outgoing operations.

For an HTTP dependency, inspect time spent obtaining a connection, establishing it, waiting for headers and reading the body. The httptrace package provides hooks that can help distinguish these stages. Do not turn those hooks into a second expensive logging workload; sample or aggregate for a bounded diagnostic purpose. Our HTTP connection-reuse guide covers response cleanup and transport reuse in more detail.

For SQL, a capped pool can move contention into connection waiting. The Go connection-management guide explains SetMaxOpenConns and the implications of acquiring limited connections. Compare acquisition waiting with query execution before increasing the pool. More connections may simply move the bottleneck into database CPU, locks or storage.

The same capacity review applies to other stores, but their driver and server behavior differ. Work through MongoDB under load and Redis under load for concrete follow-ups. A global service gate should not be enlarged just because a driver accepts a larger pool setting.

Retries create additional demand precisely when a dependency may have less capacity. If an ingress, application client and database adapter each perform three total attempts, a single logical operation can expand into as many as 27 downstream attempts along that path. That is a conditional upper bound, not a claim that every implementation always multiplies this way.

Choose one layer to own retries where possible. Specify a maximum number of attempts, a deadline shared with the original operation, backoff with jitter, and a rule for which failures are retryable. A retry budget should be observable separately from original demand. Avoid retrying an overload response immediately inside the same saturated process.

For a write, a lost response creates uncertainty: the effect may already have happened. Cancellation does not undo a committed transaction or a delivered external request. Use an application-level idempotency protocol when the operation requires it, and decide how long deduplication records remain authoritative. A request identifier that is only written to logs is not such a protocol.

Admission and retries should cooperate. A rejected request has consumed little protected capacity; an accepted operation that fails after expensive work has consumed much more. Do not hide these outcomes behind one generic error counter. The retry-pattern lesson and cache-stampede lesson explore related amplification failures.

Sixteen active requests can still exhaust memory if each retains a large payload. Estimate the live working set for a representative request, including decoded structures, response assembly, captured closures and buffered results. Track a byte-oriented bound where input sizes vary substantially. A request-count limit is most useful when operation costs are reasonably comparable.

Read bodies through an explicit size limit and reject oversized input before expensive decoding where the protocol allows it. Bound fan-out results and stream large outputs when the consumer contract permits streaming. A result channel with a small element count can still retain a large backing array through each element.

Pooling does not repair an ownership mistake. Returning a buffer to a pool while a response still references its bytes can corrupt the response under concurrency. Keeping every buffer forever to avoid corruption can create retention instead. The borrowed-buffer and pooling guide explains how to separate temporary storage from owned results; the slice backing-array guide shows why a small view can keep a larger allocation reachable.

Treat limits as a combination. An import endpoint may need a compressed-body limit, a decompressed-byte limit, a record-count limit and a cap on concurrently processed records. Writing those quantities down makes a code review much more concrete than the instruction to “use less memory.”

The crucial limiter test holds one operation inside a one-slot gate, sends another request, and expects an immediate busy response. It then cancels the first request, waits for the handler goroutine to finish, and proves a third request enters the same gate. That sequence checks refusal and recovery instead of merely checking that a channel has a chosen capacity.

Use synchronization channels to establish when a test operation has entered and when it has exited. Avoid assuming that sleeping for a few milliseconds proves a goroutine has reached a particular line. A generous watchdog can fail a stuck test, but elapsed wall-clock time should not be the primary oracle for correct ordering.

The example also checks a four-slot bound while all slots are occupied, validation without unnecessary lookup calls, the 504 path after cooperative timeout, and an already-expired parent deadline. These tests are intentionally about externally meaningful behavior. They do not establish a production throughput number.

Run the suite with go test -race ./.... The race detector documentation explains its scope: a clean run only covers the paths executed. It cannot prove absence of deadlocks, guarantee idempotency or establish that a bounded queue is suitably sized. Read the code and test changed failure assumptions as separate steps.

Measure offered requests, admitted requests, completed useful operations and rejections separately. Track successful-request latency independently from fast busy responses; an aggregate latency percentile can improve while users receive more failures. Also track deadline exhaustion, canceled requests, dependency errors and attempts per logical operation.

For every deliberate queue or acquisition gate, expose occupancy and waiting age or waiting duration where appropriate. Relate these measurements to downstream latency and resource use. A high rejection rate with stable successful throughput is a different incident from a low rejection rate with growing internal waiting and collapsing completions.

Runtime signals provide another view. The runtime/metrics reference describes metrics for live goroutines, scheduler latency and memory classes. Enumerate supported metrics for the toolchain you deploy rather than assuming every runtime exposes identical names. Pair process-level signals with endpoint and dependency measurements; one cannot substitute for the other.

Keep metric labels bounded. User IDs, raw URLs and arbitrary error messages can create a new storage problem inside the monitoring system. Prefer a small route template and classified outcome. When evidence suggests unfinished work, our goroutine-leak guide helps connect a profile to the component that owns the exit path.

Use a test dependency whose behavior you control. First establish a stable baseline at a modest offered rate. Then increase dependency latency while keeping offered demand constant. Record successful throughput, rejection rate, active work, memory and the distribution of completion times. Stop adding load when the tested environment reaches its safety limits.

Next introduce a bounded burst, a dependency error period, and client cancellations. Finally remove the fault and watch recovery. The service should return toward its baseline without spending a long period draining work that no caller still wants. If recovery remains slow, inspect retries, hidden acquisition queues, retained payloads and abandoned producers before increasing the gate.

A closed-loop load generator waits for responses before generating more work, so a slow service can reduce the generator’s offered rate. That is useful for some experiments but can conceal a demand pattern you intended to hold steady. State the arrival model in the report and distinguish scheduled arrivals from requests that the generator actually sent.

Do not combine race-detector runs with performance claims. Use race-enabled execution for correctness exploration and an appropriately built binary for capacity measurements. Record the Go version, CPU allocation, replica count, transport protocol, payload distribution and dependency configuration so a later result can be compared fairly.

A limit of 16 applies to one shared limiter instance. Ten replicas can collectively admit roughly 160 protected operations before considering other callers. Increasing replicas may help a CPU-bound application while overwhelming its shared database. Re-evaluate aggregate dependency demand during scaling, rollouts and failover rather than copying a per-process number into every deployment.

Separate optional work from the essential path where the product can tolerate degradation. For example, returning a cached recommendation may be preferable to letting an optional recommendation call occupy the same scarce allowance as payment verification. Document the user-visible fallback and distinguish it from a successful fresh result in metrics.

During shutdown, stop admitting new work, allow a bounded drain, then apply the service’s cancellation policy. The demo uses http.Server.Shutdown with a deadline and closes the server if that deadline expires. Real background workers need their own cancellation and joining protocol; changing readiness does not magically stop every goroutine already running.

Review termination behavior with the deployment system too. Load-balancer propagation, application drain time and the platform’s termination allowance must fit together. The probes-and-shutdown lesson connects those operational concerns to the Go process. A limiter that behaves well only before the first rollout is an unfinished design.

Should every Go service have a worker pool? No. A synchronous handler plus shared admission control may be simpler for request-scoped work. A pool is useful when there is a real job lifecycle, batching opportunity or separate ownership boundary. Its queue and shutdown still need explicit contracts.

Does a context timeout stop the work? Only if the implementation cooperates or the relevant operation observes cancellation. It also does not roll back completed side effects. The interview guide on cancellation is a good exercise in explaining this boundary precisely.

Can a semaphore replace a rate limiter? It protects a different quantity. Concurrency admission controls simultaneous occupation; rate limiting controls activity over time. Some systems need both, plus a payload limit and a tenant policy. Name the resource you are protecting before choosing a package.

How large should the buffer be? Large enough for a justified waiting contract under measured bursts, small enough that admitted work can still finish usefully, and bounded in retained bytes as well as items. If nobody can explain why a caller should wait, start by evaluating a no-wait policy.

For the next practice session, open bounded workers and overload. Explain the dependency, the allowed waiting, the cancellation path and the recovery signal before writing code. A strong Go service makes those answers visible in both its implementation and its operational evidence.