Go strings and Unicode: bytes, runes and real-world bugs

A profile service accepts a display name, an import worker extracts a short label, and a browser highlights the invalid part of a form. All three developers say they are handling “characters.” One counts bytes, another counts Unicode code points, and the browser reports positions in UTF-16 code units. Their individual functions pass ASCII tests. The combined feature breaks as soon as someone enters an accented name or an emoji. This is a contract failure before it is an encoding failure.

The useful question is not whether Go supports Unicode. It is which representation a particular operation promises to preserve. A transport limit concerns bytes. A decoder handles code points. A text cursor often needs grapheme boundaries. An identifier needs an explicit equivalence and security policy. A retained substring has an ownership cost unrelated to how many symbols it displays. Treating these as separate decisions makes the code easier to review and the tests much more revealing.

This original guide fills the 8 October 2026 archive edition. It was researched, tested and first published on 29 October 2026. The downloadable examples were checked with Go 1.27.3. The optional normalization module pins golang.org/x/text v0.43.0. This is a practical engineering investigation, not a claim that our small lab implements every Unicode algorithm. The five illustrations explain concepts; their shapes are not literal byte encodings or performance measurements.

The Go specification defines a string as an immutable sequence of bytes. A string can contain UTF-8 text, an ASCII protocol token, an embedded zero byte or malformed input. Its type alone does not certify which interpretation is correct. len(s) measures bytes, and s[i] selects a byte. That is precisely what a network frame parser needs, but it does not answer how many visible characters a person entered.

Consider a repository interface that returns string for both a document title and a binary digest. The representation is legal in both cases, yet the permitted operations differ. Lowercasing a digest is an application decision; decoding arbitrary digest bytes as text is usually a mistake. The interface documentation should name the content contract instead of expecting callers to infer it from the type. A validated title can have a constructor or a validation boundary without pretending that every string in the program has been validated.

Immutability also has a narrow meaning: callers cannot assign into a string's bytes through ordinary indexing. It does not mean that every substring is a separate allocation, that indexing is Unicode-aware, or that conversion from external bytes performed validation. Keep these claims separate during code review. The official strings explanation is useful background; the practical habit is to annotate each boundary with its unit and acceptance rules.

Here are concrete values printed by the accompanying lab. The last column describes these particular sequences under default extended grapheme segmentation; it is not computed by utf8.RuneCountInString.

A rune is an alias for int32, conventionally used for a Unicode code point. A sequence of runes can form one extended grapheme cluster. The emoji example includes a woman, a skin-tone modifier, a zero-width joiner and a laptop. Counting four decoded values is correct. Advertising that result as four user-visible characters would be the application bug.

Unicode's segmentation specification defines default boundaries and permits tailoring. A cluster is not a guaranteed screen width, font glyph count or culturally complete definition of a letter. Terminal alignment and proportional browser layout introduce additional concerns. For a database envelope, count bytes. For a code-point exercise, count runes. For a text editor's delete operation, choose and test a grapheme segmentation implementation. Do not silently substitute one unit because its standard-library function is convenient.

Go's range loop over a string is useful because it decodes UTF-8 while retaining an index into the original byte sequence. That combination is often misunderstood. The index does not become a rune number just because the value is a rune. The following complete program makes the distinction visible:

package main

import "fmt"

func main() {
    for offset, value := range "A😀B" {
        fmt.Printf("byte=%d rune=%U\n", offset, value)
    }
}

The offsets are 0, 1 and 5. They identify byte positions in the original six-byte string. Those offsets are suitable for slicing that original representation at decoded boundaries. They are not interchangeable with a browser selection index, a row number in a rune array or an index into a normalized copy. Once a transformation changes the representation, old positions need a mapping or a new calculation.

Imagine a validator that stores offset and then inserts a normalized version into a form. Even if both strings look identical, the saved offset may point inside a different sequence. Keep validation errors attached to the exact representation they describe, or return a named unit with sufficient context to translate it. A field called characterIndex hides this decision. Names such as utf8ByteOffset and utf16Start make incompatible operations harder to accidentally connect.

Malformed input and a literal replacement character can both produce the rune value U+FFFD during decoding. Their widths distinguish the cases. The utf8 package documentation specifies that malformed encoding decoded this way produces RuneError with width one. A valid UTF-8 encoding of U+FFFD occupies three bytes. Rejecting every RuneError value would reject legitimate text; accepting every one would lose the distinction between valid and malformed input.

The lab tests "\xff", an incomplete multibyte sequence, an overlong encoding, an encoded surrogate value and a valid "\ufffd". They are deliberately different fixtures. A screenshot of rendered text would be a poor oracle because a font may display several failures with the same replacement symbol. Inspect quoted values, byte sequences and decoder widths when diagnosing the boundary. Avoid recording actual private user input in general-purpose logs merely to obtain an example.

Choose rejection, replacement or byte preservation according to the field's contract. An API that requires valid display text can reject malformed encoding before storage. A forensic importer may need to preserve original bytes separately. A repair path should record that a transformation occurred instead of claiming that the repaired output is identical to the source. The downloadable lab deliberately rejects malformed input for its prefix functions. That policy is visible, testable and stricter than simply converting through []rune.

A byte ceiling is common in protocols, indexes and storage records. Cutting with s[:n] enforces that ceiling but can stop inside an encoded code point. Converting the whole string to runes changes the unit and adds work that may not match the requirement. Our lab validates the complete input, walks the proposed end backward to a rune boundary and copies the accepted prefix into independent storage:

func PrefixBytes(s string, n int) (string, error) {
    if n < 0 {
        return "", ErrLimit
    }
    if !utf8.ValidString(s) {
        return "", ErrInvalidUTF8
    }
    if n >= len(s) {
        return strings.Clone(s), nil
    }
    end := n
    for end > 0 && !utf8.RuneStart(s[end]) {
        end--
    }
    return strings.Clone(s[:end]), nil
}

This function has a deliberately limited promise: its result is a valid UTF-8 prefix within a nonnegative byte budget, and malformed input is rejected even if the bad suffix would have been removed. It does not preserve whole grapheme clusters. For "e\u0301" and a budget of one, the result is "e". The encoding remains valid while the accent disappears. That is acceptable only when the caller explicitly wants this byte-oriented policy.

If a UI adds an ellipsis, reserve its encoded length before selecting a prefix. Decide what happens when the entire marker does not fit, and distinguish a storage limit from a display preview. A preview should not silently replace the original value used for identity, signing or editing. The lab also includes PrefixRunes; its name is a contract, not a claim that counting code points solves every visual truncation problem.

Suppose an importer reads a large record, extracts a short label and keeps that label in a cache. The source record is otherwise discarded. A substring may still share the original backing allocation, so the cache can retain substantially more memory than the label's visible size suggests. This is a reachability problem: the garbage collector cannot release storage that remains reachable through a live value. It is unrelated to whether the contents are ASCII or multilingual text.

The strings.Clone documentation provides an explicit ownership tool: apart from the empty-string case, the result has a fresh copy of the string data. The article's prefix helpers use it because their teaching contract includes independent output storage. That is a choice for these helpers, not a recommendation to clone every assignment, return value or substring in an application.

The useful review question is where a small value outlives a much larger source. A temporary parser view consumed immediately may benefit from sharing. A short field stored for hours may warrant copying at the ownership boundary. Measure representative retained data before changing every call site. Our heap reachability guide explains why removing one reference may not free an allocation, and the allocation profiling lab separates retained memory from allocation traffic. This article intentionally supplies no invented memory savings or universal allocation benchmark.

Two strings can render alike while containing different code-point sequences. Our fixtures "é" and "e\u0301" demonstrate the simplest useful case. Raw equality reports false. If a product's policy treats canonical equivalents as the same display label, both sides need the same explicit transformation before comparison. The Unicode normalization specification distinguishes canonical normalization from compatibility normalization; they should not be selected interchangeably.

The optional lab uses NFC through Go's normalization package. CanonicalLabel rejects invalid UTF-8 and applies norm.NFC.String. The function is intentionally named for display-data handling. It is not a general account-name constructor, a password transformer or a filesystem security boundary. Those uses require additional domain rules and careful compatibility decisions.

Consider a cache that normalizes on writes but looks up raw search input. Canonically equivalent requests can miss even though all stored entries are normalized. Conversely, normalizing only during lookup can collapse multiple previously distinct records into one apparent key. Define the transformation at a consistent boundary, store the original spelling where appropriate, and decide how existing collisions are reported. A migration should inventory collisions before changing uniqueness constraints. The engineering work is not finished when one call to NFC.String produces the expected output in a demo.

In our test, NFC preserves the circled digit ①, while NFKC produces 1. That difference may be useful for a particular search experience. It can be wrong when the distinction carries meaning in a document, an identifier scheme or an exact external representation. A general helper named CleanString makes this policy difficult to notice; a field-specific transformation with examples makes it reviewable.

Start by separating the stored display value, the searchable derivative and the stable record identifier. Search may be forgiving without rewriting the text a person authored. A stable opaque ID can remain independent of display spelling. An export can preserve original content while an index stores a documented normalized key. These are architecture choices, and no single Unicode normalization form decides them for the product.

A migration exercise is more valuable than another happy-path example. Create two existing records that the proposed transformation merges. Specify whether migration fails, asks an operator to resolve the collision or allocates distinct stable identifiers. Test reads during a rolling deployment where old and new application versions coexist. Also record the transformation version alongside derived data when reproducibility matters. Without this planning, an apparently harmless cleanup can alter cache behavior, uniqueness, audit comparisons or reconciliation. The goal is a consistent data contract rather than a string that merely looks tidier in a screenshot.

Case-insensitive matching needs its own requirement. Our standard-library fixtures show strings.EqualFold("Go", "go") as true, strings.EqualFold("ß", "ss") as false, and the Kelvin sign matching K. These results are useful precisely because they defeat the assumption that every comparison is an ASCII lowercase comparison. Equal-looking spelling, canonical equivalence and case equivalence are separate relations.

The optional lab compares Straße and STRASSE using cases.Fold, which performs full Unicode case folding. That comparison succeeds in the tested version. The cases documentation also warns that folding does not normalize input and may not preserve a normal form. A matching pipeline therefore needs an explicitly chosen sequence and tests for that sequence. Do not assemble a universal identity algorithm from three unrelated helper calls and assume the ordering is immaterial.

Use an example-driven specification. Should a search query match canonical equivalents? Should it ignore case? Is the result a list of candidates or a unique account lookup? Can a locale-specific user expectation conflict with a protocol's rules? Those questions determine whether a generic fold is appropriate. Keep the original display text for presentation, test collisions in the derived index and document which transformation produces the lookup key. A forgiving search result can tolerate multiple candidates; an authorization lookup cannot casually merge principals because their names compare similarly.

The lab contains two apparently similar spellings of a familiar payment-related word: one uses Latin letters, and one substitutes a Cyrillic letter. NFC leaves them distinct. This is expected. Normalization is not a universal visual-lookalike detector, and rejecting malformed UTF-8 does not eliminate misleading valid text. The Unicode security mechanisms report is a primary reference for confusable and identifier considerations, not a promise that a single standard-library predicate implements a complete defense.

Our ASCIIKey example takes a narrower route for a deliberately machine-oriented key. It permits one to thirty-two bytes of lowercase ASCII letters, digits, hyphens and underscores. The function rejects the Cyrillic fixture. This is suitable as an example protocol namespace with a stated alphabet; it would be an exclusionary and incorrect general validator for people's names. A display label and a routing key should not accidentally share the same constructor.

For sensitive interfaces, distinguish display from authority. Show a stable identifier or verified destination when the action matters. Escape control characters in diagnostic views, avoid building terminal instructions from untrusted labels and review bidirectional display behavior in the actual UI. Do not claim the lab implements mixed-script detection, a confusable skeleton or bidirectional text security. It demonstrates why those are additional requirements. A useful security review names the threat and the boundary instead of declaring all Unicode either safe or forbidden.

A reader can split a multibyte encoding across calls. A chunk ending halfway through a character is not, by itself, evidence that the complete stream is malformed. Validating every transport chunk independently can therefore reject valid input. The opposite shortcut, accepting every partial tail indefinitely, can hide malformed input at the actual end. A streaming decoder needs state and an explicit final-input decision.

The lab uses bufio.Reader.ReadRune to handle boundaries rather than implementing an ad hoc chunk decoder. It counts only the valid prefix before an error and preserves non-EOF reader failures. Its critical distinction is the decoded width, not merely the rune value:

value, width, err := b.ReadRune()
if err == io.EOF {
    return counts, nil
}
if err != nil {
    return counts, err
}
if value == utf8.RuneError && width == 1 {
    return counts, fmt.Errorf("byte %d: %w", counts.Bytes, ErrInvalidUTF8)
}
counts.Bytes += int64(width)
counts.Runes++

This fragment is the loop body; the download supplies the complete function, imports and tests. The test reader varies chunk sizes and returns final data together with EOF. Another fixture returns valid bytes followed by a synthetic failure. Both behaviors matter: a consumer must account for final bytes and still report a real failure rather than rewriting every end condition as success. The streaming I/O guide develops that broader reader contract.

Decoder state and normalization state solve different problems. Suppose one application chunk contains e and the next contains a combining acute accent. Each chunk can be valid UTF-8. Normalizing them separately and concatenating the results does not necessarily produce the same normalized representation as processing the combined sequence. Our optional test demonstrates that difference and checks a stateful norm.NFC.Reader against the whole-input result for this fixture.

There is another subtle boundary. The Go normalization implementation uses a stream-safe approach. As discussed in the Go normalization article, a sufficiently long sequence of non-starters can lead to insertion of U+034F, the combining grapheme joiner. Our artificial input, a followed by thirty-one acute accents, exercises that behavior. The test checks for the inserted character instead of assuming every normalizer output is just a different ordering or composition of the original code points.

That observation matters when byte-exact equivalence, hashes or signatures are involved. Do not introduce this transformation into a signed payload path without checking the protocol's exact rules. Do not call the supplied helper a universal canonicalizer. For an ingestion service, name the accepted form, the implementation version, the handling of long combining sequences and the relationship between original and derived storage. The stream-safe behavior is a concrete reason to test unusual but valid input, rather than limiting the fixture corpus to ordinary names.

Our A😀B fixture has six UTF-8 bytes, three code points and four UTF-16 code units. The Go utf16 package lets the lab make the last measurement explicitly. A browser-side editing API may describe positions in UTF-16 units; a Go parser may report byte offsets. Passing the integer unchanged can highlight the wrong span even though both sides processed valid text.

Define the protocol for positions before implementing the adapter. Include the unit, the original document version and whether the interval end is exclusive. Reject a position inside an encoded unit that the contract does not permit. A selection based on yesterday's document version should not silently apply to newly normalized or edited content. Collaboration introduces another dimension: mapping positions across edits is a document transformation problem, not an encoding conversion alone.

For a small form, the simplest solution may be returning a field-level error rather than a precise span. For a code editor, translation is worthwhile but needs fixtures containing supplementary code points, combining sequences and edits before the error location. Test round trips only for valid boundary positions; arbitrary byte positions are not necessarily representable as valid text selections. The important outcome is that both sides can state what the number means. A seemingly convenient int without a unit is the source of the defect.

Reproducible text behavior needs more than a module name. The saved lab was tested using Go 1.27.3 and golang.org/x/text v0.43.0. In that combination, both unicode.Version and norm.Version report 17.0.0. The contemporary Unicode documents referenced here describe version 18.0.0. Reading the latest specification does not mean an installed library already implements all of its data or algorithms.

The selected x/text normalization tables also depend on compiler build tags. Inspection of the pinned module shows the Go 1.27 build selecting its 17.0.0 tables; the alternative table file targets an older compiler selection. Consequently, recording only go.mod is insufficient evidence for a claim about the actual tables used. Our programs print the runtime-visible constants so the receipt records the tested combination rather than an assumption about the newest release.

When a normalization or matching change affects stored lookup keys, treat dependency upgrades as data-compatibility work. Keep fixtures for previously accepted values, compare transformed outputs and inspect new collisions before rewriting indexes. A search suggestion service can often rebuild a derivative index; an account namespace may need an explicit migration plan. The reproducible-build discussion is related, but here the immediate lesson is smaller: record the selected compiler and tables, and avoid advertising conformance beyond the implementation actually exercised.

A small streaming buffer does not make an unbounded stream safe to consume. CountUTF8 has bounded working storage for decoding, but it does not set a total input limit, a network deadline or an application cancellation policy. A peer that sends valid text forever still causes indefinite work. A peer that stalls can leave the reader waiting. Text validation and resource admission are separate layers.

Define limits before expensive transformations and decide whether a second limit applies to transformed output. A byte envelope constrains transport and storage. A rune ceiling can constrain a documented code-point policy. A grapheme-oriented display rule may require additional segmentation work. None of these should be described vaguely as “a maximum name length” without a unit. For APIs, state whether the request is rejected, truncated or queued, and make error responses consistent.

Avoid rescanning the full accumulated document after each small edit when a bounded field or incremental design would suffice. Benchmark the actual workload before replacing clear code with complex caching. The claim in this article is correctness for a finite tested input, not a universal latency bound. For a handler that also waits on a database or another service, combine these rules with the deadline diagnosis workflow. Cancellation must reach the operation that can stop; a text helper cannot enforce a blocked dependency's lifecycle on its own.

The standard-library tests exercise every byte cut through the mixed Unicode fixture. For each accepted result they check validity, the byte ceiling, the original-prefix relationship and the maximal permitted rune boundary. Separate tests cover negative limits, zero limits, oversized limits and malformed suffixes. Those details prevent a superficially reasonable rewrite from silently changing the contract. The rune-prefix tests intentionally preserve the distinction between code points and grapheme clusters.

Streaming tests vary chunk boundaries, include final data with EOF and preserve a synthetic non-EOF failure. Normalization tests distinguish canonical and compatibility forms, show that case folding is separate, compare stateful chunk processing and exercise stream-safe insertion. None of those tests depends on how a particular font draws the text. A rendered screenshot is valuable for layout review, but it is not the assertion that makes an encoding routine correct.

We also ran negative controls: temporary deliberately broken variants must fail the relevant tests. Removing the byte-boundary correction, confusing a valid replacement character with malformed input or changing NFC into NFKC should be observable failures. The exact mutation results are included with the lab. Passing tests alone cannot establish complete Unicode conformance; targeted failures establish that these assertions actually notice the mistakes they were designed to catch. That is a much stronger teaching artifact than a green test that merely repeats the implementation's current output.

The included fuzzer generates arbitrary strings and byte budgets. Invalid UTF-8 must be rejected under this helper's policy. For valid input and a nonnegative budget, an accepted result must remain valid, stay within the budget and be a prefix of the original. Seeds include combining marks, an emoji sequence, an empty value and malformed bytes. A short local campaign executed 64,481 cases without a failure. That count describes this run, not a coverage guarantee for all possible text.

Do not add a property that accidentally assumes the desired conclusion. For example, “the output has at most n characters” is useless until the unit is defined. Requiring normalization to preserve original bytes would reject intended changes, while requiring all visually similar strings to normalize equally would specify an impossible or unsafe identity policy. Properties should state the exact transformation contract and leave unrelated behavior outside their scope.

Keep discovered failures as regression fixtures with a readable explanation. A five-second fuzz run is a useful supplement to deterministic cases, not a replacement for migration testing, segmentation conformance suites or realistic performance investigation. If you adopt a third-party grapheme segmenter later, run its relevant official conformance data under a pinned version and add application-specific editing cases. This lab deliberately stops short of implementing that algorithm, so it does not mislabel rune counting as complete text segmentation.

Download the complete Go Unicode lab. The standard-library module needs no third-party package. The optional nested module pins x/text and includes its checksum file. Initial dependency setup requires access to the module source or an already populated cache. Once the matching compiler and dependencies are installed, the recorded tests also pass with module downloads disabled. The bundle contains original source, tests, expected output, version notes and negative-control evidence.

# Standard-library examples
go run ./textlab
go test -race ./...
go test ./textlab -run '^$' -fuzz FuzzBytePrefix -fuzztime=5s

# Optional normalization examples, from the lab root
cd normlab
go mod download
go run .
go test -race ./...

For GopherForge Playground, save any existing draft first and start with the complete standard-library main.go from the bundle. Editing a room does not execute code. A compatible connected app device runs when its participant explicitly requests execution; other participants can inspect the shared source and device-attributed output. The MCP guide explains the same separation for an AI client. Do not assume that every native sandbox already includes the optional dependency or supports every desktop tool command.

Then adapt one boundary from your own application: a label extractor, a query normalizer or an error-span adapter. Write the input contract first. Replace private production examples with synthetic fixtures that preserve the failure shape. Add one adversarial case that would have defeated the original implementation, and make sure a deliberately broken version fails it before trusting the green result.

Ask a candidate why len("é") and a rune count differ, then change the fixture to a combining sequence. A strong answer names bytes and code points, recognizes that neither is necessarily a user-perceived character and asks what the product intends to limit. The follow-up should concern a real operation, such as truncating an API field, moving a cursor or generating a signed record, rather than rewarding memorized terminology alone.

Ask why two identical-looking map keys can miss. The answer should separate byte equality, canonical equivalence, case folding and visual confusables. An experienced engineer should also discuss where the policy belongs, how existing collisions would be migrated and why passwords or signed payloads must follow their own protocol. Suggesting NFC everywhere without considering storage and authority is an incomplete answer, even if the candidate remembers the package import.

Finally, ask whether a tiny string can retain a large input, and whether a valid UTF-8 stream can fail per-chunk validation. These connect language behavior to memory ownership and I/O contracts. Work through the strings interview guide, the runes lesson and the borrowed-buffer lesson. The goal is to explain a failing example, choose an explicit policy and verify the boundary. That skill transfers from an interview exercise to a production importer, an editor and a multilingual API.

Before shipping a text-handling change, put the acceptance policy into the review description. State whether malformed UTF-8 is rejected, repaired or preserved. Name the unit for every length and offset. Identify which representation is stored, displayed, searched and signed. Record whether a retained result shares or copies the source. For transformations, include the implementation version and at least one collision example. These statements let reviewers assess behavior without reconstructing it from scattered helper calls.

Then verify the failure paths: a cut inside an emoji encoding, an accent split across chunks, a valid replacement character, an incomplete final sequence and a reader that returns data before failing. For an identity-sensitive field, add a visually confusable but distinct value and an existing-record migration scenario. For an editor, include positions after a supplementary code point and a document revision mismatch. Each fixture should have a reason to exist and a contract it protects.

Keep the implementation proportionate to the requirement. A lowercase ASCII wire key does not need a general text editor's segmentation stack. A person's display name should not inherit that wire key's alphabet restriction. A byte-preserving protocol should not be normalized because normalization sounds cleaner. Use the course map to revisit the underlying language rules, then keep this lab beside the boundary being changed. The most durable improvement is a small, explicit contract that the next maintainer can explain and test.