Grokking Advanced Go

0% completed

Strings, Bytes, and Conversions

On This Page

The String Header

Conversions Copy

When the Compiler Avoids the Copy

Substrings Share Memory

Choosing a Way to Build a String

Bytes, Runes, and Invalid UTF-8

Key Takeaways

Building a string of 1,000 characters with += took 999 allocations in the measurement below. Building the same string with strings.Builder took 9, and telling the builder the size in advance took 1.

. . . .

Output:

+=: 999
Builder: 9
Builder with Grow: 1

The arithmetic explains the gap. Each += allocates a new string and copies all the bytes so far. The loop therefore copies 1 + 2 + ... + 999 bytes, which is 499,500 bytes, to produce a 1,000-byte result. The first += makes no allocation, since joining an empty string to "x" needs no new memory. That is why the count is 999 rather than 1,000.

Those numbers follow from how strings are stored. The sections below describe that storage, the conversions to and from []byte that copy it, the cases where the compiler avoids the copy, and how a short substring can keep a large string in memory.

The String Header

A string value is a small header of two words: a pointer to the bytes and a length. On amd64 that is 16 bytes, whatever the length of the string. A slice header has a third word for capacity, because a slice may have room to grow, whereas a string never changes after it is created.

Strings are immutable. No operation in the language changes the bytes of an existing string, so s[0] = 'x' does not compile. This rule is what makes sharing safe. Assigning a string copies only its header. Any number of strings can point to the same bytes, with no risk that one holder changes them for another.

A []byte has the opposite property, since its elements can be changed at any time. The two types look similar and hold the same kind of data, but only one of them guarantees that its bytes never change. That difference determines when a copy is needed.

Conversions Copy

Because a string must never change and a []byte can change at any time, the two cannot normally share memory. A conversion between them therefore allocates new memory and copies the bytes, as the next program shows.

. . . .

Output:

Z-100 A-100
A#100 A-100

Changing data after the conversion has no effect on sku, and changing back has no effect on sku either. If these conversions shared memory, the string would change, which the language does not allow.

When the Compiler Avoids the Copy

The compiler can skip the copy when it can prove that the converted value is used only briefly and never changed. The program below measures these cases as of Go 1.27 with a 45-byte string. It stores results in package variables, so that the compiler cannot remove the work.

. . . .

Output:

text = string(data)        1
raw = []byte(sku)          1
prices[string(data)]       0
string(data) == sku        0
range []byte(sku)          0
local string, 45 bytes     1
local string, 5 bytes      0
prices[string(data)] = 1   1

The first two lines are the general case: a result that outlives the expression needs its own memory. The next three lines show the optimizations that the compiler applies to specific forms. A map lookup with m[string(b)] uses the bytes only during the lookup. A comparison uses them only during the comparison, and a range over []byte(s) only reads them.

Assigning to the map, on the other hand, allocates once. The map may have to store the key after the statement ends, so prices[string(data)] = 1 needs its own copy of the bytes, even though this key is already present. A lookup does not need that copy.

The two local lines show a different optimization. A string that stays local and fits in 32 bytes uses a buffer on the stack (tmpStringBufSize in runtime/string.go). The 5-byte string therefore allocates nothing, and the 45-byte string allocates once. Because such a result depends on the exact form of the code, confirm it with a measurement before you rely on it.

A substring is a new header that points into the original bytes, so slicing a string allocates nothing and copies nothing. The cost appears later. The garbage collector frees a string's bytes as one block, so a short substring keeps the entire original string in memory.

A substring shares the bytes of its string

. . . .

One possible output:

substring: A-100 10 MB
clone:     A-100 0 MB

The function strings.Clone guarantees a new allocation that holds only the bytes of its argument, which lets the garbage collector free the large document. Its documentation adds a warning: overuse makes a program use more memory, since every call makes a copy. It therefore suits a small part of a large string that will be kept for a long time. One example is a value parsed from a large document and stored in a cache.

Choosing a Way to Build a String

strings.Builder keeps its bytes in a []byte and grows it in the same way that append grows any slice. That growth accounts for the 9 allocations. Its String method returns a string that shares that memory without copying it (strings/builder.go). This is safe because the builder only ever appends after that point and never changes the bytes already written.

Method	Allocations for the 1,000-byte string	Best for
`+=`	999	A few joins of short strings
`strings.Builder`	9	A loop with an unknown final size
`strings.Builder` with `Grow`	1	A loop with a known or estimated size
`strings.Join`	1, because it computes the total size first (`strings/strings.go`)	Joining an existing slice of strings

A single expression such as first + "-" + last is fine, because the compiler joins all its parts in one allocation. The problem with += appears only inside a loop, where the amount copied grows with every pass.

Bytes, Runes, and Invalid UTF-8

Some conversions change the bytes as well as copy them. The expression []rune(s) decodes UTF-8 into one 4-byte rune per code point (one encoded character), and converting back encodes those values again. A byte sequence that is not valid UTF-8 decodes to the replacement character U+FFFD. The round trip therefore does not always give back the original bytes.

. . . .

Output:

5 4 5
41 ff
41 ef bf bd
true false

The string café is 5 bytes and 4 runes, because é takes 2 bytes. A conversion to []byte and back always preserves the bytes exactly, including invalid ones. A conversion through []rune replaces the invalid byte ff with the three bytes ef bf bd, which encode U+FFFD. If your data may be binary, or may not be valid UTF-8, keep it as string or []byte, and do not pass it through []rune.

Key Takeaways

A string is a 16-byte header on amd64 over bytes that never change, which is what makes sharing them safe.
Conversions between string and []byte copy, except in forms such as m[string(b)], string(b) == s, and range []byte(s).
A substring shares the memory of the original, so use strings.Clone when you keep a small part of a large string.
+= in a loop copies all earlier bytes on every pass, while strings.Builder grows like a slice.
Call Grow on a builder when you know the final size, which reduces the allocations to 1.
Converting through []rune replaces invalid UTF-8, while converting through []byte preserves every byte.

A string is a read-only view of bytes, and each cost on this page follows from that rule. Sharing is cheap because nothing can change the bytes, while a conversion to a type that can change them must make a copy. When a program allocates more than expected around strings, look first for a loop of +=, then for conversions whose results outlive their expression.

Mark as Completed

Reading Progress

Vote for new content

On This Page

The String Header

Conversions Copy

When the Compiler Avoids the Copy

Substrings Share Memory

Choosing a Way to Build a String

Bytes, Runes, and Invalid UTF-8

Key Takeaways