Grokking Concurrency in Go

0% completed

Goroutines and What They Cost

On This Page

Starting a Goroutine

The Goroutine Stack

Goroutines and Operating System Threads

When main Returns

Loop Variables in Goroutines

Using This in an Interview

Key Takeaways

. . . .

One possible output (measured on one computer with 16 logical CPUs):

100000 goroutines in 19ms

The program starts 100,000 goroutines, and each one sends a single value into a channel that has room for all of them. The main function then receives 100,000 values. The time printed therefore covers starting every goroutine, running it, and collecting its value. On this machine, that came to about 190 nanoseconds per goroutine.

A cost that small is the reason Go programs start goroutines so freely. What exactly is a goroutine, what does it cost in memory and time, and how does it differ from an operating system thread? The answer also covers what happens to goroutines when main returns and how a loop variable behaves inside a goroutine.

Starting a Goroutine

A goroutine is a function call that runs independently of the code that started it. The statement go f(x) evaluates f and x in the current goroutine and starts a new goroutine that runs f(x). It then continues with the next statement without waiting. When f returns, its goroutine ends, and any values that f returns are discarded.

go fetch("https://example.com/a")   // a named function
go func() {                          // a function literal, called at once
	fmt.Println("running")
}()

Because the arguments are evaluated before the new goroutine starts, go fetch(url) uses the value of url at the moment the statement runs. A function literal, on the other hand, reads any variables it uses from the enclosing function when it runs, which may be later.

Every program has at least one goroutine. The function main runs in a goroutine of its own, called the main goroutine, and every goroutine that your code starts is started from it, directly or indirectly.

The Goroutine Stack

Each goroutine needs a stack, which is the memory that holds the local variables and return addresses of the function calls it is making. The Go runtime gives a new goroutine a small stack, 2 KB at minimum. When a function call needs more room, the runtime allocates a larger stack, copies the old one into it, and continues. When a goroutine no longer needs the room, the garbage collector (the part of the runtime that frees unused memory) can shrink the stack again.

The limit for one goroutine stack is large. The documentation of debug.SetMaxStack in the runtime/debug package gives the initial setting as 1 GB on 64-bit systems and 250 MB on 32-bit systems. A goroutine that exceeds it, usually through infinite recursion, crashes the program.

The program below measures the starting stack directly. It starts 10,000 goroutines that each wait forever on a channel, so all of them stay in memory at once. It then compares the stack memory in use before and after.

. . . .

Output on Linux, where the course player runs the code:

goroutines that exist: 10001
stack per goroutine: about 2 KB

The count is 10,001 because the main goroutine is included. On Linux, each new goroutine starts with the 2 KB minimum. The same program prints about 8 KB on Windows, because the runtime source (runtime/stack.go) adds 4,096 bytes to every stack on Windows for the operating system and then rounds the result up to a power of two. Since Go 1.19, the runtime may also start new goroutines with a larger stack: after each garbage collection, it sets the starting size to the average stack size it found, never below that minimum.

At the minimum sizes, 10,000 waiting goroutines need between 20 MB and 80 MB of stack. The program ends with all 10,000 goroutines still waiting. That is allowed, for the reason given in the section on main below.

Goroutines and Operating System Threads

An operating system thread is a sequence of instructions that the operating system schedules onto a CPU core. Each thread usually reserves 1 MB (Windows) to 8 MB (Linux) of stack, and creating or switching threads goes through the operating system kernel.

The Go runtime does not create one thread per goroutine. Instead, it runs many goroutines on a small number of threads. It switches a thread from one goroutine to another in its own code, without a call into the kernel. That design is why a program can hold 100,000 goroutines when 100,000 threads would need far more memory.

Many goroutines on a few threads

	Goroutine	OS thread
Created by	The Go runtime	The operating system
Starting stack	2 KB minimum, grows and shrinks	Usually 1 MB to 8 MB, fixed
Switching	Done by the Go runtime	Done by the kernel
Cost to create	A call into the Go runtime	A system call to the kernel
Identity visible to code	None	Thread ID

Goroutines are cheap, but they are not free. Each one holds its stack and anything its stack points to until it ends, so a goroutine that waits forever holds that memory forever. A goroutine that never ends because nothing will ever unblock it is called a goroutine leak. A leak of one goroutine per request grows memory until the program fails.

When main Returns

The Go specification states that when main returns, the program exits, and that it does not wait for other goroutines to complete. Every other goroutine stops at that moment, whatever it was doing, and no deferred calls run inside it.

This rule is why the stack program above can end with 10,000 waiting goroutines. It is also why a program that starts goroutines must make main wait for them. The programs so far have done that by receiving one value per goroutine.

Loop Variables in Goroutines

Starting goroutines inside a loop is the most common way to start many of them. Since Go 1.22, each iteration of a for loop has its own copy of the loop variable. A function literal started in one iteration therefore sees the value from that iteration.

. . . .

Output:

[0 1 4 9 16]

Before Go 1.22, a three-clause loop such as for i := 0; i < 5; i++ had one variable i that every iteration shared, and a range loop over a slice shared its variables in the same way. All five function literals read that same variable, often after the loop had already set it to 5. The program above was rewritten with three-clause loops (ranging over an integer also needs Go 1.22) and built with the old rules, in a module whose go.mod declares go 1.21. Three runs printed [4 9 16 25 25], [4 4 25 25 25], and [1 4 25 25 25]. The race detector reported a data race, and go vet warned loop variable i captured by func literal. The usual repair was to copy the variable inside the loop or to pass it as an argument:

for i := 0; i < 5; i++ {
	i := i // a new variable for each iteration; not needed since Go 1.22
	go func() { squares <- i * i }()
}

The new behavior applies to code in a module whose go.mod file declares go 1.22 or later. Older code that still contains i := i works the same, and the line can be deleted.

Using This in an Interview

A goroutine is a function running independently, managed by the Go runtime rather than the operating system. It starts with a small stack that grows as needed, and the runtime runs many goroutines on a few OS threads. A likely follow-up asks what happens to running goroutines when main returns. The program exits immediately, and they stop without finishing.

Key Takeaways

go f(x) evaluates f and x in the calling goroutine, then runs the call in a new goroutine without waiting for it.
A goroutine starts with a stack of 2 KB at minimum, which grows and shrinks; on Windows the starting stack measures 8 KB.
The runtime runs many goroutines on a few OS threads, so a program can hold far more goroutines than threads.
When main returns, the program exits and all other goroutines stop.
Since Go 1.22, each loop iteration has its own loop variable.
A goroutine that never ends keeps its memory, which is called a goroutine leak.

A goroutine costs a few kilobytes of memory and, on the machine measured here, less than a microsecond to start. That is little enough that Go code commonly starts one goroutine per task. The costs that matter in practice come from how long goroutines live. Every goroutine therefore needs a clear way to end and a clear owner that waits for it.

Mark as Completed

Reading Progress

Vote for new content

On This Page

Starting a Goroutine

The Goroutine Stack

Goroutines and Operating System Threads

When main Returns

Loop Variables in Goroutines

Using This in an Interview

Key Takeaways