0% completed
Neural Networks in One Lesson
On This Page
Part 1: Numbers In, Numbers Out
Part 2: One Neuron, Step by Step
Part 3: Activations, the Step That Makes Depth Work
Part 4: Layers, From Simple Patterns to Complex Ones
Part 5: A Full Pass Through a Tiny Network
Part 6: Where the Weights Come From
Part 7: What "70B Parameters" Actually Means
Part 8: What This Changes for an Engineer
Common Mistakes
Key Takeaways
The label "70B" on a model means that it has about 70 billion learned numbers. If each number is stored in 16 bits, it takes 2 bytes. So the model needs about 140 GB of memory before it reads a single word. One large data center GPU often has about 80 GB, so this model cannot even load on a single one.
Labels like "70B" appear on the download pages of open-weight models. An open-weight model is one whose learned numbers are published so that anyone can download and run it. A download page often offers several sizes, such as "8B" and "70B." The B stands for billion, and the number counts the model's parameters, the values it learns in training.
Every large language model is a kind of neural network. This lesson looks inside a small network, using the spam filter from earlier lessons as the example. It then returns to "70B" and turns that number into memory, cost, and speed.
Part 1: Numbers In, Numbers Out
From the outside, a neural network is a function that takes a list of numbers in and gives a list of numbers out. So everything a network works on must first become numbers. An image becomes a list of numbers representing the brightness of each pixel. Text also becomes numbers through steps that start in the lesson Tokens and Tokenization. For the spam filter, an email becomes a few numbers, such as how many links it contains.
Inside, the network is built from many small math units called neurons, arranged in layers. The delivery model in the previous lesson had one parameter. A neural network has many parameters, two kinds: weights and biases. People choose the network's shape, and training chooses the numbers inside it.
Part 2: One Neuron, Step by Step
A neuron is the smallest working part of a network. It does three things, in order.
- It weighs each input. A weight is a learned number that says how much one input matters, and in which direction. Each input is multiplied by its own weight. A positive weight pushes the result up, and a negative weight pushes it down.
- It adds the results, plus a bias. A bias is a learned number added to the total. It sets the neuron's starting point before any inputs are counted.
- It applies an activation. An activation is a simple fixed rule that turns the total into the neuron's output. Part 3 explains why this step matters.
Here is a worked example. All numbers in this lesson are made up for illustration. The email has three inputs:
- It contains 3 links.
- It contains the word "winner," written as 1 for yes.
- The sender is not in your contacts, written as 0 for no.
The diagram below follows these numbers through one neuron.
The neuron's weights are 0.5 for links, 2.0 for "winner," and -1.0 for a known sender. Three links times 0.5 gives 1.5. The "winner" input times 2.0 gives 2.0. The sender input is 0, so it adds nothing. The sum is 3.5, and the -1.0 bias brings the total to 2.5. The activation keeps a positive total unchanged, so the neuron outputs 2.5.
Part 3: Activations, the Step That Makes Depth Work
A common activation is called ReLU, short for "rectified linear unit." Its rule is simple. A positive total passes through unchanged, and a negative total becomes zero. So 2.5 stays 2.5, and -1.1 becomes 0. Large language models usually use smoother versions of the same idea.
Why does a network need this step? Without activations, every neuron only multiplies and adds. Any stack of layers that only multiply and add can be rewritten as one single layer that does the same work. So a hundred such layers could do no more than one.
An activation is not a multiply-and-add step, so the layers can no longer be merged into one. This lets a stack of layers learn patterns that one layer cannot. For example, the word "invoice" might be a good sign in an email from a known supplier, but a warning sign in an email from a stranger.
Part 4: Layers, From Simple Patterns to Complex Ones
A layer is a group of neurons that read the same inputs and work side by side. The outputs of one layer become the inputs of the next. A typical network has three kinds of layers:
- The input layer holds the numbers that go in. It does no math of its own.
- Hidden layers sit in the middle. They are called hidden because their outputs are used only inside the network.
- The output layer produces the final result, such as a spam score.
A network works much like a team in a mail room that checks letters for scams. Each part of the team matches a part of the network:
- Each clerk is a neuron.
- A row of clerks who all read the same material is a layer. The first row reads the letter, and each later row reads only the notes from the row before it.
- How much a clerk counts each note from the row before is a weight. Some notes count a lot, some count a little, and some count against a scam verdict.
- A clerk who writes nothing unless the evidence adds up to more than zero follows the ReLU activation.
- The head of the mail room, who gives the final verdict, is the output layer.
A network with many hidden layers is called deep, which is where the name deep learning comes from. In image networks, researchers have found that early layers tend to respond to simple patterns, such as edges. Later layers combine these into shapes, and then into whole objects. Nobody programs these patterns. They emerge from training, and they are rarely as easy to name as the clerks' jobs.
Part 5: A Full Pass Through a Tiny Network
The diagram below shows a complete network with three layers: three inputs, two hidden neurons, and one output neuron. Every arrow carries a weight.
Hidden neuron 1 is the neuron from Part 2, so it outputs 2.5.
Hidden neuron 2 has weights of -0.2, -0.5, and 3.0, and a bias of 0. Its total is -1.1, and ReLU turns that into 0. So this neuron outputs nothing for this email.
The output neuron gives hidden neuron 1 a weight of 1.2 and hidden neuron 2 a weight of -2.0, and its bias is -1.0. Its total is 3.0, plus 0, minus 1.0, which is 2.0. The output neuron uses a different activation, called a sigmoid, which turns any total into a score between 0 and 1. A total of 2.0 becomes about 0.88, so the network rates this email as likely spam.
Running inputs through a network from start to end like this is called a forward pass. Every answer a model gives comes from forward passes.
Now count the learned numbers. There are 6 weights between the input and hidden layers, and 2 more between the hidden and output layers. Each of the 3 neurons also has one bias. That makes 11 parameters in total.
The playground below runs this forward pass with the same numbers as the diagram. It uses only the Python standard library. For each neuron, it prints the weighted inputs, the bias, the total, and the output. It then counts the parameters and calculates the memory the 8B and 70B models need for their weights at 16 bits and 4 bits. Parts 7 and 8 explain these memory figures.
Press Run, and follow the totals: 2.5 for hidden neuron 1, -1.1 for hidden neuron 2, and 2.0 for the output neuron. The spam score is 0.88. The count is 8 weights plus 3 biases, or 11 parameters, as above. The memory table shows 16 GB and 4 GB for the 8B model, and 140 GB and 35 GB for the 70B model.
Now change KNOWN_SENDER to 1 and run it again. Hidden neuron 2 now outputs 1.9. The output neuron weights that value at -2.0, so the total output falls to -3.0, and the spam score drops to 0.05. To see what ReLU does, set USE_RELU to False instead. Hidden neuron 2 then passes -1.1 unchanged, and the score rises to 0.99.
Part 6: Where the Weights Come From
Nobody sets these numbers by hand. Training starts with random weights and biases, and then it runs the loop from How a Model Learns. The network predicts, the loss measures the error, and every parameter moves a small step to reduce it. Over millions of examples, the parameters settle into useful values.
This has two results. First, a network's learning is spread across many weights at once. No single weight holds a fact such as "this sender is safe." So you cannot fix one wrong answer by editing one number.
Second, the weights are the model. An open-weight model file is mostly a very long list of these numbers, plus a description of the network's shape.
Part 7: What "70B Parameters" Actually Means
Now return to the download page. "70B" means that the model has about 70 billion parameters. They are the same kind of weights and biases as the 11 in the small spam network.
A large language model is arranged differently from the small network, as the lesson The Transformer at Box Level shows. But most of its parameters are still weights in layers of weighted sums, stacked dozens of layers deep.
Every parameter must be in memory while the model runs, usually GPU memory. In a common 16-bit format, each parameter takes 2 bytes. So 70 billion parameters need about 140 GB just for the weights. A large data center GPU often has about 80 GB of memory, so this model needs at least two. The 8B model needs about 16 GB, which fits on one large GPU.
Size also affects speed and cost. In a standard model, every piece of text the model writes runs through all of its weights. So a model with ten times more parameters does roughly ten times more math for each piece. Some models, called mixture-of-experts models, use only part of their weights for each piece.
Part 8: What This Changes for an Engineer
Do the memory math first. Multiply the parameter count by the bytes per parameter. Teams often reduce this number with quantization, which stores each parameter in fewer bits, such as 4 instead of 16. That cuts the memory to about a quarter, often with a small loss of quality. The later lesson "Small, Local and Open Models" covers this trade-off.
The table below gives rough figures for the weights alone. Running a model also requires extra memory for the requests it handles.
| Model size | Weights at 16 bits | Weights at 4 bits |
|---|---|---|
| 8 billion parameters | About 16 GB | About 4 GB |
| 70 billion parameters | About 140 GB | About 35 GB |
Treat parameter count as a size, not a score. Larger models are often more capable, but a smaller model can do well on a narrow task at a lower cost. Test the candidates on your own task before you choose.
Add knowledge through the input or more training. What a model learned is spread across billions of weights, so you cannot simply edit one fact. To give a model new facts, put them in the request, or train the model further. A later lesson, Prompting vs RAG vs Fine-Tuning, compares these options.
Common Mistakes
Mistake 1: Reading parameter count as a quality score. More parameters often help, but training data and design matter too. Measure models on your own task.
Mistake 2: Choosing a model before checking that it fits. At 16 bits, every billion parameters needs about 2 GB of memory. Check your hardware before you plan around a model.
Mistake 3: Picturing neurons as brain cells, or weights as stored facts. Artificial neurons were loosely inspired by brain cells, but each one is only a weighted sum and a simple rule. Knowledge is spread across many weights, not kept in one place.
Key Takeaways
- A neuron is a weighted sum plus an activation. It multiplies inputs by weights, adds a bias, and applies a simple rule.
- Activations let layers stack. Without them, many layers could do no more than one.
- Parameters are the learned numbers. Training sets every weight and bias, and what the model learns is spread across many of them.
- Parameter count sets memory, speed, and cost. At 16 bits, each parameter takes 2 bytes, so a 70B model needs about 140 GB.
The next lesson, Training vs Inference, separates the two phases of a model's working life. Training sets the parameters once, at great cost. Inference uses them on every request, at a small cost each time.
Reading Progress
0%
On This Page
Part 1: Numbers In, Numbers Out
Part 2: One Neuron, Step by Step
Part 3: Activations, the Step That Makes Depth Work
Part 4: Layers, From Simple Patterns to Complex Ones
Part 5: A Full Pass Through a Tiny Network
Part 6: Where the Weights Come From
Part 7: What "70B Parameters" Actually Means
Part 8: What This Changes for an Engineer
Common Mistakes
Key Takeaways