The Transformer at Box Level

{
  "hidden_size": 4096,
  "num_hidden_layers": 32,
  "vocab_size": 128000
}

These lines come from the configuration file of a small open-weight model. The field names are the ones real files use, but the values here are illustrative. The model card, the short document that describes the model, adds two more facts. The model is a decoder-only transformer, and it has 8 billion parameters.