Bottom to Top, Every Weight

A decoder-only language model is assembled exactly as the chapter lays it out: token embeddings, plus position, then $N$ identical blocks, then a readout.

part	size
vocabulary	$32{,}000$ tokens
width $d$	$768$
blocks	$12$
position vectors	learned, one vector of width $768$ for each of $1{,}024$ positions
readout	its own matrix of shape $768 \times 32{,}000$ , turning the top vector into one score per vocabulary token

Inside every block, the learned weights are:

four attention projection matrices — queries, keys, values, and the output projection — each $768 \times 768$ ;
two feed-forward matrices — one $768 \times 3072$ that widens to $4d$ , one $3072 \times 768$ that squeezes back.

Ignore biases, and ignore the layer normalizations' learned scales and shifts.

How many parameters does the whole model hold?

Give your answer in millions, rounded to 2 decimal places — a model of $50{,}000{,}000$ parameters would be $50.00$ .

Bottom to Top, Every Weight

Hints