A team wants to halve a model's memory by storing its numbers in 16 bits instead of 32. Two 16-bit formats are on the table, and they split their bits differently.
Every format below stores a number as
A value that falls between two storable numbers is rounded to the nearer one. A value bigger than the largest storable number becomes infinity, which is called an overflow.
| format | total bits | exponent bits | allowed | mantissa bits |
|---|---|---|---|---|
| FP32 | 32 | 8 | to | 23 |
| FP16 | 16 | 5 | to | 10 |
| BF16 | 16 | 8 | to | 7 |
| FP8 (E5M2) | 8 | 5 | to | 2 |
Two values from a training run have to be stored:
What does each 16-bit format actually store for these two values?
Select all that apply.