Two Ways to Ask for a Better Fake

The generator never sees a real example. Everything it learns arrives as a derivative handed back through the discriminator — move your output this way and I would have been more convinced. So the size of that derivative is the size of the lesson.

Take the smallest possible judge. The generator emits one number $x$ per fake; the discriminator's head turns it into a logit and squashes it:

$z = w x + b, \qquad s = \sigma(z) = \frac{1}{1 + e^{-z}}$

$s$ near $1$ means "real", near $0$ means "fake". There are two ways to write down what the generator wants, and they are not the same function:

Both are driven down by fooling the judge. What differs is the slope on the way there.

Task: write generator_feedback(outputs, weight, bias), where outputs is the list of numbers the generator produced for a batch of fakes and weight, bias are the head's $w$ and $b$. Return four numbers, each rounded to 4 decimal places:

1. the mean value of the original objective across the batch,
2. the mean value of the non-saturating objective,
3. the mean magnitude of the derivative of the original objective with respect to $x$,
4. the mean magnitude of the derivative of the non-saturating objective with respect to $x$.

• Differentiate with respect to $x$ — the number the generator actually controls — not with respect to $s$ or $z$.
• "Magnitude" means absolute value, taken per sample and then averaged.
• Logarithms are natural. outputs is never empty, and weight may be negative.

A large loss and a usable gradient are not the same thing. One of these two forms can report a batch as almost nothing to fix at the precise moment the discriminator is rejecting every single sample in it.