Three Losses, Two Misses

Newton's method swaps the learning rate for $1/f''$, so it cares about one property of a loss more than any other: its curvature, the second derivative. A team wants to compare how sharply three common losses bend when a prediction misses by a little and when it misses by a lot.

Two regression losses, written as functions of the error $r = \hat{y} - y$ on a single example:

• Squared error: $L(r) = r^2$
• Huber loss with $\delta = 1$:

$$L(r) = \begin{cases} \tfrac{1}{2}r^2 & \text{if } |r| \le 1 \\ |r| - \tfrac{1}{2} & \text{if } |r| > 1 \end{cases}$$

One classification loss. A yes/no classifier looks at an example whose true answer is yes. It emits a raw score $z$, turns it into a probability with the sigmoid $p = \sigma(z) = \dfrac{1}{1 + e^{-z}}$, and pays the cross-entropy loss $-\ln p$. The model's parameters set $z$ directly, so treat this loss as a function of $z$:

$$L(z) = -\ln \sigma(z)$$

The two situations to compare:

Take each loss's second derivative (with respect to $r$ for the two regression losses, and with respect to $z$ for cross-entropy) at the small miss and at the big miss.

Which statement is correct? Values are rounded to two significant figures.