Nudge It Both Ways

You have just written the backward pass for a small regression network, and you want to check it before you trust it. The network takes two inputs, has two ReLU hidden neurons, and has one output neuron with no activation:

$h_1 = \text{ReLU}(w_{11}x_1 + w_{12}x_2 + b_1), \qquad h_2 = \text{ReLU}(w_{21}x_1 + w_{22}x_2 + b_2)$

$\hat y = v_1 h_1 + v_2 h_2 + c, \qquad L = (\hat y - y)^2$

Its current parameters are:

$w_{11}$	$w_{12}$	$b_1$	$w_{21}$	$w_{22}$	$b_2$	$v_1$	$v_2$	$c$
$0.5$	$0.2$	$-0.87$	$-0.25$	$0.5$	$0.25$	$2$	$1$	$-0.5$

The training example is $x_1 = 1$ , $x_2 = 2$ , with target $y = 1.5$ .

Your backward pass reports $\dfrac{\partial L}{\partial w_{11}} = -3.76$ .

To check it you use the lesson's nudge-and-compare, nudging in both directions. Run the forward pass once with $w_{11}$ raised by $h$ and once with it lowered by $h$ , keeping every other parameter fixed, and compute the central-difference estimate

$\frac{L(w_{11} + h) - L(w_{11} - h)}{2h}$

You pick $h = 0.05$ so the arithmetic can be done by hand.

What value does this check produce?

Round to 3 decimal places.

Nudge It Both Ways

Hints