A Graph That Differentiates Itself

In this chapter you walked the graph for $L = (a \cdot b + c)^2$ backwards by hand: start at the output with $1$ , multiply by each box's local derivative on the way back, and add wherever one value feeds more than one box. A machine learning library performs that same walk automatically, on whatever graph it recorded. Here you build the walker.

A graph arrives as a list of operation nodes. Each node is a tuple (name, op, args), and every name inside args refers either to an input or to another node:

`op`	`args`	value of the node
`"add"`	`[x, y]`	$x + y$
`"mul"`	`[x, y]`	$x \cdot y$
`"pow"`	`[x, k]`	$x^k$ , where `k` is a whole number written into the node, not the name of a value
`"tanh"`	`[x]`	$\tanh(x)$ , available as `math.tanh`

The chapter already gave you the local derivatives for adding, multiplying and raising to a power. The new box is $\tanh$ : if $t = \tanh(x)$ , then

\frac{dt}{dx} = 1 - t^2

so, like the squaring box, it needs only the value saved during the forward pass.

Task: write backward(inputs, nodes, output).

inputs is a dict mapping each input name to its number.
nodes is the list of operation nodes. It is not necessarily listed in an order in which the nodes could be computed.
output is the name of the value being differentiated. It is usually the last node to be computed, but not always.

Return a dict mapping every input name to $\partial(\text{output}) / \partial(\text{input})$ , with each value rounded to 4 decimal places. An input with no route to output gets 0.0.

A value may be used by several nodes, and one node may even use the same value for both of its arguments.

The chapter's own graph, listed out of order and run with $a = 3$ , $b = -1$ , $c = 4$ :

A Graph That Differentiates Itself

Examples

Hints