Backpropagation answers one question in a single backward walk: how does the output respond to every dial? There is a mirror-image method that walks forward instead, and it answers a different question in one pass: how does every box respond to one dial?
The trick is to carry two numbers through the forward pass instead of one. Every value becomes a pair
where the slope is the derivative of that value with respect to the one dial you picked. A pair like this is called a dual number. The pass starts with the dial you picked at slope 1 (it changes exactly as fast as itself), and every other input at slope 0, as is every plain constant.
Each box then turns its incoming pairs and into an outgoing pair, using rules you already know: the sum, product and quotient rules, and the power rule with the chain rule.
op | value | slope |
|---|---|---|
"add" | ||
"sub" | ||
"mul" | ||
"div" | ||
"pow" |
Because the slopes ride along with the values, nothing has to be stored for a walk back. The price is that one pass covers only one dial, so a full gradient needs one forward pass per dial, where backpropagation needs a single backward walk. In exchange, one forward pass reports the slope at every box.
Task: write dual_sweep(inputs, steps, dial).
inputs is a dict mapping each input name to its number.steps lists the graph's boxes in an order in which they can be computed, as tuples (name, op, args). args holds two entries, each either the name of an input or an earlier box, or a plain number (a constant). For "pow", the second entry is always a whole-number exponent written into the box.dial is the name of the input you are differentiating with respect to.Return a dict mapping every box's name (not the inputs) to the pair (value, slope), both rounded to 4 decimal places.
That is the chapter's own graph. The last slope, 42, is , the number the backward walk found, and on the way the same pass also reports and .