Through the Narrow Middle

A small autoencoder works on greyscale strips: each strip is a list of $n$ pixel brightnesses between 0 and 1. The encoder squeezes the strip down to a code of $k$ numbers, the decoder rebuilds the strip from that code alone, and the reconstruction loss scores how good the copy is.

Encoder: one layer, ReLU. enc_w is a list of $k$ rows, one per code number, and each row holds $n$ weights, one per pixel. enc_b holds $k$ biases. Code number $j$ is

c_j = \max\Big(0,\ \sum_{i} W_{ji}\, x_i + b_j\Big)

where $W_{ji}$ is enc_w[j][i] and $b_j$ is enc_b[j].

Decoder: one layer, sigmoid. dec_w is a list of $n$ rows, one per rebuilt pixel, and each row holds $k$ weights, one per code number. dec_b holds $n$ biases. Rebuilt pixel $i$ is

\hat{x}_i = \sigma\Big(\sum_{j} V_{ij}\, c_j + d_i\Big), \qquad \sigma(z) = \frac{1}{1 + e^{-z}}

where $V_{ij}$ is dec_w[i][j] and $d_i$ is dec_b[i].

Reconstruction loss. Subtract, square, then average over every pixel:

L = \frac{1}{n}\sum_{i} (\hat{x}_i - x_i)^2

Task: write autoencode(x, enc_w, enc_b, dec_w, dec_b) and return a tuple (code, rebuilt, loss):

code: the list of $k$ code numbers,
rebuilt: the list of $n$ rebuilt pixels,
loss: a single number.

Round every returned number to 4 decimal places. Round only at the end: compute the loss from the unrounded rebuilt pixels.

Notice what the loss compares against. There is no label anywhere: the target is the input itself.

Through the Narrow Middle

Examples

Hints