Two Penalties, One Gap

A team is regularizing a linear model with five features. The features happen to be orthonormal — mutually perpendicular, each of length 1 — and for that special case both standard regularizers have an exact closed form, so no iterative solver is needed.

Write $z_j$ for the unpenalized (ordinary least squares) coefficient of feature $j$. Then for penalty strength $\lambda$:

• Ridge, which penalizes $\lambda \lVert w \rVert_2^2$: $w_j^{\text{ridge}} = \frac{z_j}{1 + \lambda}$

• Lasso, which penalizes $\lambda \lVert w \rVert_1$: $w_j^{\text{lasso}} = \mathrm{sign}(z_j) \cdot \max\big(\lvert z_j \rvert - \lambda,\; 0\big)$

The five OLS coefficients are:

The team is not given $\lambda$. Instead they have a sparsity budget: the deployed model must keep exactly two non-zero coefficients, and they use the smallest whole number $\lambda$ for which the lasso solution achieves that. The very same $\lambda$ is then plugged into the ridge formula, so that the two methods can be compared fairly.

How far apart are the two resulting weight vectors? Compute the Euclidean distance between them:

$\big\lVert\, w^{\text{ridge}} - w^{\text{lasso}} \,\big\rVert_2$

Give your answer rounded to two decimal places — for example 1.23.