Sixteen Runs to Spend

A team has budget for exactly 16 full training runs. They want to tune two settings, the learning rate and the dropout rate, and they are choosing between two plans.

Plan 1: grid search. One run for every combination of

• learning rate $\in \{0.0001,\ 0.001,\ 0.01,\ 0.1\}$
• dropout rate $\in \{0.2,\ 0.3,\ 0.4,\ 0.5\}$

which is $4 \times 4 = 16$ runs.

Plan 2: random search. 16 runs, each choosing its settings independently of the others:

• draw a number $u$ uniformly at random between $-4$ and $-1$, and use learning rate $10^{u}$. Learning rates therefore range from $0.0001$ to $0.1$, spread evenly across the powers of ten;
• draw a dropout rate uniformly at random between $0.2$ and $0.5$.

Unknown to the team, this model only trains well when the learning rate lies between $10^{-2.8}$ and $10^{-2.2}$ (about $0.0016$ to $0.0063$). Anywhere from $0.2$ to $0.5$, the dropout rate makes almost no difference.

For each plan, what is the probability that at least one of its 16 runs uses a learning rate inside that good band?