The App That Reverses Itself

Both groups contain exactly 400 students, so their composition is balanced and the 20-percentage-point overall gap is a fair estimate of the app's causal effect.

The app group did better among strong-prior students but worse among weak-prior students, so the app helps only well-prepared learners.

Overall the app group improved more often (77.5% vs 57.5%), yet within both prior-level groups the app group improved less often. The overall gap comes from the app group holding three times as many strong-prior students.

The app group improved more often overall (77.5% vs 57.5%) and also within each prior level, so every way of slicing the data favours the app.