Shuffle one column of a loan table and score the same model again. The bigger the drop in accuracy, the more the model relies on that column.
The trained model gets 10 of 11 held-out applicants right. Only Fay is missed. Shuffle one column, then score the same model again.
