Value Iteration Step

Value iteration answers "how good is it to be here?" by repeatedly applying one update until the numbers stop moving. This is a single sweep of that update.

For every state, look at each available action, work out its expected value, and keep the best:

Task: write value_iteration_step(transitions, values, gamma) returning the updated values, rounded to 4 decimal places.

transitions[s] is a list of actions, and each action is a list of possible outcomes [prob, next_state, reward].

• A state with no actions is terminal — keep its current value unchanged.
• Compute every new value from the old values list. Don't let an update you just made feed into the next state's calculation.

The max is what separates this from policy evaluation. Evaluation averages over what a fixed policy would do; iteration assumes you'll always pick the best action, which is why it converges to the optimal values without ever being told a policy.