The Rerun With a Bigger Step

The small learning rate was the whole problem, and the larger one fixed it: training loss now drops steeply and then flattens, the healthy shape. The validation rise is ordinary overfitting, so ship the saved copy with the lowest validation loss.

Run 2 rules out bad labels: training loss could only fall from $1.39$ to $0.02$ if the labels carried a real pattern. Validation always trails training, so the fix is simply more steps at this learning rate.

Nothing learned carries over to new photos: $1.39$ is $\ln 4$, the score of a network that ignores the photo and gives each species $25\%$, and validation never beats it at any saved step. Training loss most likely falls through memorising photos, so first check that the labels match them.

The new learning rate overshoots: a loss that climbs at every saved step, as validation does here, means each step lands higher up the far side of the valley. A learning rate between the two should work.