I agree this is not how you should evaluate. But it is not how things are done in (correctly executed) ML. What is described is a validation set, not a test set. The test set should not be involved in training or(hyper)parameter optimization at any time.