The problem is the methodology they use to hold them out. For a truly independent validation set, they need to hold out the material before augmentation, not after.
If you hold out after augmentation, then you leverage biases from the training regimen already and hence you artificially boost your model's performance. This is not sufficient to demonstrate your model is generalizing properly.
In analogy: instead of taking leaves off of different trees, they are taking leaves from different branches from the same tree.
Yes, but due to it being derived from the same underlying source dataset, it is effectively evaluating on the training dataset, not an independent validation/ test dataset.
The difference is subtle but important. If we expect the model to truly outperform a general model, it should generalize to a completely independent set.
Yes, this is the main concern I have with this result as well.
In other words, rather than plucking different leaves (augments) from the same branch or tree (source dataset), you should be evaluating it on an entirely different tree.
This paper in essence does not have a validation dataset, it only has a training dataset and evaluates on a subpopulation (even though that population was never trained on)
In analogy: instead of taking leaves off of different trees, they are taking leaves from different branches from the same tree.