I've done similar things and used GitHub Copilot to scan a folder of 40 CVs and rate them -> I then review the top 10 CVs and comment on every rating whether I agree or disagree and why -> I then asked AI to re-rate all the CVs according to my comments. -> I then reviewed all the CVs against their ratings; the AI did a much better job for that 2nd round after its learnings.
It took more time than if I just reviewed the 40 CVs myself, but that was an experiment, and I think it shows the AIs can be trained on your comments. And if there is enough training and a good knowledge system that allows AI to apply the learning in those trainings, it can eventually become a lot more accurate at this task?