Now that such a model exists, I think we can expect lobbying for making this mandatory. For the children...
Once adopted this will lead to an increase in randomly locked out accounts due to model false positives, because looking for something very rare will lead to most items flagged being false positives (Bayes statistics etc.).
This blog had a few interesting articles on the limitations of these technologies
https://www.hackerfactor.com/blog/index.php?/archives/971-FB...
While this is comparing the currently used hash based approaches a classification model will have similar problems.
During the early career of Ursula von der Leyen (current EU president) in Germany she was championing a law just like that. It was something like "stop signs" for the internet and was based on DNS based blocks. The law was passed and was quickly overturned once her party won the election. During the campaign she and her party were denouncing everybody opposing the law as being friendly to csam. This feels like a bad re-run of the same script to me.
Recorded talk for the slides in this post.