Why can't we automate data cleaning?
4 comments
We have, there are quite a few data wrangling, or 'data munging' tools out there.
On a side note: Does "data wrangling" and (especially) "data munging" sound cringey to anyone else? It's hard to put my finger on why exactly but the terms are really offputting.
It’s clear that data cleaning, like modelling, is not immune to automation. As a result, it’s likely that data scientists will find themselves leaning more and more into their subject matter expertise, communication and engineering skills in the future, rather than spending their time on dealing with missing values, hyperparameter optimization or model selection.
What we can't do is create a generalised function called clean(data) that can be applied to any dataset. The reason is that your original format, data source, goals, domain knowledge, personal judgement vary so much that it makes it pointless to even try.