My team has been trying to modernize pandas from a different tact. Regardless of struggle with the syntax, it seems Pandas is very sticky, and we don't predict much migration to other data science languages. Instead of refining the syntax, we have combined it with a spreadsheet GUI (https://github.com/mito-ds/monorepo). Here, we worry less about writing perfect syntax ourselves, and let the GUI write the code for functions like pivot tables and merges that work well visually.
I think this is spot on. I agree that Excel users want to stick with Excel, but they do run into major issues that are solved by code. Namely: their data size is too large, Excel is too slow, and they struggle to get repeatability from their work.
I am building Mito[1], a spreadsheet interface for Python. Every edit you make in the spreadsheet generates the equivalent Python. It is a bridge between the workflows of Excel users and Python users, and allows Excel users to reap Python's benefits without needing to know how to code.
Spreadsheets versus programmers is a war that can be much more peaceful. In my work, I have found that, especially in data science, the spreadsheet user and the programmers are often trying to accomplish similar tasks, but the “language barrier” between them leads to much more fragmented workflows. I also think this article does a good job of identifying spreadsheets as a low-code programming language — spreadsheets are immensely powerful pieces of software. Along these lines — I’ve been building Mito (https://trymito.io/), a spreadsheet GUI for Python. Every edit you make in the spreadsheet generates the equivalent Python code.
I have been working on this problem for a little while. I don't think the issue is that Excel is inherently worse for data wrangling -- it just isn't scalable or easily traceable. My solution is a spreadsheet extension for Jupyter Notebooks, that lets you transform your data visually and outputs the equivalent Pandas.
Automated data prep/transformation is definitely a useful solution, and obviously is going to be a part of most data wrangling tools in the future. I think the danger is a lack of visibility and tracing, something a combo between spreadsheets and pandas provides well -- I think :)
I used to work for a company making a bet against live collaboration in spreadsheets. I now see the error in my ways. A spreadsheet party in Excel would just be a nightclub that one person can enter at a time.