Pandas AI(medium.com)
medium.com
Pandas AI
https://medium.com/@fareedkhandev/pandas-ai-the-future-of-data-analysis-8f0be9b5ab6f
58 comments
Hey SilverBirch, I'm the author of PandasAI. In terms of the naming, that's a very good point, and I must confess I didn't even consider that when I published the repo. It just made sense to me, since the goal was to add AI on the top of pandas, to call it "PandasAI" (pandas + AI). For it was more of a tribute to the library, but I totally get your point, and I thank you for pointing it out.
I'm trying to reach out the pandas founder, and I'll definitely change the name of the repo if he prefers to.
As of the quality of the output, we are constantly working on improving the quality of it. It provides good enough results in most of the cases, but it still suffers from some hallucinations, although we have already fixed many either programmatically or with prompt engineering.
As of the dataframe, we don't pass the whole df, but only the head and 5 columns with shuffled data (also redacting the sensitive ones).
Any questions, feel free to ask!
I'm trying to reach out the pandas founder, and I'll definitely change the name of the repo if he prefers to.
As of the quality of the output, we are constantly working on improving the quality of it. It provides good enough results in most of the cases, but it still suffers from some hallucinations, although we have already fixed many either programmatically or with prompt engineering.
As of the dataframe, we don't pass the whole df, but only the head and 5 columns with shuffled data (also redacting the sensitive ones).
Any questions, feel free to ask!
It is bullshit to call this pandas AI.
I use pandas all the time and could give quite a bit of feedback but not when you are basically trying to conflate and leverage the hard work of another project for yourself.
Not a big deal though because no one is going to use this tool as is.
I use pandas all the time and could give quite a bit of feedback but not when you are basically trying to conflate and leverage the hard work of another project for yourself.
Not a big deal though because no one is going to use this tool as is.
I think the biggest area for growth for LLM based tools for data analysis is around helping users _understand what edits they actually made_.
I'm a co-founder of a non-AI data code-gen tool for data analysis -- but we also have a basic version of an LLM integration. The problem we see with tooling like Pandas AI (in practice! with real users at enterprises!) is that users make an edit like "remove NaN values" and then get a new dataframe -- but they have no way of checking if the edited dataframe is actually what they want. Maybe the LLM removed NaN values. Maybe it just deleted some random rows!
The key here: how can users build an understanding of how their data changed, and confirm that the changes made by the LLM are the changes they wanted. In other words, recon!
We've been experimenting more with this recon step in the AI flow (you can see the final PR here: https://github.com/mito-ds/monorepo/pull/751). It takes a similar approach to the top comment (passing a subset of the data to the LLM), and then really focuses in the UI around "what changes were made." There's a lot of opportunity for growth here, I think!
Any/all feedback appreciated :)
EDIT: Also, shout out to JupyterAI (https://github.com/jupyterlab/jupyter-ai) -- it's an official Jupyter project with some really awesome LLM support directly in JupyterLab. I saw it debuted at JupyterCon last week :)
I'm a co-founder of a non-AI data code-gen tool for data analysis -- but we also have a basic version of an LLM integration. The problem we see with tooling like Pandas AI (in practice! with real users at enterprises!) is that users make an edit like "remove NaN values" and then get a new dataframe -- but they have no way of checking if the edited dataframe is actually what they want. Maybe the LLM removed NaN values. Maybe it just deleted some random rows!
The key here: how can users build an understanding of how their data changed, and confirm that the changes made by the LLM are the changes they wanted. In other words, recon!
We've been experimenting more with this recon step in the AI flow (you can see the final PR here: https://github.com/mito-ds/monorepo/pull/751). It takes a similar approach to the top comment (passing a subset of the data to the LLM), and then really focuses in the UI around "what changes were made." There's a lot of opportunity for growth here, I think!
Any/all feedback appreciated :)
EDIT: Also, shout out to JupyterAI (https://github.com/jupyterlab/jupyter-ai) -- it's an official Jupyter project with some really awesome LLM support directly in JupyterLab. I saw it debuted at JupyterCon last week :)
Hey narush, I'm the author of PandasAI. The way PandasAI works is slightly different. It's not the LLM that edits data, but a python script. This is in my opinion the best approach, as it gives a predictable result.
There are still some potential issues, including hallucinations, that we are taking care of, but all in all we hope that this could represent an advancement in the way we interact with data.
Any questions, feel free to ask :)
Any questions, feel free to ask :)
I’m laughing at the part where they say they’re going to do a complex task and the chosen example is “remove null values”.
df = df.dropna()
Wow
Edit: more seriously though… this is a non deterministic pipeline that seems to mutate your data in place with no indication of what it did.
df = df.dropna()
Wow
Edit: more seriously though… this is a non deterministic pipeline that seems to mutate your data in place with no indication of what it did.
> non deterministic pipeline that seems to mutate your data in place with no indication of what it did
/dev/random has been around for a while, i wonder why they didn't just use that. ;)
/dev/random has been around for a while, i wonder why they didn't just use that. ;)
When the package returns that the AI couldn't do the task but the data is modified anyway... that is a recipe for disaster.
It would be much better if the AI just helped you write the code, it's more ergonomic and auditable, and it will scale better when your data frame is measured in the 10s to 10,000s of megabytes.
It would be much better if the AI just helped you write the code, it's more ergonomic and auditable, and it will scale better when your data frame is measured in the 10s to 10,000s of megabytes.
When I saw the NaNs, my first thought was the example was going to be predict the NaN values, not drop them.
And shouldn't it have just used fillna to boot?
I think that ambiguity is part of the charm of natural language instructions
AI enhanced data analysis is nice, but this is not how I would have implemented it. Passing the data frame as part of the query limits the quantity of data you can handle, and you’re at risk of the quiet adding undesired operations on your data.
I expected that this would be a Copilot like approach that was tuned for pandas specifically, and you pass it your data frame schema/column names or something, and it passing back some code to evaluate locally.
I’m definitely not excited about passing the data frame itself and letting the llm handle it. Maybe with time I’ll get there, but I think this is a step too far for anyone that needs reproducible/verifiable results.
I expected that this would be a Copilot like approach that was tuned for pandas specifically, and you pass it your data frame schema/column names or something, and it passing back some code to evaluate locally.
I’m definitely not excited about passing the data frame itself and letting the llm handle it. Maybe with time I’ll get there, but I think this is a step too far for anyone that needs reproducible/verifiable results.
I understand what you mean and kinda agree.
I think that you entered in a "niche" discussion now. Because I see that people are building tools with LLM in the data analysis space trying to go more in the business direction, where product managers (and other stakeholders, mainly in startups) don't need to write a single line of code and get the data visualization/analysis that they want.
I think that you entered in a "niche" discussion now. Because I see that people are building tools with LLM in the data analysis space trying to go more in the business direction, where product managers (and other stakeholders, mainly in startups) don't need to write a single line of code and get the data visualization/analysis that they want.
There is a package for querying dataframes with SQL: https://github.com/yhat/pandasql/. I assumed this was going to be LLM -> SQL -> Pandas.
That (LLM -> SQL -> Pandas using PandaSQL) was my approach with Datasloth [1] some time ago. It then grew into Qluent [2] which I now work on full time, along the lines of what the sibling comment wrote about managers being able to analyze their data without coding.
[1] https://github.com/ibestvina/datasloth
[2] https://qluent.com/
[1] https://github.com/ibestvina/datasloth
[2] https://qluent.com/
While interesting, the fact that it doesn't output code is worrisome. This because the function is no longer idempotent, each call to the AI can yield a different result.
Sketch is similar, but can do code generation (with `.sketch.howto`): https://github.com/approximatelabs/sketch
> This because the function is no longer idempotent, each call to the AI can yield a different result.
Also, it makes it harder to verify correctness, and may make processing larger datasets (or repeatedly processing similar datasets) more expensive.
> This because the function is no longer idempotent, each call to the AI can yield a different result.
Also, it makes it harder to verify correctness, and may make processing larger datasets (or repeatedly processing similar datasets) more expensive.
Not a lawyer, but this seems to violate trademark usage (unless there is a written agreement).
"You may not use or register, in whole or in part, our Marks as part of your own trademark, service mark, domain name, company name, trade name, product name or service name.
You may not use names or trademarks that are similar to ours without first obtaining written approval. You therefore may not use an obvious variation of any of our Marks or any phonetic equivalent, foreign language equivalent, takeoff, or abbreviation for a similar or compatible product or service."
(Numfocus is the legal entity behind pandas and other open source projects): https://numfocus.org/trademark-guidelines
"You may not use or register, in whole or in part, our Marks as part of your own trademark, service mark, domain name, company name, trade name, product name or service name.
You may not use names or trademarks that are similar to ours without first obtaining written approval. You therefore may not use an obvious variation of any of our Marks or any phonetic equivalent, foreign language equivalent, takeoff, or abbreviation for a similar or compatible product or service."
(Numfocus is the legal entity behind pandas and other open source projects): https://numfocus.org/trademark-guidelines
I asked GPT-4 this
can you visit https://pandas.pydata.org/about/governance.html and tell me if I am allowed to use the term 'pandas' in the name of another unaffiliated project, for example 'pandas-ai'
--
Based on the BSD 3-Clause License under which pandas is released, neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission[ 1] . This means that to use the term 'pandas' in the name of another unaffiliated project such as 'pandas-ai', you would likely need to get written permission from the pandas project's copyright holders.
However, please note that this is not legal advice, and it would be a good idea to consult with a lawyer who specializes in open-source software or intellectual property law to ensure that you're in compliance with all legal requirements.
[1] https://github.com/pandas-dev/pandas/blob/main/LICENSE
can you visit https://pandas.pydata.org/about/governance.html and tell me if I am allowed to use the term 'pandas' in the name of another unaffiliated project, for example 'pandas-ai'
--
Based on the BSD 3-Clause License under which pandas is released, neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission[ 1] . This means that to use the term 'pandas' in the name of another unaffiliated project such as 'pandas-ai', you would likely need to get written permission from the pandas project's copyright holders.
However, please note that this is not legal advice, and it would be a good idea to consult with a lawyer who specializes in open-source software or intellectual property law to ensure that you're in compliance with all legal requirements.
[1] https://github.com/pandas-dev/pandas/blob/main/LICENSE
Can HN please ban these low-effort + low-contribution "I asked GPT your question" responses? These add nothing to the conversation, as you have no idea if the response is accurate or a hallucination. HNers should know better.
[deleted]
I agree it violates trademark usage (also not a lawyer). But even if it is legal, it's still unethical, since they explicitly asked projects to not use their names in the doc you linked. And even if you believe it is ethical from some vaguely libertarian "'information wants to be free' means I should be able to do whatever I want online", it is still a bad idea if you actually want to work with the pandas team. If you need some PR to be merged into any numfocus project (numpy, jupyter, dask, among others) that only benefits pandas AI, I think they would be fully justified in letting it sit.
[deleted]
I’ve been using GPT-4 to help with some complex pandas work, and it’s been an excellent assistant.
That said, I don’t get this product. Using query as code is no help at all. Integrating GPT into jupyter might be useful - I’m sure it’s been done - but not this.
That said, I don’t get this product. Using query as code is no help at all. Integrating GPT into jupyter might be useful - I’m sure it’s been done - but not this.
Can you expand? I don't see what's wrong with it: suppose you're experimenting at the REPL, you ask pandas-ai to produce a graph; it seems almost correct/what you wanted so you ask it to emit the code implementing it, and copy that into your own code for modification.
maybe he is using the gpt4 plugin code interpreter
How have you been using it? Helping build more complex pandas code through querying? Its pretty good but it does breakdown on complex specific tasks from my experience.
The examples in this post are too trivial to tell what the capacities of pandas-ai are. If you've got pandas open you'll surely already know how to remove NaNs and create histograms, even multi coloured ones(!).
> When posing a question, it’s crucial to remember that the entire dataframe is passed along with the question every time, so it may not be an ideal solution for handling large datasets.
This really limits this package as I frequently want to perform multiple operations on a data frame. Does OpenAI not allow sessions so the data frame must be passed each call?
This really limits this package as I frequently want to perform multiple operations on a data frame. Does OpenAI not allow sessions so the data frame must be passed each call?
You can't really solve this with sessions. Even when you use chatgpt in the web console the entire history of the chat is submitted each time you generate a new response. The history is the only way these models know what you've been talking about. The history _is_ the session. I suppose there will be ways around this eventually, though. For this particular application it might suffice for slightly large datasets to just send a sample.
I think the approach here is kind of dumb, though. It works significantly better to send the AI meta-data and let it write queries (with a limit on result count) iteratively to answer your questions.
I think the approach here is kind of dumb, though. It works significantly better to send the AI meta-data and let it write queries (with a limit on result count) iteratively to answer your questions.
That's exactly what I was wondering. Why send the entire dataset when you can just send the schema and ask the AI to generate queries and then translate query results back into natural language?
The funny part about this is that PandasAI was apparently able to figure out what he intended to type in making that simple histogram. There must be some autocorrection buried in the pile of algorithms at work there.
His request was for PandasAI to -
"Plot the histogram of countries showing for each the gpd,..."
The histogram was constructed correctly according to what he thought he typed. Scary mind-reading stuff here with that AI stepping in and correcting our defective requests without even asking - " Did you mean to request GDP?"
This AI could serve as the annoying know-it-all at all of your parties, correcting everyone's anecdotes and misconceptions in real time. It only needs a high-pitched nasal voice with a simulated west Texas drawl playing from your Alexa or on-screen as a Poindexter-in-a-Stetson avatar.
His request was for PandasAI to -
"Plot the histogram of countries showing for each the gpd,..."
The histogram was constructed correctly according to what he thought he typed. Scary mind-reading stuff here with that AI stepping in and correcting our defective requests without even asking - " Did you mean to request GDP?"
This AI could serve as the annoying know-it-all at all of your parties, correcting everyone's anecdotes and misconceptions in real time. It only needs a high-pitched nasal voice with a simulated west Texas drawl playing from your Alexa or on-screen as a Poindexter-in-a-Stetson avatar.
If you want to partially automate data analysis tasks and your memory of pandas api is a bit fuzzy, one option is:
(1) run notebook in VS-code https://code.visualstudio.com/docs/datascience/jupyter-noteb...
(2) install Github Copilot extension https://marketplace.visualstudio.com/items?itemName=GitHub.c...
Then if you load in pandas and a DataFrame you want to analyse, it quite often suggests the next stage of your data-analysis work, either right away, as you write the first step in some chain of pandas methods.
(1) run notebook in VS-code https://code.visualstudio.com/docs/datascience/jupyter-noteb...
(2) install Github Copilot extension https://marketplace.visualstudio.com/items?itemName=GitHub.c...
Then if you load in pandas and a DataFrame you want to analyse, it quite often suggests the next stage of your data-analysis work, either right away, as you write the first step in some chain of pandas methods.
I hate pandas. Polars is significantly better designed. Please, everyone, stop using Pandas.
I don't hate pandas but after using R and Dplyr, I didn't want to touch pandas ever again. Now with Polars, I'm enjoying EDAs and replacing pandas on our ML stack has been fun. I could see a case being made for Pandas if they emulate Dplyr.
Any experience with Polars & ArticDB? (I've been wondering how Polars could improve how I use Artic.... 20k+ smaller dataframes updated often from squirts back from IoT devices)
Espouse the benefits of polars vs pandas and ill consider it. At this point i know pandas very well and, while it had some nuances, its pretty good at what it does!
Not the OP but I’ll give my opinion. Ultimately they are similar but polars has definitely benefited from “going second” ie seeing a successful example and how people use it and knocking off some rough edges.
1. Polars has a narrower API that seems a bit cleaner and more logical vs pandas is more maximalist. Matter of taste but https://franzdiebold.github.io/polars-cheat-sheet/Polars_che... Will give you a flavour
2. Polars perf seems quite a lot better. This may not matter for your use case but if it does it does.
3. Polars supports both eager and lazy evaluation modes and you can choose whichever makes sense for your use case. I like this but sone people find lazy eval confusing to debug.
4. polars can do things like parallelization under the hood
5. Polars is less inclined to guess what you mean about things like type conversions and will instead fail and throw until you fix your code to be more explicit. Pandas is a bit more lassez faire. I guess this reflects the dev philosophy of the respective teams.
Here’s someone else’s writeup with some benchmarks https://towardsdatascience.com/pandas-vs-polars-a-syntax-and...
1. Polars has a narrower API that seems a bit cleaner and more logical vs pandas is more maximalist. Matter of taste but https://franzdiebold.github.io/polars-cheat-sheet/Polars_che... Will give you a flavour
2. Polars perf seems quite a lot better. This may not matter for your use case but if it does it does.
3. Polars supports both eager and lazy evaluation modes and you can choose whichever makes sense for your use case. I like this but sone people find lazy eval confusing to debug.
4. polars can do things like parallelization under the hood
5. Polars is less inclined to guess what you mean about things like type conversions and will instead fail and throw until you fix your code to be more explicit. Pandas is a bit more lassez faire. I guess this reflects the dev philosophy of the respective teams.
Here’s someone else’s writeup with some benchmarks https://towardsdatascience.com/pandas-vs-polars-a-syntax-and...
Thanks interesting diffs. I think functionality is the missing diff at this point. Appreciate the parallelization.
Polars seems somewhat cumbersome to use.
I’m a fan of pandas, but it is also somewhat cumbersome to use until one has learned its peculiarities.
Pandas has a variety of design elements I would characterize as "insane."
Could you give examples? I don't use a lot of Pandas.
This should be able to output the python code that a developer could use to arrive at the same conclusion. Then you can double check its work, and "freeze" the codebase into something deployable.
I can't see how this is different from langchain's pandas or Python agents https://python.langchain.com/en/latest/modules/agents/toolki...
If you want to try something similar online, https://hal9.ai — Python based. Code is either processed directly on your database or in Pandas. Viewing the code is optional, but advanced users can opt-in to view and modify the code.
With a non-deterministic GPT backing the code generation, their .run(question) API looks, frankly, terrible.
> pandas_ai.run(df, "plot the histogram of countries showing for each the gd, using different colors for each bar")
Am I right that this could potentially give different answers when run multiple times?
> pandas_ai.run(df, "plot the histogram of countries showing for each the gd, using different colors for each bar")
Am I right that this could potentially give different answers when run multiple times?
The medium article is ok, if blocked at times. This is just a summary, not by the package author.
You can jump to the code at https://github.com/gventuri/pandas-ai to see more of what it's trying to do.
You can jump to the code at https://github.com/gventuri/pandas-ai to see more of what it's trying to do.
Does this ship the entire dataframe to openai? If so probably wont work for any DF of reasonable scale.
You don't want to ship your GB frame to OAI every time your data changes materially? ;)
To the author (if you're here) and if you have any documentation covering it (didn't have proper time to check it), but I'd like to know how could I give some context of my columns to "pandas AI" and get reliable results.
I'll probably use this in something I'm working on - op (https://opapp.io). Kind of like Jupyter Notebooks but more tightly tied to dataframe libraries.
Sketch is another take on integrating an LLM into Pandas:
https://github.com/approximatelabs/sketch
https://github.com/approximatelabs/sketch
The library might be great. But from the examples shown, there is zero reason to use it as all of these can be done easily without giving your data to OpenAI.
I could be wrong, but my assumption is the only thing that is actually being sent to openai is the shape of your data, not the data itself. The text query gets converted to whatever normal filtering/aggregating mechanism exist within pandas.
the larger question I have is... if you're already advanced enough to use pandas, do you really need/want this?
the larger question I have is... if you're already advanced enough to use pandas, do you really need/want this?
These tools really need to support self-hosted model integration. We have LLaMA.
I want to applaud this effort, coming up with new ways of manipulating pandas dataframes is sorely needed. This project along with sketch[1] takes a conversational approach.
I have been building my own open source tool for wrangling pandas dataframes - Buckaroo[2]. The aim of Buckaroo is to automate common data cleaning and exploration techniques, and provide a GUI to quickly access transforms and see their results along with the python code to perform the transform. The jupyter notebook is great for presentation, but clunky for iterative exploration.
Here are the most common operations I do when exploring a new dataset
1. Assess - What are the columns in this dataset, what are their ranges of values, how do these columns vary together. Buckaroo offers toggable summary stats to show this info.
2. Initial Clean - Are all of the date columns parsed as datetimes? Are columns that contain integers typed as integers (a single string in the colum will force the whole column to type object). Clean the types with safeint/dropna...
3. Filter to a subset of data based on criteria. Row wise filtering.
4. Perform some type of analysis. Group by with different aggregations, a plot... Something more complex
Other operations that I regularly perform
5. Concattenating similar dataframes. pd.concat is simple, knowing whether your columns match up, along with their types is difficult. I want a UI that shows where they match and where they dont. This normally will require an extra data cleaning step
6. Joining tables/dataframe. I want a UI that tells me Is there a natural key to join on? What percentage of rows are joined vs not matched.
Buckaroo currently does an initial version 1,2, and 4. Buckaroo allows these steps to be done in a single notebook cell, instead of many separate cells iteratively built up. There are planned features that address all of these use cases. In addition Buckaroo has been engineered to enable easy extension by users. Plug your own functions in, and make them quickly accessible.
In my view a point and click gui is the best way to quickly accomplish these tasks as opposed to a conversational AI approach. The conversational AI approach doesn't provide enough structure around common tasks.
PS: This morning I added a "Related Projects" [3] Section to the Buckaroo docs. If Buckaroo doesn't solve your problem, look at one of the other linked projects (like Mito).
[1] https://github.com/approximatelabs/sketch
[2] https://github.com/paddymul/buckaroo
[3] https://buckaroo-data.readthedocs.io/en/latest/FAQ.html
I have been building my own open source tool for wrangling pandas dataframes - Buckaroo[2]. The aim of Buckaroo is to automate common data cleaning and exploration techniques, and provide a GUI to quickly access transforms and see their results along with the python code to perform the transform. The jupyter notebook is great for presentation, but clunky for iterative exploration.
Here are the most common operations I do when exploring a new dataset
1. Assess - What are the columns in this dataset, what are their ranges of values, how do these columns vary together. Buckaroo offers toggable summary stats to show this info.
2. Initial Clean - Are all of the date columns parsed as datetimes? Are columns that contain integers typed as integers (a single string in the colum will force the whole column to type object). Clean the types with safeint/dropna...
3. Filter to a subset of data based on criteria. Row wise filtering.
4. Perform some type of analysis. Group by with different aggregations, a plot... Something more complex
Other operations that I regularly perform
5. Concattenating similar dataframes. pd.concat is simple, knowing whether your columns match up, along with their types is difficult. I want a UI that shows where they match and where they dont. This normally will require an extra data cleaning step
6. Joining tables/dataframe. I want a UI that tells me Is there a natural key to join on? What percentage of rows are joined vs not matched.
Buckaroo currently does an initial version 1,2, and 4. Buckaroo allows these steps to be done in a single notebook cell, instead of many separate cells iteratively built up. There are planned features that address all of these use cases. In addition Buckaroo has been engineered to enable easy extension by users. Plug your own functions in, and make them quickly accessible.
In my view a point and click gui is the best way to quickly accomplish these tasks as opposed to a conversational AI approach. The conversational AI approach doesn't provide enough structure around common tasks.
PS: This morning I added a "Related Projects" [3] Section to the Buckaroo docs. If Buckaroo doesn't solve your problem, look at one of the other linked projects (like Mito).
[1] https://github.com/approximatelabs/sketch
[2] https://github.com/paddymul/buckaroo
[3] https://buckaroo-data.readthedocs.io/en/latest/FAQ.html
In general though, this is a good idea, definitely some obstacles to overcome though - and I suspect many of them involve solving problems that have to be solved on the OpenAI side, not on the user side. For example "send my entire data frame to OpenAI" seems a bit like writing a blank cheque, at the least they could mitigate that on the user side by having a pandas_ai.estimate which could take a run command and analyze the tokens to come up with a cost estimate. The other thing is obviously you want to cache the dataframe on the OpenAI side but that's definitely going to be tricky for these devs.
I wonder if the authors have checked if the data analysis is actually correct too, I guess that the AI is generating the code, not the actual analysis so if the code is wrong you'll get the wrong analysis on the right data rather than the right analysis on the wrong data.