I think you're probably right. As an example of this challenge, I've noticed that engineers who don't have a background in ML often lack the "mental models" to understand how to think about testing ML models (i.e. statistical testing as opposed to the kind of pass/fail test cases that are used to test code).
The way I look at this is that plexe can be useful even if it doesn't solve this fundamental problem. When a team doesn't have ML expertise, their choices are A) don't use ML B) acquire ML expertise C) use ChatGPT as your predictor. Option C suffers of the same problem you mentioned, in addition to latency/scalability/cost and the model not being trained on your data etc. So something like Plexe could be an improvement on option C by at least addressing the latter pain points.
Plus: we can keep throwing more compute at the agentic model building process, doing more analysis, more planning, more evaluation, more testing, etc. It still won't solve the problem you bring up, but hopefully it gets us closer to the point of "good enough to not matter" :)
Interesting! We don't currently support this explicitly.
You could wrap the Plexe-built model in a scikit-learn Estimator like I mentioned, and you can specify the desired input/output schema of the model when you start building it, so it will fit into your Pipeline.
This is an interesting requirement for us to think about though. Maybe we'll build proper support for the "I want to use this in a Pipeline" use case :)
No, not by default. In fact, the default installation of plexe doesn't include deep learning libraries.
Plexe _can_ build deep learning models using `torch` and `transformers`, and often the experimentation process will include some NN-based solutions as well, but that's just one of the ML frameworks available to the agent. It can also build models using xgboost, scikit-learn, and several others.
You can also explicitly tell Plexe not to use neural nets, if that's a requirement.
Do you mean being able to wrap the created model in a scikit-learn Pipeline? This isn't something we've thought about and we haven't explicitly built support for it, though we could.
As of now, I think you could relatively easily wrap the plexe model, which has a `predict()` method, in a scikit-learn Estimator. You could then plug it into a Pipeline.
What do you have in mind? How would you want to use this with scikit-learn pipelines?
Agree. We've designed a mechanism to enable any of the agents to ask for input from the user, but we haven't implemented it yet. Especially for more complex use cases, or use cases where the datasets are large and training runs are long, being able to interrupt (or guide) the agents' work would really help avoid "wasted" one-shot runs.
Regarding more complicated models and demanding users, I think we'd need:
1. More visibility into the training runs; log more metrics to MLFlow, visualise the state of the multi-agent system so the user knows "who is doing what", etc.
2. Give the user more control over the process, both before the building starts and during. Let the user override decisions made by the agents. This will require the mechanism I mentioned for letting both the user and the agents send each other messages during the build process.
3. Run model experiments in parallel. Currently the whole thing is "single thread", but with better parallelism (and potentially launching the training jobs on a separate Ray cluster, which we've started working on) you could throw more compute at the problem.
I'm sure there are many more things that would help here, but these are the first that come to mind off the top of my head.
What are your thoughts? Anything in particular that you think a demanding user would want/need?
Just to clarify, we're not directly using the LLMs as the "predictor" models for the task. We're making the LLMs do the modeling work for you.
For example, take the classic "house price prediction" problem. We don't use an LLM to make the predictions, we use LLMs to model the problem and write code that trains an ML models to predict house prices. This would most likely end up being an xgboost regressor or something like that.
As to your point about evals, great question! We've done some testing but haven't yet carried out a systematic eval. We intend to run this on OpenAI's MLE-Bench to quantify how well it actually does as creating models.
Currently it decides whether to make its own model or fine-tune a relevant one based primarily on the problem description. The agent's ability to analyse the data when making decisions is pretty limited right now, and something we're currently working on (i.e. let the agent look at the data whenever relevant, etc).
I guess that kind of answers your second question, too: it does not currently detect issues with the training data. But it will after the next few pull requests we have lined up!
And yes, completely agree about data cleaning vs. model building. We started from model building as that's the "easier" problem, but our aim is to add more agents to the system to also handle reviewing the data, reasoning about it, creating feature engineering jobs, etc.
Hey, this is super cool! We found a few projects working on similar things to Plexe, but were not aware of yours. Thanks for sharing, will check it out!
Hey, one of the authors here! I completely agree with your comment. Training ML models on a clean dataset is the "easy" and fun part of an ML engineer's job.
While we do think our approach might have some advantages compared to "2018-style" AutoML (more flexibility, easier to use, potentially more intelligence solution space exploration), we know it suffers from the issue you highlighted. For the time being, this is aimed primarily at engineers who don't have ML expertise: someone who understands the business context, knows how to build data processing pipelines and web services, but might not know how to build the models.
Our next focus area is trying to apply the same agentic approach to the "data exploration" and "feature ETL engineering" part of the ML project lifecycle. Think a "data analyst agent" or "data engineering agent", with the ability to run and deploy feature processing jobs. I know it's a grand vision, and it won't happen overnight, but it's what we'd like to accomplish!
Hey, I'm one of the authors of Plexe. Overall, I'd say we like smolagents: it's simple, easy to understand, and you can get a project set up very quickly. It also has some neat features, such as the "step callbacks" (functions that are executed after every step the agent takes).
However, the library does feel somewhat immature, and has some drawbacks that hinder building a production application. Some of the issues we've ran into include:
1. It's not easy to customise the agents' system prompts. You have to "patch" the smolagents library's YAML templates in a hacky way.
2. There is no "shared memory" abstraction out of the box to help you manage communication between agents. We had to implement an "ObjectRegistry" class into which the agents can register objects, so that another agent can retrieve the object just by knowing the object's key string. As we scale, we will need to build more complex communication abstractions (tasks queues etc). Given that communication is a key element of multi-agent systems, I would have expected a popular library like smolagents to have some kind of built-in support for it.
3. No "structured response" where you can pass a Pydantic BaseModel (or similar) to specify what structure the agent response should have.
4. "Managed agents" are always executed synchronously. If you have a hierarchy of managed agents, only one agent will ever be working at any given time. So we'll have to build an async execution mechanism ourselves.
I think we've run into some other limitations as well, but these are the first that come to my mind :) hope this helps!
The way I look at this is that plexe can be useful even if it doesn't solve this fundamental problem. When a team doesn't have ML expertise, their choices are A) don't use ML B) acquire ML expertise C) use ChatGPT as your predictor. Option C suffers of the same problem you mentioned, in addition to latency/scalability/cost and the model not being trained on your data etc. So something like Plexe could be an improvement on option C by at least addressing the latter pain points.
Plus: we can keep throwing more compute at the agentic model building process, doing more analysis, more planning, more evaluation, more testing, etc. It still won't solve the problem you bring up, but hopefully it gets us closer to the point of "good enough to not matter" :)
Would love to hear your thoughts on this.