Lessons for Building AI-Driven Products(blog.insightdatascience.com)
blog.insightdatascience.com
Lessons for Building AI-Driven Products
https://blog.insightdatascience.com/moving-towards-managing-ai-products-5268c5e9ecf2
18 comments
A lot of good comments in this thread.
I am a data scientist and of course I love playing with the latest deep learning frameworks, but If I'm putting an ML model in production I want something that: will pass QA, is easy to interpret, will run in real-time and for those requirements it's hard to beat random forest, boosted trees, logistic regression and naive bayes
Can anyone here offer advice for doing anomaly detection in distributed systems?
I'm not looking for advice on which models to use, per se. I'm more interested in how to go about things as a single-person team (building data warehousing infrastructure, etc.)
I'm not looking for advice on which models to use, per se. I'm more interested in how to go about things as a single-person team (building data warehousing infrastructure, etc.)
If I could add to this: realize that “AI” will be a small fraction of the overall product. And important one, one without which the product would likely not exist, but in terms of effort spent it will be small. So no, you don’t need an army of ML researchers to ship it. You need maybe one or two, and the rest of your money is better spent on hiring engineers, sales/bizdev people, and a great product manager. People don’t give a shit about models. They do give a shit about whether your product solves a problem. Use AI as a means to an end, not an end in itself. Focus narrowly on a concrete problem, don’t try to do everything at once.
Yes, exactly. And most of the time, you can get by with a simple model like a random forest, so you just need to have a few analysts/engineers doing feature engineering, and feeding it into your model.
Almost everyone using more complicated architectures (e.g. neural networks) are adding unnecessary complexity. [1]
[1]: Unless you're doing computer vision.
Almost everyone using more complicated architectures (e.g. neural networks) are adding unnecessary complexity. [1]
[1]: Unless you're doing computer vision.
Agreed, we've also written about why the best use of your time is to start with a baseline (even in Computer Vision!) https://blog.insightdatascience.com/always-start-with-a-stup...
And before a baseline, start with formulating a problem in a way that’s applicable to a real world product. Not just “image classification” but “image classification to improve QA in underwater basket weaving, a 10 billion dollar industry, from which we could generate $100m/yr in revenue by selling a product that does A, B, and C to an estimated 100k customers each of which would have $1000 LTV”.
I’m in an AI startup right now where the founders are hell bent on doing nothing in particular, and I’ll be bailing in a month or so.
I’m in an AI startup right now where the founders are hell bent on doing nothing in particular, and I’ll be bailing in a month or so.
I'd love to hear about your experiences — email in my profile.
I've been doing some writing and speaking on why AI seduces founders into pursuing lots of cool ideas instead of marketable products. I'd love another data point.
Could do it anonymously if you'd like, or hold off until after you bail.
I've been doing some writing and speaking on why AI seduces founders into pursuing lots of cool ideas instead of marketable products. I'd love another data point.
Could do it anonymously if you'd like, or hold off until after you bail.
[deleted]
Question: looking at the article's "AI Product Management" infographic, one item includes the phrase, "Adopt a bread-first approach to building a product". Is that supposed to be "breadth-first"?
It means to go for revenue first :-)
Their lesson, "Build Breadth-first (Data/Pipeline/Model) instead of Depth-first (AI model)" is insightful. Once the pipeline/etc. are built perhaps the AI part is not even needed any more. Finding this out early can save you a lot of trouble.
Another perspective, in which we talk about on integrating AI into existing workflows, among other lessons:
Smith, Reid G., and Joshua Eckroth. "Building AI Applications: Yesterday, Today, and Tomorrow." AI Magazine 38.1 (2017): 6-22.
https://www2.stetson.edu/~jeckroth/downloads/smith-eckroth-2...
Another perspective, in which we talk about on integrating AI into existing workflows, among other lessons:
Smith, Reid G., and Joshua Eckroth. "Building AI Applications: Yesterday, Today, and Tomorrow." AI Magazine 38.1 (2017): 6-22.
https://www2.stetson.edu/~jeckroth/downloads/smith-eckroth-2...
Not only that, but breadth-first makes it easy to: a) Iterate much faster on models; b) Keep better track of results and code and models, which is already quite difficult in ML; c) Enable other teams to work on other parts of the pipeline, in parallel.
As someone who is a product manager in the commercialized AI SaaS space, the most important pieces of feedback I would give a new PM here:
1)Don't let your -brilliant- colleagues try to force their -brilliantly complex- solution of a problem - clearly define market problems, and don't let the team try to go the route of trying to force fit a solution to a market problem. Market problems come first.
2)Frame the market problems appropriately for your ML/AI teams, and practice trying to frame the problem from a variety of angles. Framing from different angles promotes the 'Ah-ha' moment in terms of the right way to solve the problem from the ML side.
3)Don't commit serious time to a model before having a naive solution to benchmark against. Always have a naive solution to compare against the AI solution. 'Naive' here may be a simple linear regression, RMSE, or multi armed bandit/Thompson sampling.
1)Don't let your -brilliant- colleagues try to force their -brilliantly complex- solution of a problem - clearly define market problems, and don't let the team try to go the route of trying to force fit a solution to a market problem. Market problems come first.
2)Frame the market problems appropriately for your ML/AI teams, and practice trying to frame the problem from a variety of angles. Framing from different angles promotes the 'Ah-ha' moment in terms of the right way to solve the problem from the ML side.
3)Don't commit serious time to a model before having a naive solution to benchmark against. Always have a naive solution to compare against the AI solution. 'Naive' here may be a simple linear regression, RMSE, or multi armed bandit/Thompson sampling.
>Always have a naive solution to compare against the AI solution. 'Naive' here may be a simple linear regression, RMSE, or multi armed bandit/Thompson sampling.
This cannot be stressed enough, optimism bias will always push the scientist towards the 'more interesting/complete/new' method and model, but a seasoned practitioner will have the discipline to always establish a baseline (<1 days work).
This cannot be stressed enough, optimism bias will always push the scientist towards the 'more interesting/complete/new' method and model, but a seasoned practitioner will have the discipline to always establish a baseline (<1 days work).
In my experience, simplifying a solution over time is the best way to ensure its adoption and long-term success. That applies both to UX/UI and the supporting computation/ML.
Could be better written but here are my thoughts on the same topic from earlier last year: https://medium.com/towards-data-science/hard-earned-advice-f...