Business questions engineers should ask when interviewing at ML/AI companies(medium.com)
medium.com
Business questions engineers should ask when interviewing at ML/AI companies
https://medium.com/@danielgross/seven-questions-to-ask-when-interviewing-for-an-ml-job-1963ccee3a19
114 comments
Holy shit. That is amazing. I watched two minutes of video and I now know less than I did when I started. It took me a disturbingly long period of time to decide whether or not it was a parody.
This is awesome. I can't wait to deploy real-time AI assisted learning to my staff of ... wait, who? why?
I can't decide if that is awful in the sociopath sense or amazing in the "cult movie" sense. So why choose? It's both. Definitely both.
I decoded it from the first 60 seconds. It's a courseware platform for internal use in companies, presented as white label, the company can pass it as their own product.
I added this comment:
This product is certified buzzword compliant.
This product is certified buzzword compliant.
Is this...real?
Yes. Actually so.
https://www.edcast.com
https://www.edcast.com
Did any of you succeed in asking such questions to companies (ML/AL or not) and ended up receiving reasonable responses to your questions?
My experiences have never been great when asking such loaded questions. To be clear, I interviewed extensively at small startups and I'm only taking those interviews into consideration for this comment.
The HR and technical interviewers dedicate a large amount of time for questioning you, and reserve the last few minutes to answer your questions. I lost count of how many times the interviewers would just blabber something meaningless in a rush, rather than answer a questions with patience and honesty. Of course, I made my decisions on whether or not to join the team based on such experiences. However, I cannot think of any remarkable instances where the interviewers answered such questions without getting impatient.
Any chats with directors/PMs/C-level executives after receiving an offer were also not very informative. I walked out several times going in and coming back from such chats with no questions properly answered because: (a) they are still figuring out, or (b) they cannot discuss certain details because you haven't said yes to them.
My experiences have never been great when asking such loaded questions. To be clear, I interviewed extensively at small startups and I'm only taking those interviews into consideration for this comment.
The HR and technical interviewers dedicate a large amount of time for questioning you, and reserve the last few minutes to answer your questions. I lost count of how many times the interviewers would just blabber something meaningless in a rush, rather than answer a questions with patience and honesty. Of course, I made my decisions on whether or not to join the team based on such experiences. However, I cannot think of any remarkable instances where the interviewers answered such questions without getting impatient.
Any chats with directors/PMs/C-level executives after receiving an offer were also not very informative. I walked out several times going in and coming back from such chats with no questions properly answered because: (a) they are still figuring out, or (b) they cannot discuss certain details because you haven't said yes to them.
I always answered this for Graphistry candidates. We take a vertical data product mindset to AI -- so just one solid piece of solving the human-in-the-loop investigation problem for security and fraud teams trying to help their responders get a handle on incidents/APIs/logs -- so perhaps we're less concerned about image. For startups with real tech, competition shouldn't be the thing that matters, but execution.
For us, a great interview is one where the candidate teaches us and thereby demonstrates the potential to be a force multiplier for the team. Also, someone asking these kinds of questions suggests we may be able to put them in front of a customer!
For us, a great interview is one where the candidate teaches us and thereby demonstrates the potential to be a force multiplier for the team. Also, someone asking these kinds of questions suggests we may be able to put them in front of a customer!
While I appreciate the nature of these questions, many of them exclude network-driven consumer businesses. You know, those like Tinder, Facebook, Twitter, Medium, Yelp, FourSquare... If you're spending time with a consumer company like that, I'd focus less on LTV and more on organic growth rate.
This is silly.
You’re the one being evaluated here. Can you code, or are you just a “sophisticated” smartass?
The trick is to find out all these answers WITHOUT asking the interviewer.
The trick is to find out all these answers WITHOUT asking the interviewer.
Some other questions I would want the answers to:
1) Is a Non-Compete Agreement required and what are the terms (time and geographic coverage; compensation during the non-compete time frame)?
2) Is the company going to assert ownership over IP created before employment began?
3) Is the company going to assert ownership over IP created that is unrelated to current responsibilities (or IP in a area unrelated to the company's current business), especially IP created on the employee's own time, not using any company resources?
1) Is a Non-Compete Agreement required and what are the terms (time and geographic coverage; compensation during the non-compete time frame)?
2) Is the company going to assert ownership over IP created before employment began?
3) Is the company going to assert ownership over IP created that is unrelated to current responsibilities (or IP in a area unrelated to the company's current business), especially IP created on the employee's own time, not using any company resources?
In California, the non-compete is not worth the paper it's written on: http://prodigylaw.com/CA_Death_of_Non-Compete.html and the state with the strictest non-compete enforcement, Massachusetts, seems to be close to the long-anticipated reform: https://www.faircompetitionlaw.com/2017/10/10/massachusetts-... and https://www.faircompetitionlaw.com/2017/10/31/taking-aim-at-.... With the reformed bill in place, the non-compete clause in Massachusetts will also be a hollow threat unless multiple (unrealistic) conditions are met.
These are great questions to ask any company, not just AI/ML companies. Also, if you're planning to do it:
4) What is the companies policy on employees moonlighting / doing side contract work for other companies not related to it's business?
Inquire about this before you get your offer. Ask me how I know...
EDIT: Another one, which is kind of risky but I personally think is important and would ask any small company:
5) How transparent/open are you about the company's financial health? Are the books open or semi-open?
Keeping in mind that some companies will not share this information and may even find the question offensive. One time I asked the owner/founder this, and he would not share--I found out later through a side channel that he was very offended that I asked for this information (but hired me anyway because he needed someone like me).
4) What is the companies policy on employees moonlighting / doing side contract work for other companies not related to it's business?
Inquire about this before you get your offer. Ask me how I know...
EDIT: Another one, which is kind of risky but I personally think is important and would ask any small company:
5) How transparent/open are you about the company's financial health? Are the books open or semi-open?
Keeping in mind that some companies will not share this information and may even find the question offensive. One time I asked the owner/founder this, and he would not share--I found out later through a side channel that he was very offended that I asked for this information (but hired me anyway because he needed someone like me).
Red flag if asking this question courteously causes offence.
"Why does anyone need this? Like all advice, this sounds deceptively simple. But make sure you get a very compelling answer here."
Right now I'm at the Web Summit, in Lisbon. I saw a few ML/AI startups here, but I was surprised there weren't a lot more. There was some imbalance. There were many startups trying to create new online social networks for niche groups -- business professionals, women, entrepreneurs, sales professionals, etc. I do think there is some innovation that can happen in social networks, but generally I regard it as an over-crowded space. For the most part, the benefits offered by the Internet seemed to have been mostly absorbed by the 1994-2008 cycle, and what's left is fairly minor compared to what happened previously.
One of the few areas where I still see the possibility of significant traction (the creation of large companies) is with information gathering and a combination of ML and NLP. We've already seen a wave of startups which were no more than pretty GUIs over existing technology. At the risk of being unfair, BigML.com is just a nice interface over existing ML tools, and API.ai is just a nice interface over some NLP tools. AmenityAnalytics goes a bit further combining their web scraping and NLP scripts with a nice interface that customers can use to filter the incoming data. But there is still a wide space for companies to go much further in this field.
All the same, I agree this is a good question: "Why does anyone need this?" You should really ask it whenever you are joining any startup.
Right now I'm at the Web Summit, in Lisbon. I saw a few ML/AI startups here, but I was surprised there weren't a lot more. There was some imbalance. There were many startups trying to create new online social networks for niche groups -- business professionals, women, entrepreneurs, sales professionals, etc. I do think there is some innovation that can happen in social networks, but generally I regard it as an over-crowded space. For the most part, the benefits offered by the Internet seemed to have been mostly absorbed by the 1994-2008 cycle, and what's left is fairly minor compared to what happened previously.
One of the few areas where I still see the possibility of significant traction (the creation of large companies) is with information gathering and a combination of ML and NLP. We've already seen a wave of startups which were no more than pretty GUIs over existing technology. At the risk of being unfair, BigML.com is just a nice interface over existing ML tools, and API.ai is just a nice interface over some NLP tools. AmenityAnalytics goes a bit further combining their web scraping and NLP scripts with a nice interface that customers can use to filter the incoming data. But there is still a wide space for companies to go much further in this field.
All the same, I agree this is a good question: "Why does anyone need this?" You should really ask it whenever you are joining any startup.
"It's like Facebook, but for $group"
Groan
Groan
Shouldn't the interviewing engineer know the answers to these business questions beforehand during their research of the company? (particularly if the company is upfront about their ML/AI-usage)
In my experience the answers I find on the company home page and the answer I get when asking the question in an interview only rarely sync up perfectly and the difference can be very illuminating.
Good question! Many pre-launch startups lack an online presence, so it’d be hard to tease out answers before talking to someone.
Usually you're only going to get good answers to these questions after they've made you an offer.
These questions should be asked about every startup, not only ML/AI.
> Google got great because of PageRank, but it stayed great due to network effects.
I think this overstates network effects and under-appreciates branding and simply reinvesting significant capital into continued R&D.
I think this overstates network effects and under-appreciates branding and simply reinvesting significant capital into continued R&D.
One of the primary factors that enabled Google to stay great was their ability to crawl and index the ever growing web at a rate no other company could.
Agreed. There are many flavors of moats. I originally wrote something more complex but it quickly turned into an amateur HBS study about Google. I hope the point is interpreted more metaphorically.
I've heard the line that Google succeeds because of network effects so many times on HN that I feel like that's the only business concept commenters know about.
I would suggest that in many cases the "data network effect" advantage of ML startups is overappreciated relative to the branding advantage they have. For example, DDG and others can, for the most part, match Google in quality of search, but almost no one will use them because of entrenched branding, this is especially true IMO in smaller ML vertical industries ideal for startups. e.g., if there is only room for one ML company that analyzes farm pipe leaks through camera, then branding and market dominance helps considerably there.
I may be wrong, I'm (mostly) a DL hardware guy, this is just my two cents :)
I may be wrong, I'm (mostly) a DL hardware guy, this is just my two cents :)
>For example, DDG and others can, for the most part, match Google in quality of search
Even for the long tail? I'm pretty skeptical of this.
Even for the long tail? I'm pretty skeptical of this.
Google will I'm sure vehemently deny that it is unbeatable but I think with 60k+ searches every second, Google will keep getting better. I think Google is unbeatable in search as we know it.
I'm not counting out google, but there are only two searches that actually count from a business perspective. product and services. All the rest are cost of doing business.
If you do your research on google and then buy from amazon or somewhere else and don't use the adword link google's business strategy fails.
If you do your research on google and then buy from amazon or somewhere else and don't use the adword link google's business strategy fails.
Correct statment is.
Google got great because of PageRank, but it stayed great due to AdWords.
Google got great because of PageRank, but it stayed great due to AdWords.
>Google got great because of PageRank, but it stayed great due to AdWords.
Google got great because of PageRank, but it stayed viable due to AdWords.
Fixed that for ya
Google got great because of PageRank, but it stayed viable due to AdWords.
Fixed that for ya
And to add to that - I don't think when Google was launched, anybody thought about AdWords. This statement is technically true:
> How do you make money? Be on the lookout for what I call “multistage rockets”: “Today, we’re doing X. But our grand plan is to do Y, which will be really profitable”. These usually fail.
But, it is also completely non-informative. All startups usually fail. All (completely) new endeavors, even in established companies, usually fail. That does NOT mean though that it's not worth a try. You cannot generally anticipate apriori what will work and what not - and you CERTAINLY cannot anticipate it from the business plan (what was the business plan of Facebook? Instagram?). If you stand any chance to anticipate "what will work" - it's from the team, not the business plan.
> How do you make money? Be on the lookout for what I call “multistage rockets”: “Today, we’re doing X. But our grand plan is to do Y, which will be really profitable”. These usually fail.
But, it is also completely non-informative. All startups usually fail. All (completely) new endeavors, even in established companies, usually fail. That does NOT mean though that it's not worth a try. You cannot generally anticipate apriori what will work and what not - and you CERTAINLY cannot anticipate it from the business plan (what was the business plan of Facebook? Instagram?). If you stand any chance to anticipate "what will work" - it's from the team, not the business plan.
unfortunately, you are just a data point for investors risk calculations.
Should the engineers ask the same question when a high profile VC, like YC have already invested on the company? The questions look more like what a VC should (have) asked before funding.
Many ML/AI companies especially if funded and therefore highly visible may get acquired as the market is very hot. Isn't that good for the engineers?
Many ML/AI companies especially if funded and therefore highly visible may get acquired as the market is very hot. Isn't that good for the engineers?
If equity is part of your compensation, you're an investor and should be doing the appropriate due diligence.
If you're a friend of the VC and he suggests you work for a portfolio company, then maybe you can rely on the VC's judgment without asking questions.
General it's always legitimate to ask these questions. Why? Because they cut straight to how the company will be valued, by a customer or by a potential acquirer.
General it's always legitimate to ask these questions. Why? Because they cut straight to how the company will be valued, by a customer or by a potential acquirer.
> Many ML/AI companies especially if funded and therefore highly visible may get acquired as the market is very hot. Isn't that good for the engineers?
Only if the potential acquirers are happy with the answers to the same questions. As a percentage of net worth, individual engineers have a lot more at stake than VC's or potential acquirers.
Only if the potential acquirers are happy with the answers to the same questions. As a percentage of net worth, individual engineers have a lot more at stake than VC's or potential acquirers.
Most of the big companies(apple/google/microsoft/fb/amazon) at current stage are acquiring for talent/IP, not necessarily market fit.. they do have enough money to do the R&D without worrying about finding a market fit soon
Not sure why ML/AI is in the title. Seems applicable to any startup.
The gotcha here is that most desirable AI startups SV wants to fund now (understandably) follow a pattern more closely described here:
http://www.bradfordcross.com/blog/2017/6/13/vertical-ai-star...
Generally, their advantage is in the way they use data to solve a vertical specific problem. They more "bolt on" AI. AI isn't really the focus of most of these companies. This is just what investors want to hear about.
The article here isn't calling them out specifically, but AI infra startups are what are being discussed here as the "hammer looking for a nail" startup.
YC funded a few of them including Skymind (my company) and deepgram (mainly audio). There are a lot more of these startups such as vicarious that mainly publish papers but raise tons of funding hoping for a deepmind style exit (they didn't really have a business model).
With that context out of the way, I'll also maybe just state some learnings from the other side of the table.
Skymind has typically made money trying to reduce dependencies on cloud providers by decoupling the AI infra from google cloud and co from the cloud provider itself.
A horizontal play like this is by its nature very hard. We maintain the whole stack including our own framework. We also started in 2014 and have a decent foothold in enterprise. I wouldn't recommend trying to do this in 2017.
That being said, one other semi similar horizontal AI startup that is being started are the chip companies. Those are significantly harder to run than even what we're doing.
The most common "failed" type of startup that I think of when we think "horizontal" plays are Machine Learning as a service, which is hugely a loss leader for the various cloud providers (some companies are building on top of these providers though).
This is where you see Metamind, Alchemy API, Scaled Inference (founded in 2015), even Nervana before they sold were trying to a "nervana asic cloud" among others.
Maybe a lot of investors from SV view this as a waste of time dissecting, but I would personally love to see a bit more content on acknowledging some of these trends in the market.
The allure of "AI as a service" and the horizontal dev tools infra play is that you can try to build the next AWS similar to what the container and database companies are trying to do. Execution is definitely key for this to work though. Research also can't be the primary focus.
I won't comment on what will or won't work there
Generally, their advantage is in the way they use data to solve a vertical specific problem. They more "bolt on" AI. AI isn't really the focus of most of these companies. This is just what investors want to hear about.
The article here isn't calling them out specifically, but AI infra startups are what are being discussed here as the "hammer looking for a nail" startup.
YC funded a few of them including Skymind (my company) and deepgram (mainly audio). There are a lot more of these startups such as vicarious that mainly publish papers but raise tons of funding hoping for a deepmind style exit (they didn't really have a business model).
With that context out of the way, I'll also maybe just state some learnings from the other side of the table.
Skymind has typically made money trying to reduce dependencies on cloud providers by decoupling the AI infra from google cloud and co from the cloud provider itself.
A horizontal play like this is by its nature very hard. We maintain the whole stack including our own framework. We also started in 2014 and have a decent foothold in enterprise. I wouldn't recommend trying to do this in 2017.
That being said, one other semi similar horizontal AI startup that is being started are the chip companies. Those are significantly harder to run than even what we're doing.
The most common "failed" type of startup that I think of when we think "horizontal" plays are Machine Learning as a service, which is hugely a loss leader for the various cloud providers (some companies are building on top of these providers though).
This is where you see Metamind, Alchemy API, Scaled Inference (founded in 2015), even Nervana before they sold were trying to a "nervana asic cloud" among others.
Maybe a lot of investors from SV view this as a waste of time dissecting, but I would personally love to see a bit more content on acknowledging some of these trends in the market.
The allure of "AI as a service" and the horizontal dev tools infra play is that you can try to build the next AWS similar to what the container and database companies are trying to do. Execution is definitely key for this to work though. Research also can't be the primary focus.
I won't comment on what will or won't work there
The fastest way to make progress on these business questions is often to build hacky MVPs that look like they're doing something smart, but behind the scenes are powered by humans or dead-simple algorithms, and get them in front of customers ASAP.
I recently joined a seed-stage startup solving a business problem via audio analysis in the manner I described above. I'm not spending much time doing ML yet, but I'm banking on my belief that we're solving a valuable problem (customers want to buy our hacky MVP) and that ML can and will be needed to scale our solution. By deeply understanding the customer as a first step, I think the ML systems we build will be business critical and enduring. Time will tell
I recently joined a seed-stage startup solving a business problem via audio analysis in the manner I described above. I'm not spending much time doing ML yet, but I'm banking on my belief that we're solving a valuable problem (customers want to buy our hacky MVP) and that ML can and will be needed to scale our solution. By deeply understanding the customer as a first step, I think the ML systems we build will be business critical and enduring. Time will tell
> dead-simple algorithms
The most successful models I've ever built have been logistic regression models. If you can rephrase your problem in a way that's amenable to run-of-the-mill statistical techniques, you can frequently achieve much better results than you can with 'deep learning'.
The most successful models I've ever built have been logistic regression models. If you can rephrase your problem in a way that's amenable to run-of-the-mill statistical techniques, you can frequently achieve much better results than you can with 'deep learning'.
If you're going to do logistic regression, at least call it a single-layer neural network with a sigmoid activation function.
So true. I have had to do this.
I had a pretty good regression model but it was not taken seriously. So I wrote it using "a neural network in TensorFlow" and the next thing you know the whole company is asking me how it works and what it does.
I had a pretty good regression model but it was not taken seriously. So I wrote it using "a neural network in TensorFlow" and the next thing you know the whole company is asking me how it works and what it does.
This feels so dirty, but these kinds of tricks work. Your stakeholders get to participate in the titillating fiction that they are on the bleeding edge of technology, and you get to deploy a scalable, explainable, and (hopefully) high-performing solution. That's 95% of a win.
I'm totally "borrowing" that for later use.
Amen.. I like to use exotic techniques just out of intellectual curiosity and to put my education to use, but ultimately a basic linear regression is all people want! Ease of interpretation is paramount.
Question 6. How often do people talk about ARPUs with potential hires? I am sure people will tell you that there are 150 million potential users but not their revenue models. It is easy to under shoot or overshoot ARPU.
[deleted]
This is really solid advice when looking at it backwards too--if you, as a founder, cannot produce reasonable and compelling answers to these questions, you don't know what you're doing, where you're going, or how to actually get there.
Great read.
Great read.
This advice applies to all kinds of fledgling businesses, not just ML/AI.
Yes, quite good actually.
Unfortunately, sometimes the hype is so strong, the cart comes before the horse and gets acquired for quite a lot - which is why these companies exist.
Unfortunately, sometimes the hype is so strong, the cart comes before the horse and gets acquired for quite a lot - which is why these companies exist.
The best use of these questions is to get an idea whether your equity might be worth anything. Ask how many customers they have, how fast they're growing, how long does it take to onboard customers, etc. What do they see as an exit (is it a 10M, 100M, 500M a unicorn?). When do they see it happening?
If, let's say, you have 0.01% of a Series A that exits at a billion in five years. You'd expect to be diluted by half and there is around a 20% chance of exit that high so roughly ((0.01%billion)/2).20 /5 or about 2k a year in 'expected value'(or why, numerically at least, it's likely not worth it to take a paycut to work at a startup unless you're seeing substantial equity).
If, let's say, you have 0.01% of a Series A that exits at a billion in five years. You'd expect to be diluted by half and there is around a 20% chance of exit that high so roughly ((0.01%billion)/2).20 /5 or about 2k a year in 'expected value'(or why, numerically at least, it's likely not worth it to take a paycut to work at a startup unless you're seeing substantial equity).
Please let me know where I can find today’s Series A ready companies that have a 20% chance of being worth $1b in 5 years time.
That figure probably oughta be at least one order of magnitude less.
That figure probably oughta be at least one order of magnitude less.
Agree with the other commenter that this still seems overly optimistic, but also I think as stated your math is off and it’d be 20k/year?
So. Much. B.S.
There's only one question they need to ask:
"How much are you going to pay me?"
Edit: The reason anyone would wax poetically about everything else is because he or she realizes that they are not being paid enough. It is like day trading - the moment it goes against a bad trader he or she calls it "investment"
Edit: Thanks for the down votes! It makes me feel warm and fuzzy inside to know that it is still possible to swindle people!
There's only one question they need to ask:
"How much are you going to pay me?"
Edit: The reason anyone would wax poetically about everything else is because he or she realizes that they are not being paid enough. It is like day trading - the moment it goes against a bad trader he or she calls it "investment"
Edit: Thanks for the down votes! It makes me feel warm and fuzzy inside to know that it is still possible to swindle people!
Nah. This is a good list of questions. Partly because startups will include equity, and you need to know these things (and more) to make a guess at valuing the equity. And partly because a company without strong answers is likely to be out of business soon; changing jobs unexpectedly can be expensive, especially if you get stiffed on wages.
But mostly because it really sucks to spend a couple of years building stuff that never gets used. There are a few people who seem happy to get paid even if nothing is achieved. But most engineers I know do the work because the like building things that get used, things that are valuable.
But mostly because it really sucks to spend a couple of years building stuff that never gets used. There are a few people who seem happy to get paid even if nothing is achieved. But most engineers I know do the work because the like building things that get used, things that are valuable.
Unless you are employee #1-5, the correct approach to valuing equity is to not consider it to be worth anything. For employees 1-5, divide the numbers the CEO gives you by at least 10 to account for risk, and then demand a competitive base pay.
Yes and no. I agree should expect zero to come out of it, as it's a pretty low probability you'll see anything. But you still have to do the work of valuing it.
First off, the people you're negotiating with see it as valuable, so it's a very bad negotiating move to say, "Keep your dumb equity and just give me cash." You'd be slightly better off just pointing to their family photo and saying, "Wow, your kids are ugly." Second, going through the work of valuing equity is necessary to see how the investors (that is, they people actually paying your salary) look at the company. Third, on the off chance you are successful, you're going to really wish you took the equity part of the negotiation seriously.
First off, the people you're negotiating with see it as valuable, so it's a very bad negotiating move to say, "Keep your dumb equity and just give me cash." You'd be slightly better off just pointing to their family photo and saying, "Wow, your kids are ugly." Second, going through the work of valuing equity is necessary to see how the investors (that is, they people actually paying your salary) look at the company. Third, on the off chance you are successful, you're going to really wish you took the equity part of the negotiation seriously.
People you’re negotiating with want you to buy their line that their equity is worth something so they could pay you less. What would you get as eg employee #10? Half a percent or less. Most startups exit under $100M, and your 0.5% turns into 0.25% due to dilution. It’s also vested over 4 years. We’re talking $250k max over 4 years, with a 5% chance you’ll get anything at all. You’d have to be seriously arithmetically challenged to take that seriously.
People are arithmetically challenged. People also like to boast about doing good etc. Typically they realize how stupidly naive they were just around the time they no longer have ability to say "That's nice to know. So how much are you going to pay me?"
The down votes on calling spade a spade demonstrate it quite well
The down votes on calling spade a spade demonstrate it quite well
"How much are you going to pay me?"
Sure, but the 'correct' answer to that question depends on a lot of other factors related to the job. If you want me to work long hours, doing dull, morally dubious, work in a high stress environment you're going to have to pay me a lot more.
Sure, but the 'correct' answer to that question depends on a lot of other factors related to the job. If you want me to work long hours, doing dull, morally dubious, work in a high stress environment you're going to have to pay me a lot more.
You are looking at the other factors to justify the smaller pay than you know you are get elsewhere. Should you be telling people about how much money you make you would be using a word "but" followed by the list of these factors.
Sleep on that.
Sleep on that.
You are looking at the other factors to justify the smaller pay than you know you are get elsewhere
Absolutely. Money is just one of many factors I consider when choosing a job. I see no 'shame' in that. I honestly have/earn enough money to cover all my expenses, so getting a bit more isn't that super important to me. As long as you meet my floor I'm willing to negotiate away money for other perks. I'd rather have time to spend my money rather than just more money.
Absolutely. Money is just one of many factors I consider when choosing a job. I see no 'shame' in that. I honestly have/earn enough money to cover all my expenses, so getting a bit more isn't that super important to me. As long as you meet my floor I'm willing to negotiate away money for other perks. I'd rather have time to spend my money rather than just more money.
Typical of SV to focus solely on the business viability of a firm to the detriment of everything else. I’d like to work somewhere knowing that I have a clean conscience. Why not add:
Where do you get your data from?
If your data is sourced from users of your product, do you tell them what you’re collecting?
Where do you get your data from?
If your data is sourced from users of your product, do you tell them what you’re collecting?
Huh? There are dozens, if not hundreds, of things one should consider when contemplating taking a job. That an article focuses on things on one particular category (business viability) does not imply that the author thinks that things from other categories (such as ethical considerations) are less important.
They essentially all say "tell" their users where they are getting the data. It's in the privacy policy. But people don't read it.
Exactly. Informing the user does nothing except maybe clear your conscience. Expecting the user to extrapolate to how you will use that data, what will be inferred from it, what risks come with centralizing and storing it, all of that is not the users job. It’s the job of the collector to be as conservative as possible. Any company that treats users privacy as a resource to be extracted is not somewhere I want to work.
There many other factors to optimize for, like the people you’ll work with, the technologies you’ll work on, commute, etc.After working for a few companies who (tried to) make ML products, I think one of the most important questions to ask is "who owns the data you are building models on?". It is way harder for a company to build good models off data they don't have complete access to and full knowledge about. The worst (and unfortunately common) scenario for companies trying to do AI is that data scientists don't have full access to all priors necessary to build good models, and the owners of the data don't really know much about it either. Spells death of company
Well said. It is a good tester of knowing how clear the people understand the problem themselves. Many people (particularly non-science background ones, no offence) don't know which kind of data they should access to build a product and, even worse, whether they have access to the data.
Not knowing the answer of this question = they do not know they are travelling on Titanic at best, if not a boat full of holes.
Not knowing the answer of this question = they do not know they are travelling on Titanic at best, if not a boat full of holes.
[deleted]
And here I was thinking these would be questions about the moral / ethical / social implications of the ways in which many companies use these technologies..
Is it different from https://en.wikipedia.org/wiki/Wisdom_of_the_crowd
Good questions, and they apply to many more companies than just ML/AI
If startup: When do you need the next financing round?
I am in academia, but some of my students have returned somewhat disgruntled from their internships in companies claiming to do machine learning, while in practice their systems were primarily rule-based and the work consisted of writing rules [1].
I now recommend students to find out whether companies are just name dropping ML or doing serious ML work. Typically, the best way is to ask former employees/interns. But I question directed at this also would not hurt.
[1] There is nothing wrong with that, but it sets the wrong expectations.
I now recommend students to find out whether companies are just name dropping ML or doing serious ML work. Typically, the best way is to ask former employees/interns. But I question directed at this also would not hurt.
[1] There is nothing wrong with that, but it sets the wrong expectations.
I work in industry (sort of) and this is very true.
One of the reasons for this is that the people designing the systems don't know how to do anything else.
There's such a huge, huge gap between using ML to do the same thing everyone else is doing (say... image classification, or maybe regression on volume vs sales...) and solving unique business problems using ML.
One of the reasons for this is that the people designing the systems don't know how to do anything else.
There's such a huge, huge gap between using ML to do the same thing everyone else is doing (say... image classification, or maybe regression on volume vs sales...) and solving unique business problems using ML.
Anybody recognizes how similar these questions are to what an investor would ask? This is what I'm always telling developers. You are an investor. You invest your life, health, best hours of 5+ days a week, ambition. Your investment is not as important to the company as a few million bucks would be, but to you it should be more valuable than a few million bucks. It's all you have. And you want to give it to someone in exchange for money, you better make sure that someone is worth your investment. So read what investors are asking a prospect company, and ask the same questions. Only accept if you would give them a million bucks.
Randy Pausch said it best (paraphrase), "You can always make more money, you can never get more time." It's the most valuable thing you have, and one way or another you are investing in a place where you work.
Let's give props to Seneca:
For we are mistaken when we look forward to death; the major portion of death has already passed. Whatever years be behind us are in death’s hands.
https://www.lettersfromastoic.net/letter-1/
For we are mistaken when we look forward to death; the major portion of death has already passed. Whatever years be behind us are in death’s hands.
https://www.lettersfromastoic.net/letter-1/
I could add another from Seneca: 'For many men, the acquisition of wealth does not end their troubles, it only changes them ' A compilation of some great Stoic quotes http://arandomquote.com/categories/stoic/
Thanks for reminding me. The Last Lecture is good. Since it's about 9 years ago, some younger HN readers may have missed it:
https://www.youtube.com/watch?v=ji5_MqicxSo
Cancer sucks.
Cancer sucks.
Very similar to Heielmeier questions: http://www.design.caltech.edu/erik/Misc/Heilmeier_Questions....
Pretty universal set of things to check for in new ideas.
Pretty universal set of things to check for in new ideas.
So, if I'm interviewing with a startup in AI, and asked all these questions, how many good answers I should expect? Is it ok for a startup to have a defensible business to solve a problem 10x better, knows how to make money in a big market, but have no experience in marketing and haven't talked to many potentially users?
As a wanna be startup founder, I found my ideas have bad answers to at least 2-3 of these questions.
As a wanna be startup founder, I found my ideas have bad answers to at least 2-3 of these questions.
> haven't talked to many potentially users?
Any bad answer to these questions is a showstopper, yes. It's an indicator that this company is definitely going to fail.
In your example, not having talked to many customers is the classical Juicero approach: Build a solution, then search for a problem that this solution is solving.
Not talking to customers means not understanding if there is demand for something, and if there are customers who would like to pay for a solution. (Problems exist in markets where target groups are not used or not willing to pay for digital solutions)
If you're starting out, I highly recommend reading the blog of Amy Hoy and Alex Hillmann (https://stackingthebricks.com/) - they defend the idea to first go on a "sales safari" where you simply obvserve your target group, and then find a solution for their pains.
As a founder, you want to continuously talk to potenttial customers about your idea or your prototype: make them use it and perform thinking-aloud tests.
Any bad answer to these questions is a showstopper, yes. It's an indicator that this company is definitely going to fail.
In your example, not having talked to many customers is the classical Juicero approach: Build a solution, then search for a problem that this solution is solving.
Not talking to customers means not understanding if there is demand for something, and if there are customers who would like to pay for a solution. (Problems exist in markets where target groups are not used or not willing to pay for digital solutions)
If you're starting out, I highly recommend reading the blog of Amy Hoy and Alex Hillmann (https://stackingthebricks.com/) - they defend the idea to first go on a "sales safari" where you simply obvserve your target group, and then find a solution for their pains.
As a founder, you want to continuously talk to potenttial customers about your idea or your prototype: make them use it and perform thinking-aloud tests.
Good points. But how many customers you talk to is enough? If I'm building a product for enterprise, is talking to one keen potential customer enough? For a first time founder, I can't imagine someone without much connections can get the chance to talk to multiple potential enterprise customers.
What about a product targeting the mass market, if I'm building a mobile app, is talking to a dozen friends enough? or should I pay for a market research which can reach hundreds of strangers.
Do you talk to customers with prototypes of the features or do you just have a basic skeleton and say 'what if I have this and that'?
I actually subscribed to Amy Hoy's mailing list but haven't checked it out for a while...
What about a product targeting the mass market, if I'm building a mobile app, is talking to a dozen friends enough? or should I pay for a market research which can reach hundreds of strangers.
Do you talk to customers with prototypes of the features or do you just have a basic skeleton and say 'what if I have this and that'?
I actually subscribed to Amy Hoy's mailing list but haven't checked it out for a while...
>> 2. How was this problem being solved before the AI came around? Was the pre-AI
“manual” solution good enough? Common answer: “we’re replacing humans.” That
isn’t enough. Often having a human is desirable (bedside manner, dexterity,
perfection a requirement). Often a human is affordable due to margin
structure. You’re looking to get a sense that the product provided is
something that was never possible before, 10X better, or just-as-good but 10X
cheaper. Not 20% cheaper. 10X.
That "10X" sounds like a bit of a heuristic, but is it a good one? Surely, what you're looking for is some assurance that the rewards from the use of machine learning justify its cost and that the profit the company can make out of it is higher than the profit they would be (or were) making without it?
It doesn't even matter even the profit is really much higher. In business, as in war (and, er, board games) any edge you can get is enough to push ahead. If a company can make even 1% more of what it did before, thanks to some new technology or whatever other trick, then that trick is worth pursuing in earnest.
That "10X" sounds like a bit of a heuristic, but is it a good one? Surely, what you're looking for is some assurance that the rewards from the use of machine learning justify its cost and that the profit the company can make out of it is higher than the profit they would be (or were) making without it?
It doesn't even matter even the profit is really much higher. In business, as in war (and, er, board games) any edge you can get is enough to push ahead. If a company can make even 1% more of what it did before, thanks to some new technology or whatever other trick, then that trick is worth pursuing in earnest.
I was going to say something about that my self.
I'm thinking it's more like this TED talk about something being remarkable: https://www.ted.com/talks/seth_godin_on_sliced_bread
I'm thinking it's more like this TED talk about something being remarkable: https://www.ted.com/talks/seth_godin_on_sliced_bread
Ai experts and practitioners are among smartest people you ever meet. Opportunity cost is huge
> "Reverse engineering from the technology to the market almost never works."
Completely disagree.
If the technology is creating some value for someone, and
if you have only one smart business dude that knows how to approach the right people and validate markets,
reverse engineering from technology to market _does_ work.
Source: Did that myself.
Completely disagree.
If the technology is creating some value for someone, and
if you have only one smart business dude that knows how to approach the right people and validate markets,
reverse engineering from technology to market _does_ work.
Source: Did that myself.
You can't really ask the hard questions during the interview phase. Before you get a job offer, they have all the leverage, and people don't like being asked hard questions when they're the ones interviewing.
Best thing to do is try to get a contact inside the company, then when you get the offer, call back and start asking those questions.
Even better still - don't agree to an interview unless you know in advance that the job will move your career in a good direction, then don't even bother asking questions. It's not like they have to tell you the truth anyway.
Best thing to do is try to get a contact inside the company, then when you get the offer, call back and start asking those questions.
Even better still - don't agree to an interview unless you know in advance that the job will move your career in a good direction, then don't even bother asking questions. It's not like they have to tell you the truth anyway.
"Before you get a job offer, they have all the leverage"
Only if you are in a situation where you need this job, right now. Try to avoid being in that situation. It isn't always possible, no, but there are reasonably steps you can take to try, such as prioritizing saving up a cushion over creature comforts and applying for multiple jobs.
And if you are good at a hot tech, which ML is right now, you probably do have some leverage just from rarity.
Yes, the perfectly free market of labor may be a theoretical construct, but don't give in to the despair that you have no control over your employment and that they have all the power. It'll become a self-fulfilling prophecy.
Only if you are in a situation where you need this job, right now. Try to avoid being in that situation. It isn't always possible, no, but there are reasonably steps you can take to try, such as prioritizing saving up a cushion over creature comforts and applying for multiple jobs.
And if you are good at a hot tech, which ML is right now, you probably do have some leverage just from rarity.
Yes, the perfectly free market of labor may be a theoretical construct, but don't give in to the despair that you have no control over your employment and that they have all the power. It'll become a self-fulfilling prophecy.
True; I mean to say until you have a job offer, you have no idea whether or not they are willing to negotiate anything with you. It's easy to fool yourself based on how friendly your interviewers are. So many times I've come out of the interview feeling like I got the job and then been told that they're going in a different direction.
During the "do you have any questions for me?" phase (which every good interviewer should make time for), these kinds of questions will make you stand out (in a good way) from the rest of the crowd. Just don't twist the thumbscrews, y'know?
https://www.youtube.com/watch?v=kzT3yfe2o-I