Ask HN: What does an ML Engineer do?
42 comments
> I was shocked to learn that my new employer has a very different job definition for ML Engineer than the one I was familiar with.
Job titles do not have consistent meaning across regions or industries. People use buzzwords and hype to recruit funding and talent.
You've learned an important lesson: use interviews to gather information about companies and teams. "Can you describe in broad strokes a typical project for this role?" is a perfectly reasonable question to ask a hiring manager.
Job titles do not have consistent meaning across regions or industries. People use buzzwords and hype to recruit funding and talent.
You've learned an important lesson: use interviews to gather information about companies and teams. "Can you describe in broad strokes a typical project for this role?" is a perfectly reasonable question to ask a hiring manager.
This 1000%.
I was an ML engineer, and to me it was far more about data management/cleaning than nitty gritty algorithms improvements, but that's definitely not industry standard.
I've also found "Product Engineer" job listings to range from essentially just a backend engineer to a product designer who codes.
I was an ML engineer, and to me it was far more about data management/cleaning than nitty gritty algorithms improvements, but that's definitely not industry standard.
I've also found "Product Engineer" job listings to range from essentially just a backend engineer to a product designer who codes.
This. I would add that role titles also exist for negotiation purposes. For example, a new employee may prefer to be called "research engineer" instead of "ML DevOps" to open up future career opportunities, even if the work done is the same.
If it were my business or my team I would want an ML Engineer to relentlessly knock down any barriers to getting models into production.
The deployment of ML is still per-paradigmatic in that people today deploy products based on luck, blood, sweat and tears. Many projects fail and many more never get started.
I am sure that somebody somewhere had a team that gets results consistently, I can partially picture what it would take to do so, but that's not the usual condition. (That team is so productive that it doesn't need to hire more people!)
For instance, you might not want to do any of these things, but any of them could be essential to putting something in production:
People who do these things today will be able to say tomorrow that they were part of a successful team. Other people will expect great praise because they diddled around with Word2Vec for a summer.
Some day the social structures will be in place and they might have a role made for you that will make you productive but for now you must ask: who do you want to be?
The deployment of ML is still per-paradigmatic in that people today deploy products based on luck, blood, sweat and tears. Many projects fail and many more never get started.
I am sure that somebody somewhere had a team that gets results consistently, I can partially picture what it would take to do so, but that's not the usual condition. (That team is so productive that it doesn't need to hire more people!)
For instance, you might not want to do any of these things, but any of them could be essential to putting something in production:
* manually classifying 10,000 images or documents
* figuring out how to run a program in the Java and C++ debuggers at the same time
* be able to convince somebody that their research programme is a dead end
* survive a 40 minute build process
* push back against a 40 minute build process and reform it
* reproduce a result from a scientific paper
* get a "data scientist" (who thinks he is too good to listen to you, or even to the CEO) to standardize their python distribution so their work isn't wasted
* find the scalability bottleneck in a system, where it is, not where you want it to be.
People who do these things today will be able to say tomorrow that they were part of a successful team. Other people will expect great praise because they diddled around with Word2Vec for a summer.
Some day the social structures will be in place and they might have a role made for you that will make you productive but for now you must ask: who do you want to be?
I overall agree with the sentiment on delivery and needing to deal with a variety of issues, but one nit:
If it were my business or my team I would want an ML Engineer to relentlessly knock down any barriers to getting models into production.
My favorite is when they knock down the question of whether a certain project even needs ML to focus on getting ML into production.
Many teams fail at ML because there was an essential task that nobody wanted to do that didn't get done.
Many teams fail at ML because there was a task that didn't need ML (or at least anything more than a linear model) that is made opaque to anyone but an ML engineer after they implement it.
If it were my business or my team I would want an ML Engineer to relentlessly knock down any barriers to getting models into production.
My favorite is when they knock down the question of whether a certain project even needs ML to focus on getting ML into production.
Many teams fail at ML because there was an essential task that nobody wanted to do that didn't get done.
Many teams fail at ML because there was a task that didn't need ML (or at least anything more than a linear model) that is made opaque to anyone but an ML engineer after they implement it.
> * manually classifying 10,000 images or documents *
Ah, the 50's era sweatshop mentality! Most excellent. I would fire a person who wasted days doing such a stupid operation. Take 10 minutes, do some research, find a better way. Spend some company money to get it done today. If you can't figure a good way, start asking up the chain.
Ah, the 50's era sweatshop mentality! Most excellent. I would fire a person who wasted days doing such a stupid operation. Take 10 minutes, do some research, find a better way. Spend some company money to get it done today. If you can't figure a good way, start asking up the chain.
Very curious how you think ML without data is going to work. 10 minutes of research will neither give you a solution to that nor will it procure the data needed.
In the end, ground truth data is almost always going to originate from humans putting in man-hours to create it, one after the other. Whether those humans need to be ones that are paid ML Engineer salaries for this is an important business question, but someone will have to do the work.
In the end, ground truth data is almost always going to originate from humans putting in man-hours to create it, one after the other. Whether those humans need to be ones that are paid ML Engineer salaries for this is an important business question, but someone will have to do the work.
Yup.
People complained about the cost of getting an expert to cough up secrets about their thought process in the 1970s.
The essential problem is not different in magnitude with ML because you need to design the classification to be solvable and then get the right answers for many hard cases.
From an academic perspective ML is advantaged in that multiple people can train algos on the same data, which leads to a certain kind of progress through competition.
When it comes to most projects where you need the model you have to supply your own training set. Sometimes there is a download or a weird trick, but that conversation that ‘we can’t afford you to make the training set’ leads to bad outcomes:
The team can still win the game by default in 2021 because your competitor thinks it is too good to do those things.
People complained about the cost of getting an expert to cough up secrets about their thought process in the 1970s.
The essential problem is not different in magnitude with ML because you need to design the classification to be solvable and then get the right answers for many hard cases.
From an academic perspective ML is advantaged in that multiple people can train algos on the same data, which leads to a certain kind of progress through competition.
When it comes to most projects where you need the model you have to supply your own training set. Sometimes there is a download or a weird trick, but that conversation that ‘we can’t afford you to make the training set’ leads to bad outcomes:
* you never get a training set
* you download the wrong data from Kaggle
* you get an unusable training set
* you wind up spending more time of expensive people herding inexpensive people, when….
The team can still win the game by default in 2021 because your competitor thinks it is too good to do those things.
My subjective view coincides with your new employer. I think that's pretty common.
ML Scientist and Data Scientist versus ML Engineer and Data Engineer.
ML Scientist and Data Scientist versus ML Engineer and Data Engineer.
Same at my fairly large research center. We also use ML researcher to denote folks who are hands on with model building and algorithms.
Sounds like their title for the job should’ve been ML Infra or ML DevOps engineer to indicate it’s a supporting role versus a primary practitioner. Annoying, sorry OP.
I'm on the same boat, but about SRE.
Basically every company have their own definition. For some companies, a SRE is just a traditional software engineer who happens to know how to use Kubernetes, where for others, a SRE is a "modern systems guy", with focus on IaC and automation.
Guess you're experiencing the same but regarding ML.
Basically every company have their own definition. For some companies, a SRE is just a traditional software engineer who happens to know how to use Kubernetes, where for others, a SRE is a "modern systems guy", with focus on IaC and automation.
Guess you're experiencing the same but regarding ML.
And then there are many companies where there are still silos, almost no automation, systems engineers are more focused on machines than the product and the company advertises SRE role- simply because its the trend.
Designing the systems that run the 'ML pipeline', or end to end data collection > train -> deploy -> infer -> improve. I don't see 'ML Engineers' as advancing the state of the art but they should be fluent in recent techniques, know how to apply them, optimize them, know hardware is needed to efficiently train, etc.
I think it often depends on how large the team is. On a team with many developers per model, ML Engineer to me means someone who does the engineering to support models and is distinct from a researcher or a scientist. ML engineers will not do the science side of the equation (model architectures, optimizing scientific metrics like accuracy) except for when their engineering work impacts the science side (e.g. figuring out how to optimize model throughput without impacting scientific metrics).
On smaller teams, the titles mean less - but I rarely see ML engineer positions for small teams.
ML involves both science and engineering. ML engineer by definition sits on the engineering side of the fence, but how much of the science they are involved in will vary by company.
On smaller teams, the titles mean less - but I rarely see ML engineer positions for small teams.
ML involves both science and engineering. ML engineer by definition sits on the engineering side of the fence, but how much of the science they are involved in will vary by company.
How did you get through a job description and the interview process without this coming up? Was it a bait and switch?
The job description was pretty ambiguous, and the interview consisted of pretty standard ML/math questions with some data engineering mixed in. There was one brief offhand comment during the interviews that confused me about the role, but since this was a much larger and more established tech company than my previous job, and since the salary was much higher, I figured that they knew what they were doing and also there was no way they could justify that kind of compensation for someone who wasn't an ML practitioner.
What I am finding out is that this is a very large team of data engineers who are attempting to graduate to ML engineers, but I'm not sure how familiar they are with this space. The purpose of their organization seems to be to support an offshore data science team in Singapore.
What I am finding out is that this is a very large team of data engineers who are attempting to graduate to ML engineers, but I'm not sure how familiar they are with this space. The purpose of their organization seems to be to support an offshore data science team in Singapore.
It may be definition differences between the two companies but here's the first thing that came to mind: by any chance is your new employer a much larger organization? In smaller companies you get to wear many hats and typically have responsibility over a wider stretch of <work/tools/technologies>. In larger companies things tend to be much more siloed. So in a larger organization, the things you did in your previous position would likely be spread across multiple positions and teams.
My understanding is ML engineer gets machine learning models running in production. They would own the infrastructure that enables teams to deploy ML models, interface with engineering teams to agree on contracts/endpoints/UI/etc, and work on scaling models when/if that time comes. I would view Data Scientist and ML engineer as distinct sister roles, and not as a subset.
I've worked in environments where ML engineers were "data science plus", and those were really unhealthy work environments - it ended up feeling like a prestige war with lots of cookie licking and turfing. When you're at the point of needing differentiated roles, you really need someone to focus on infrastructure to deploy ML models and someone else to focus on managing stakeholders, wrangling data, and developing proof of concepts.
I've worked in environments where ML engineers were "data science plus", and those were really unhealthy work environments - it ended up feeling like a prestige war with lots of cookie licking and turfing. When you're at the point of needing differentiated roles, you really need someone to focus on infrastructure to deploy ML models and someone else to focus on managing stakeholders, wrangling data, and developing proof of concepts.
At our company, ML Engineers do both the modeling work and the larger engineering that allows those models to interact with the product. So all the models are built by ML engineers, as well as a lot of the production data engineering that gets any needed data to and from the models in production.
Those kinds of full stack ML engineers feel increasingly rare to me. Partly because people from academic backgrounds sometimes lack software engineering knowledge, and vice versa. Partly because it’s just a lot of stuff to do well.
Those full stack roles are still relatively common (though decreasingly so, as you pointed out) at companies with small data science orgs, on the order of 10 people. But they're still usually called "data scientist," while I think of more specialized roles like MLE, ML ops, data engineer, data ops, etc. as cropping up in bigger and/or more mature data science teams. Calling it MLE instead of data scientist could be a screening technique though.
That's essentially the job role that I'm familiar with, however this new role is everything except for feature and model engineering.
I would say, some combinations of:
- Helping with feature engineering at scale
- Putting models into production
- All of the environmental stuff (Framework versions, security patching, etc)
- Monitoring models and model drift
- Supporting hot/swapping models with zero downtime
- Model persistence, A/B testing and evaluation in production
- Distributed hyperparameter tuning
- Helping with feature engineering at scale
- Putting models into production
- All of the environmental stuff (Framework versions, security patching, etc)
- Monitoring models and model drift
- Supporting hot/swapping models with zero downtime
- Model persistence, A/B testing and evaluation in production
- Distributed hyperparameter tuning
I work for a company where half the company consists of ML Developers. They match your definition. Your job is what we call ML Ops.
Kind of interested how you got hired to a completely different role though and how that didn't come up in the interview.
Kind of interested how you got hired to a completely different role though and how that didn't come up in the interview.
In my experience, if you have a PhD you can get a job working with the models. It will likely be called something fancier.
Without the PhD, I feel like you're just a data jockey. Just clean the data and create APIs for the PhDs and consumers to use.
Without the PhD, I feel like you're just a data jockey. Just clean the data and create APIs for the PhDs and consumers to use.
Most of the time they allow the company to say they work on this new flashing buzzword tech while burning through capital doing stuff a few if else statements could easily do
>... never works on machine learning directly.
Meaning, does not push the train button and watch loss curves?
What's out of this role's scope at the new company?
Meaning, does not push the train button and watch loss curves?
What's out of this role's scope at the new company?
At my last role I designed the forecasting systems for a major retail chain, which required inventing a new deep learning model architecture for time series problems, due to some peculiar properties of their sales data. I take great pride in my work.
I don't understand this answer. All of these activities are out of scope in your current role?
Do you know of any publicly available resources that walk through some intuitions in using forecasting systems in retail sales? Basically a dummy's guide that helps non-tech retail leaders better understand how this works, how accurate it can be, and when is appropriate to use it (and when it isn't).
For business users, unfortunately none on hand at the moment. That being said, one of the biggest issues companies seem to encounter is that forecasting requires logging of at least several years of business operations, especially if the target variable is highly seasonal. The need for sufficient historical data can make it difficult to commit to a new forecasting initiative at the drop of a hat. There first needs to be a culture of retaining as much data possible, since even external variables from other business areas may be useful in making predictions. You can get around this somewhat with synthetic data and piecing together alternative data sources, but it's not ideal.
I would also say the ease of implementing a forecasting solution on a given dataset is inversely proportional to the value it provides. They're mainly useful for situations in which there are either too many independent time series for a group of human business analysts to track, or for situations in which the target variable is unruly for some reason. For forecasting problems that are simple enough to be modeled by state space methods or BSTS, those methods won't be able to provide results that are all that superior to what an experienced analyst could produce, since the series data itself is already quite predictable.
Edit: Also, there is no one-size-fits-all approach when it comes to time series, and anyone who tells you differently is trying to sell you something. Time series is weird within ML because each dataset has unique properties that necessitates the use of potentially wildly different modeling techniques.
I would also say the ease of implementing a forecasting solution on a given dataset is inversely proportional to the value it provides. They're mainly useful for situations in which there are either too many independent time series for a group of human business analysts to track, or for situations in which the target variable is unruly for some reason. For forecasting problems that are simple enough to be modeled by state space methods or BSTS, those methods won't be able to provide results that are all that superior to what an experienced analyst could produce, since the series data itself is already quite predictable.
Edit: Also, there is no one-size-fits-all approach when it comes to time series, and anyone who tells you differently is trying to sell you something. Time series is weird within ML because each dataset has unique properties that necessitates the use of potentially wildly different modeling techniques.
Thanks for the reply. I actually think you might be in a good position to blog about this given your experience. There is definitely a huge segment of business leaders who could benefit from understanding these concepts in a non-tech way, from someone like yourself who has implemented these types of systems.
I'm not sure there is a general introduction, because forecasting in retail sales is extremely dependent on how the forecasts are used, and there are many schools of thought about that.
There's the classic "inspired from manufacturing" approach where you have production targets for your factory, and your planning tool decides how much to purchase from suppliers, and when. Transposed to retail sales, you obviously don't have production targets, so you use forecasting (usually average or median forecasts, daily or weekly), inflate the amounts a bit for "safety stock", and pretend that the forecasts are the targets. You'll usually notice this is the case if the retailer has a Forecasting department that is separate from its Purchasing department, and forecasts are judged on a metric that compares them to actual values (rather than on the outcomes of the decisions made based on them).
Once you allow your forecast method to return something other than time series, you can adapt it to the actual supply chain, and you have a lot more variety depending on how purchasing (and pricing!) decisions are made.
For instance, if you order every week from a given supplier, you instead want to forecast the total sales over the coming week, given as a probability distribution ("there is a P% chance of selling more than Q units). You can then use the actual dollar costs of not having enough units (the margin of a single sale) and of having too many units (storage, carrying and opportunity costs). This lets you optimize dollars instead of how close the forecast was to an actual observation.
There are many levers that can be driven by forecasts, such as raising prices if there isn't enough inventory to serve the demand before the supplier's next delivery, or lowering prices to make room in a warehouse for higher-margin products, or selecting which product appears on the front page of your e-commerce website or in the default position of a configurator drop-down.
There are also limitations, usually around data availability. I would say most companies _that can pay for forecasting development_ have an extensive history of sales data. However, you likely also need information about suppliers (how long do they take to deliver?), about historical stock levels (this product sold zero units over an entire month, was it because it was out-of-stock, or is this a relevant signal for our forecasts?), about price changes and promotional events (usually to explain a sudden jump in sales), and so on. These are not always available. And then, there are entire industries (such as fashion) where individual products never have more than a few months of historical data, because there's a new collection coming out every season.
If you're interested, my employer is producing a "Supply Chain Lectures" series on YouTube which deals with forecasts among other things (but really, the idea is to look at supply chains as a whole, instead of just forecasting): https://www.lokad.com/lectures
There's the classic "inspired from manufacturing" approach where you have production targets for your factory, and your planning tool decides how much to purchase from suppliers, and when. Transposed to retail sales, you obviously don't have production targets, so you use forecasting (usually average or median forecasts, daily or weekly), inflate the amounts a bit for "safety stock", and pretend that the forecasts are the targets. You'll usually notice this is the case if the retailer has a Forecasting department that is separate from its Purchasing department, and forecasts are judged on a metric that compares them to actual values (rather than on the outcomes of the decisions made based on them).
Once you allow your forecast method to return something other than time series, you can adapt it to the actual supply chain, and you have a lot more variety depending on how purchasing (and pricing!) decisions are made.
For instance, if you order every week from a given supplier, you instead want to forecast the total sales over the coming week, given as a probability distribution ("there is a P% chance of selling more than Q units). You can then use the actual dollar costs of not having enough units (the margin of a single sale) and of having too many units (storage, carrying and opportunity costs). This lets you optimize dollars instead of how close the forecast was to an actual observation.
There are many levers that can be driven by forecasts, such as raising prices if there isn't enough inventory to serve the demand before the supplier's next delivery, or lowering prices to make room in a warehouse for higher-margin products, or selecting which product appears on the front page of your e-commerce website or in the default position of a configurator drop-down.
There are also limitations, usually around data availability. I would say most companies _that can pay for forecasting development_ have an extensive history of sales data. However, you likely also need information about suppliers (how long do they take to deliver?), about historical stock levels (this product sold zero units over an entire month, was it because it was out-of-stock, or is this a relevant signal for our forecasts?), about price changes and promotional events (usually to explain a sudden jump in sales), and so on. These are not always available. And then, there are entire industries (such as fashion) where individual products never have more than a few months of historical data, because there's a new collection coming out every season.
If you're interested, my employer is producing a "Supply Chain Lectures" series on YouTube which deals with forecasts among other things (but really, the idea is to look at supply chains as a whole, instead of just forecasting): https://www.lokad.com/lectures
Great info, thanks for sharing! Will certainly check out the lectures.
I think it depends on the company. In my experience most companies have it halfway between the things you described (some data engineering + some modeling). Companies with less established infra generally have more data engineering work.
Imo unless you are a rich company or have a well funded research arm, it mostly seems wasteful to make someone's job pure modeling.
Imo unless you are a rich company or have a well funded research arm, it mostly seems wasteful to make someone's job pure modeling.
Pure modelling teams are a pointless waste of space. It's entirely analogous to setting up an SQL team and then telling everyone else not to run queries.
More generally, in order to provide useful solutions, you need some domain and data expertise, which tends to come from engaging with product, the business and the data.
If you don't have that, you may as well replace your modelling team with a giant loop over the sklearn fit and predict API.
More generally, in order to provide useful solutions, you need some domain and data expertise, which tends to come from engaging with product, the business and the data.
If you don't have that, you may as well replace your modelling team with a giant loop over the sklearn fit and predict API.
We call that role ML Ops at our work. It's a mix of data eng and dev ops.
[deleted]
I'd say what you've mentioned are the 2 typical definitions.
Either its an expert at building and training ml models, or its an expert at putting ml models in production.
Either its an expert at building and training ml models, or its an expert at putting ml models in production.
Yeah I think in general “ML engineers” are building models (can include feature engineering) and then deploying them into production.
Its whatever you want it to be, which is the greatest thing.
Ex ML Engineer at a company.
Ex ML Engineer at a company.
What is the generally accepted definition of an ML Engineer?