I became a software engineer after 5 years in data science(zachary-thomas.medium.com)
zachary-thomas.medium.com
I became a software engineer after 5 years in data science
https://zachary-thomas.medium.com/how-i-became-a-software-engineer-after-5-years-in-data-science-e0c56b872257
57 comments
Yeah that's fair -- maybe just focusing on becoming an expert in data and ML engineering in the long-run is a more prudent career choice than totally dropping those for another corner of engineering. There's some wisdom here.
Don’t take it as discouragement from being a generalist, in er… general.
But strike where the iron is hot.
Right now the future is ML, not learning the intricacies of the 2022 tech stack.
Heck, I’m an automation enthusiast, but if that’s my only trick and I’m good at it, I’m out of work like next week ;)
But strike where the iron is hot.
Right now the future is ML, not learning the intricacies of the 2022 tech stack.
Heck, I’m an automation enthusiast, but if that’s my only trick and I’m good at it, I’m out of work like next week ;)
I will add that certain SWE fields rely upon and interact heavily with DS and for anyone wanting to make the switch, this is a natural "in".
Any company with a rigorous approach to product development or even SRE will rely heavily on metrics. The ability to create, maintain, monitor, transform, understand and reason about those metrics is invaluable. To decide what to work on and the quantify your contribution, the ability to effectively run experiments is invaluable. This will naturally fit into a structured product management cycle.
This will be a way more natural fit than, say, "purer" software development (eg writing device drivers).
So much of successful software engineering is not just how to do something but deciding what to do (and not do).
Any company with a rigorous approach to product development or even SRE will rely heavily on metrics. The ability to create, maintain, monitor, transform, understand and reason about those metrics is invaluable. To decide what to work on and the quantify your contribution, the ability to effectively run experiments is invaluable. This will naturally fit into a structured product management cycle.
This will be a way more natural fit than, say, "purer" software development (eg writing device drivers).
So much of successful software engineering is not just how to do something but deciding what to do (and not do).
Yes -- one thing I struggle with is choosing whether to work on data engineering projects (which I know I'm good at and can provide immediate value) or challenging myself with more "traditional" full-stack engineering work where I can learn a lot.
Maybe a 60/40 mix is optimal
Maybe a 60/40 mix is optimal
I was hoping for more advice for those pursuing this transition, but it was more just an account of Google's incredible support for his wish to change careers. He never mentions his employer by name, but I think it'd be helpful to point this out. Those of us on the outside (I and a few other DS's I know are doing this) are spending 6 months or more grinding leetcode and studying system design, doing mock interviews, and applying to hundreds of jobs. So it can be much more painful if you don't have an employer that holds your hand (and pays you) through this entire transition.
You're right, Google has been very supportive -- very thankful for that! That said, if you work at a technology company, you CAN build a portfolio and self-advocate for projects with engineering teams your data science team already works with.
I think it's worth seeing how far you can get with this approach before interviewing for engineering roles at other companies.
One notable caveat with my approach: pursuing a portfolio approach took me a whole year to switch from DS to engineering. As you point out, grinding leetcode may only require half that time.
I think it's worth seeing how far you can get with this approach before interviewing for engineering roles at other companies.
One notable caveat with my approach: pursuing a portfolio approach took me a whole year to switch from DS to engineering. As you point out, grinding leetcode may only require half that time.
ME (at not-Google): I wanna do a 12-week dev bootcamp!
MY LEAD: Uhhhh... What if it was two weeks long, tops? Or even less?
MY ASSIGNED HR PERSON: Great, let's process your leave of absence! We'll cut your benefits and convert your stock options so you'll have to buy them out a lot sooner than you were prepared to!
MY LEAD: Uhhhh... What if it was two weeks long, tops? Or even less?
MY ASSIGNED HR PERSON: Great, let's process your leave of absence! We'll cut your benefits and convert your stock options so you'll have to buy them out a lot sooner than you were prepared to!
Taking that much time off of any job is pretty hard to do!
The real goal of a bootcamp though (besides networking) is adding a project to your portfolio. You can then show off this project to hiring managers to demonstrate your competence.
You don't need to do a full-time bootcamp to make this happen! I just checked and Flatiron school and Hack Reactor both have online part-time full-stack bootcamps. Also, while not as big of an investment, online platforms like Codecademy have career paths with a capstone project you can complete and add to your portfolio.
So, overall there are a variety of pathways to do this. Totally fair to point out though a bootcamp of some kind is a nice option in the off-chance it's a possibility.
The real goal of a bootcamp though (besides networking) is adding a project to your portfolio. You can then show off this project to hiring managers to demonstrate your competence.
You don't need to do a full-time bootcamp to make this happen! I just checked and Flatiron school and Hack Reactor both have online part-time full-stack bootcamps. Also, while not as big of an investment, online platforms like Codecademy have career paths with a capstone project you can complete and add to your portfolio.
So, overall there are a variety of pathways to do this. Totally fair to point out though a bootcamp of some kind is a nice option in the off-chance it's a possibility.
I didn't end up doing a bootcamp in the end. I sort of wish I did: friends of mine who did ended up with some strong connections as a result. Then again, someone else I know had an extremely negative experience with Hack Reactor and lost thousands of dollars in it, so maybe it's for the best that I didn't go.
[deleted]
Hey everyone, OP here -- thanks for reading or discussing.
Curious if anyone else has switched from data science to software engineering, or has thoughts on comparing the two as a career choice. I'm all ears!
Curious if anyone else has switched from data science to software engineering, or has thoughts on comparing the two as a career choice. I'm all ears!
I have always been an engineer, but when I started as an intern in 2015, most of my work centered around helping analyses: cleaning up CSVs, data mining, minor processing and transformations, etc. None of it was connected to production but these were ad-hoc requests. I think this start was invaluable to my general philosophy today (devops or die), and believe there are a lot of missed opportunities identifying the right analysts that should take the next step into complexity and help craft better devops processes for analysts. They're developers too.
One thing I don't think gets talked about enough: How much time analysts spend doing analysis vs. time spent dealing with data quality/architecture/process. So much analyst time is lost in the latter, which I think contributes greatly to burnout. But the growing intersection with engineering has and will continue to address this, albeit slowly.
One thing I don't think gets talked about enough: How much time analysts spend doing analysis vs. time spent dealing with data quality/architecture/process. So much analyst time is lost in the latter, which I think contributes greatly to burnout. But the growing intersection with engineering has and will continue to address this, albeit slowly.
I run a small data consulting company and whenever I find an open minded data scientist or analyst I tell them to consider data and software engineering or business analytics (learning about real business problems). DA, BA and SE roles are harder to find, manage more complexity, but most importantly are closer to having an impact as they are the ones close to prod. It’s a hard pitch, especially for juniors, AI and DS are very hyped but it’s hard to guess how much they will grow (or not). My current hunch is that AI and DS will be productized in libraries and services, so the majority of companies will not need people fully dedicated to build models.
Yep. In any case, building a model is a just part of the puzzle -- serving it in production is a whole other skillset that is very valuable. imho
My educational background is much closer to data science (Math with CompSci Minor B.Sc., Data Science M.Sc.) but I've never worked in a pure data science role.
Training and tweaking models looks like the easy part of developing data driven products. Hiring compenent enough people also seems easier than for software engineers.
Many ML libraries produce good enough results without having to design elobare models myself. I see data science more as yet another tool in the belt of a a software engineer - like server admin, CI/CD, IaC, databases,...
Training and tweaking models looks like the easy part of developing data driven products. Hiring compenent enough people also seems easier than for software engineers.
Many ML libraries produce good enough results without having to design elobare models myself. I see data science more as yet another tool in the belt of a a software engineer - like server admin, CI/CD, IaC, databases,...
Sorry for the tangent, but medium is so bad. Can’t read content without installing the app or logging in. Part of the reason I stopped writing on medium and jumped to Substack.
Oh, I'm sorry about that! Here's a friend link that should allow you to bypass any paywall (if you've already read a couple paywalled posts): https://zachary-thomas.medium.com/how-i-became-a-software-en...
Here's an earlier draft of the post on Quip: https://gstudent.quip.com/ILF1ABpvjsSh/Can-a-Data-Scientist-...
Here's an earlier draft of the post on Quip: https://gstudent.quip.com/ILF1ABpvjsSh/Can-a-Data-Scientist-...
Thank you for sharing!
Unless studied software engineering, you can switch to become an "engineer"? Wouldn't be software developer term more fitting?
Upvoted not because I agree, but because I'm genuinely interested in the discussion. What is the difference in your opinion?
Software engineering is a branch of engineering in many countries. Engineering follows the engineering design process [1] and uses applied science. So, to me, people doing software engineering (regardless of laws / licenses) are following an approximation of the engineering design process. A software developer is anyone getting paid to write or produce software regardless of approach or knowledge of CS topics.
On a spectrum, there are software developers that follow the engineering design process and use all sorts of knowledge of applied science, but there are also those who make it all up as they go along and rediscover or rename concepts independently. Each approach comes with its own associated benefits and tradeoffs, and there are times and places where the latter end of the spectrum can be desirable.
At least that's my hot take.
[1] https://www.mcgill.ca/engineeringdesign/step-step-design-pro...
On a spectrum, there are software developers that follow the engineering design process and use all sorts of knowledge of applied science, but there are also those who make it all up as they go along and rediscover or rename concepts independently. Each approach comes with its own associated benefits and tradeoffs, and there are times and places where the latter end of the spectrum can be desirable.
At least that's my hot take.
[1] https://www.mcgill.ca/engineeringdesign/step-step-design-pro...
Just my own idiolect but I see SREs and the related competencies and practices as software engineers, whereas I see "programmers" as more of a creative endeavour and "developer" as on the spectrum between programmer and SRE
This only opinion piece on the issue worth reading, because it's the only one that bothered to seek the opinions of the people that would know best: those people that have been software engineers _and_ some other kind of engineer.
https://www.hillelwayne.com/post/are-we-really-engineers/
https://www.hillelwayne.com/post/are-we-really-engineers/
I think I'm missing something, but why is an opinion piece needed on this issue? Software engineering is a licensed engineering profession in many places. It used to be possible to be licensed as a software engineering in the United States [1, 2](little demand caused the licensing exams to be discontinued, possibly because regulation wasn't enforced, according to [2]). So, I think it's fair to say enough of the engineering profession has spoken to say software is a branch of engineering.
What I think are the relevant extracts from the article:
> “Do you consider software engineering actually engineering?”
> Of the 17 crossovers I talked to, 15 said yes.
> That said, many of the crossovers [3] also added an additional qualification: software engineering is real engineering, but a lot of people who write software aren’t doing software engineering. This is not a problem with them, rather a problem with our field: we don’t have a rich enough vocabulary to talk about what these developers do.
The engineering field has developed a vocabulary for different professionals in the field. Why can't this be applied to the software field since it is a form of engineering? There are engineers, technologists, technicians, and trades (at least where I am). The issue is there are education and work experience requirements that lead to licensing, which people will say are inequitable and gatekeeping [4]. From 2, there was an experience-only path available to get software engineering licensure, so if the field really wants it, there seems to be precedent to allow for alternative routes into those titles.
[1] https://ncees.org/engineering/pe/software/
[2] https://www.nspe.org/resources/pe-magazine/may-2018/ncees-en...
[3] The definition of "crossovers:" "people who used to be professional engineers and then became professional software developers. I call these people crossovers, hybrids between the two worlds."
[4] I'm sympathetic to these concerns. I studied engineering technology and worked as a technologist at the beginning of my career. I wanted to become a mechanical engineer, but I couldn't afford to return to school full time to complete the remaining 2 years for the BEng. (My school offered a 2 year diploma for engineering technologists, which led directly to years 3 and 4 year of the BEng.) When I wanted to complete the degree, a full time course load was required (7 courses per term at my school) for the engineering program to maintain accreditation.
What I think are the relevant extracts from the article:
> “Do you consider software engineering actually engineering?”
> Of the 17 crossovers I talked to, 15 said yes.
> That said, many of the crossovers [3] also added an additional qualification: software engineering is real engineering, but a lot of people who write software aren’t doing software engineering. This is not a problem with them, rather a problem with our field: we don’t have a rich enough vocabulary to talk about what these developers do.
The engineering field has developed a vocabulary for different professionals in the field. Why can't this be applied to the software field since it is a form of engineering? There are engineers, technologists, technicians, and trades (at least where I am). The issue is there are education and work experience requirements that lead to licensing, which people will say are inequitable and gatekeeping [4]. From 2, there was an experience-only path available to get software engineering licensure, so if the field really wants it, there seems to be precedent to allow for alternative routes into those titles.
[1] https://ncees.org/engineering/pe/software/
[2] https://www.nspe.org/resources/pe-magazine/may-2018/ncees-en...
[3] The definition of "crossovers:" "people who used to be professional engineers and then became professional software developers. I call these people crossovers, hybrids between the two worlds."
[4] I'm sympathetic to these concerns. I studied engineering technology and worked as a technologist at the beginning of my career. I wanted to become a mechanical engineer, but I couldn't afford to return to school full time to complete the remaining 2 years for the BEng. (My school offered a 2 year diploma for engineering technologists, which led directly to years 3 and 4 year of the BEng.) When I wanted to complete the degree, a full time course load was required (7 courses per term at my school) for the engineering program to maintain accreditation.
Sure! My company calls the role "software engineer" so that's how I describe the role but I think developer is also a good word for it
oh no not this tired tangent again.
The ramblings of someone who’s low skill and low ambition usually
it's not clear to whom you are referring
People obsessed with protecting the term “engineer” are usually those who want to gatekeep their jobs and salary. In other words, they are worried about competition and want to keep the career “prestigious” and kick the ladder away.
It is a bit facetious to say that all analysts are engineers, or that it is "easy" for any analyst to become an engineer.
There is overlap, but they are not identical skillsets.
Analysts are certainly developers, but engineering involves more of a sense of scalability, analysis can do this, but it is generally more granular.
There is overlap, but they are not identical skillsets.
Analysts are certainly developers, but engineering involves more of a sense of scalability, analysis can do this, but it is generally more granular.
I’m under the impression that programmers’ salary progression typically compares quite favorably to classical engineers’.
We also have support engineers with only highschool diploma! And account managers...
Software Engineer and Software Developer are used interchangeably for the most part and I don't think you can really get any meaningful distinction just by the name (in most countries at least).
Some countries (Canada I think being one?) may require you be licensed/registered with an engineering body before you're allowed to call yourself an engineer, but thats not the case in most countries AFAIK
Some countries (Canada I think being one?) may require you be licensed/registered with an engineering body before you're allowed to call yourself an engineer, but thats not the case in most countries AFAIK
Indeed. I worked at a Canadian company that got a legal nastygram from the professional association saying to stop calling us engineers.
Then I started working for an American company that was like shrug no laws against that.
I just can’t say professional engineer.
The whole thing makes a ton of sense when we’re building bridges or airplane safety rated computer systems, but not so much when building websites or whatnot.
Then I started working for an American company that was like shrug no laws against that.
I just can’t say professional engineer.
The whole thing makes a ton of sense when we’re building bridges or airplane safety rated computer systems, but not so much when building websites or whatnot.
I work on critical software and have the 'engineer' label, but do have a nagging feeling it's less structured than 'normal' engineering disciplines. I'm taking a part-time BEng to see if that's the case though!
You can definitely call yourself a professional software engineer in the US.
The US doesn't have software "engineering" degrees, so how does one exactly study software engineering? Computer science, too, is not "engineering".
There are colleges and universities in the USA that offer SE degrees, CMU is of note[0]
[0] https://www.ece.cmu.edu/academics/ms-se/index.html
[0] https://www.ece.cmu.edu/academics/ms-se/index.html
The UK does, you can get chartered engineer status with a BSc & MSc in Software Engineering or Computer Science if it's applied enough with the IET. When I looked at the US i'm 99% sure I found similar degrees in Software Engineering too.
I’ve never seen a “software engineering” degree on a resume as an interviewer in the us.
Fwiw, I have. They're becoming more common - and probably a good thing too, covering important non-CS skills like requirements engineering that are mostly made up in small tech companies without that experience.
It does…
NCEES even offered a Professional Engineering license in it for awhile. Since discontinued due to lack of interest.
https://ncees.org/ncees-discontinuing-pe-software-engineerin...
https://ncees.org/ncees-discontinuing-pe-software-engineerin...
If you do any sort of measurement, analysis, or design before implementing a system, you're leveraging engineering principles.
Examples: measuring data scale for growth rate and future needs, calculating incremental costs of cloud instances and data stores, designing and proving a state machine, instrumenting systems, measuring and stress testing a system against load, understanding back pressure and dynamical behaviors of distributed systems, designing threaded or active/active systems, vector clocks, consensus algoritms, test suites, etc. etc.
I've had to write short inductive proofs in several of the systems I've built.
It's engineering. The sooner we get over the imposter syndrome debate of what we can and cannot call ourselves and embrace the full scope and possibilty of what we can achieve, the sooner we can become better and more capable software engineers.
Examples: measuring data scale for growth rate and future needs, calculating incremental costs of cloud instances and data stores, designing and proving a state machine, instrumenting systems, measuring and stress testing a system against load, understanding back pressure and dynamical behaviors of distributed systems, designing threaded or active/active systems, vector clocks, consensus algoritms, test suites, etc. etc.
I've had to write short inductive proofs in several of the systems I've built.
It's engineering. The sooner we get over the imposter syndrome debate of what we can and cannot call ourselves and embrace the full scope and possibilty of what we can achieve, the sooner we can become better and more capable software engineers.
>> I've had to write short inductive proofs in several of the systems I've built.
In what context was this necessary?
In what context was this necessary?
This is pretty common in the Blockchain space to verify transactions behave correctly when arriving out of sync with tools like Agda, Coq, etc... I assume it's the same on databases & I heard Leslie Lamport give a talk at work where he mentioned AWS used TLA+ to prove some of its properties.
To provide evidence that a recursive algorithm actually works maybe?
I've worked with a bunch of engineers, and while I respect and admire them tremendously, I'd not want to be lumped in with the kind of code they generally write. And don't get me started on the typical code a scientist writes!
Let the downvotes commence.
Edit: This is a gentle ribbing, but there really does seem to be a different approach in the philosophy of how to solve a problem. Engineers seem to eschew abstraction while software folks embrace is (sometimes to a fault). It's actually pretty fascinating.
Let the downvotes commence.
Edit: This is a gentle ribbing, but there really does seem to be a different approach in the philosophy of how to solve a problem. Engineers seem to eschew abstraction while software folks embrace is (sometimes to a fault). It's actually pretty fascinating.
If they wanted to built their own abstractions they’d be scientists.
Actually this is one thing that has always confused me about “software engineers,” at least as someone with an engineering education who isn’t doing engineering work really: we learned how to do particular types of problems very well and reliably, and generally learned a bunch of math tricks. But at a fundamental level the material that the physics students were learning was basically more complicated. Scientist has always felt like a more prestigious title to me. Since most programmers have computer science degrees, why don’t we call ourselves computer scientists?
Actually this is one thing that has always confused me about “software engineers,” at least as someone with an engineering education who isn’t doing engineering work really: we learned how to do particular types of problems very well and reliably, and generally learned a bunch of math tricks. But at a fundamental level the material that the physics students were learning was basically more complicated. Scientist has always felt like a more prestigious title to me. Since most programmers have computer science degrees, why don’t we call ourselves computer scientists?
> Since most programmers have computer science degrees, why don’t we call ourselves computer scientists?
Because they aren’t doing scientific research, they are applying the products of such research to design (and build; with software the distinction is less significant than with many physical items) products, so its somewhere between architecture/engineering and constructiom, rather than science.
Because they aren’t doing scientific research, they are applying the products of such research to design (and build; with software the distinction is less significant than with many physical items) products, so its somewhere between architecture/engineering and constructiom, rather than science.
Very odd. Usually engineers love abstraction where I am coming from. The simpler the solution, better it usually is.
Helps understanding the whole system, making it less complex. Simplicity causes less bugs.
Of course, you can do too much abstraction.
Of course, you can do too much abstraction.
Have you never run into Matlab code with a few hundred global variables mutating frequently? :)
Okay. I was referreing into software engineers, not data engineers :-)
Ah, I was referring to general engineers, like chemical or mech engineers :)
People use 'engineer' differently in different parts of the world.
The more you know broadly, the smarter you’ll sound. And generally, you will tend to be more useful and fulfilled.
If you want a career that is focused, you have to choose a focus. I’ve struggled with that. You need a broad base of knowledge to be good at understanding all things, and how they connect.
But eventually someone needs a lightbulb changed, and I’m testing the grounding of a nearby lightswitch while on the phone with a contractor about shielding noise from the antiquated home stereo system.
Edit: I should probably mention explicitly.
Investing in TensorFlow or CUDA is one thing. Investing in certain ML techniques or modeling domains another—and python yet another—
Systems and general engineering is a completely messier enchilada. You still get paid!
So. I guess. Do better than I. Choose a scale. It’s impossible to focus on everything unless that’s just an impulse you’ve got :)