So You Can't Find Yocto People(logikalsolutions.com)
logikalsolutions.com
So You Can't Find Yocto People
https://www.logikalsolutions.com/wordpress/information-technology/yocto/
16 comments
Totally agree. I did some Yocto work and it was really rewarding, just not monetarily rewarding. So I haven't done that work since. But it's nice to get hardware working with the correct Yocto config.
That's crazy. I always looked at embedded stuff thinking "that looks complex and hard, but I guess that's why it must be well paid".
What's up with that?
What's up with that?
As a full-stack web engineer who's ended up being competent with Yocto, I think a major factor is which companies need embedded Linux or proper microcontroller embedded work.
They're not you VC, SASS, zero marginal cost software first company. They're hardware manufacturers with physical product margins. The embedded project I work on only ships on a few thousand units/year, and it's only a small component of the overall system.
I think within that group of companies, the comp for embedded vs "software" isn't significantly different. "Software" just has FANG/VC backed salary inflation making the averages look crazy.
They're not you VC, SASS, zero marginal cost software first company. They're hardware manufacturers with physical product margins. The embedded project I work on only ships on a few thousand units/year, and it's only a small component of the overall system.
I think within that group of companies, the comp for embedded vs "software" isn't significantly different. "Software" just has FANG/VC backed salary inflation making the averages look crazy.
The FAANGs have definitely inflated prices, but in those companies the average revenue per employee is just orders of magnitude larger than anything a physical product company can produce. Giving a software engineer 2x or 5x salary compared to embedded is literally noise on the balance sheet.
Because you're a line item in a product's bill of materials and nothing more.
I work on Yocto-based products and am responsible for the platform, but working on recipes and kernel mods is only 10% of my work, and by the end you're not touching it at all. If you're making kernel and rootfs changes in validation you have a serious problem.
I'm not sure what projects need someone that is hacking recipes 40 hours a week.
I work on Yocto-based products and am responsible for the platform, but working on recipes and kernel mods is only 10% of my work, and by the end you're not touching it at all. If you're making kernel and rootfs changes in validation you have a serious problem.
I'm not sure what projects need someone that is hacking recipes 40 hours a week.
I have, like the author, dabbled in Yocto/OpenEmbedded. I disagree with his "professionals" take. It is the biggest bundle of duct tape and hacks I have ever seen, but I don't really see that as a bad thing. It's just the only way to get stuff done with open source.
It is mainly handled by armies of contractors fixing stuff that constantly breaks from (uncaring) vendors that all gets chucked into a giant pot that has to work to ship. Vendors patching other vendors' recipes, etc...
Frankly, I think Yocto enables a more agile approach to embedded than not. I love it and I hate it.
It is mainly handled by armies of contractors fixing stuff that constantly breaks from (uncaring) vendors that all gets chucked into a giant pot that has to work to ship. Vendors patching other vendors' recipes, etc...
Frankly, I think Yocto enables a more agile approach to embedded than not. I love it and I hate it.
And I can't find Yocto (well, embedded linux) employers. Quite ironic!
How about: You can't find Yocto developers because Yocto is a gigantic pain in the ass because you have to figure out and do everything the "Yocto Way(tm)" in spite of horrifically terrible documentation.
We ate the cost to upgrade the hardware simply so we could run mainstream Debian and punt Yocto to the garbage can.
If I'm being more charitable: Yocto has been made irrelevant by the progress of technology. Either I have a system that is capable of running full blown Linux or I have a system that has no hope of running anything that isn't completely bespoke. The number of times I have enough resources to run "I Can't Believe It's Not Linux" but not actual Linux has converged toward zero.
We ate the cost to upgrade the hardware simply so we could run mainstream Debian and punt Yocto to the garbage can.
If I'm being more charitable: Yocto has been made irrelevant by the progress of technology. Either I have a system that is capable of running full blown Linux or I have a system that has no hope of running anything that isn't completely bespoke. The number of times I have enough resources to run "I Can't Believe It's Not Linux" but not actual Linux has converged toward zero.
While I don't agree with the tone of your post, I completely agree with the message :)
30 - 10 years ago; Yocto (and its kin) served a very specific purpose. Building super tiny Linux distributions. If you have 4MiB worth of flash (storage) there's only so much you can cram in their.
Any "modern" linux distribution might still require more then that (excluding the likes of Linux from Scratch, as that falls in into the yocto category). Being CPU bound tends not to be the major issue, storage and RAM bound however is.
Personally, I've used debian in an embedded setting which can work great, but recently I've been using Alpine Linux for embedded and if you have a bit of storage (say 32 - 64MiB SPI NOR, or just some eMMC) you are good with Alpine as well.
Self-plug on not-yet-finished image builder around alpine: https://gitlab.com/esbs/builder
One other thing I think is greatly wrong with yocto, is the fact that it is super complex to do CI with it (because its one big thing) and build-times of 3.5 hours are ridiculous. Yes if I had to rebuild every single alpine package I'd probably end up with those build-times as well. But once a packages is built and vetted, just go with it (in your mirror if you will)
As for building the whole toolchain from scratch, sure, there might be exotic reasons to do that, but come on. Just stick your build environment in a container, and use that. Getting the toolchain from a "vendor" (alpine being one) is not a crime...
30 - 10 years ago; Yocto (and its kin) served a very specific purpose. Building super tiny Linux distributions. If you have 4MiB worth of flash (storage) there's only so much you can cram in their.
Any "modern" linux distribution might still require more then that (excluding the likes of Linux from Scratch, as that falls in into the yocto category). Being CPU bound tends not to be the major issue, storage and RAM bound however is.
Personally, I've used debian in an embedded setting which can work great, but recently I've been using Alpine Linux for embedded and if you have a bit of storage (say 32 - 64MiB SPI NOR, or just some eMMC) you are good with Alpine as well.
Self-plug on not-yet-finished image builder around alpine: https://gitlab.com/esbs/builder
One other thing I think is greatly wrong with yocto, is the fact that it is super complex to do CI with it (because its one big thing) and build-times of 3.5 hours are ridiculous. Yes if I had to rebuild every single alpine package I'd probably end up with those build-times as well. But once a packages is built and vetted, just go with it (in your mirror if you will)
As for building the whole toolchain from scratch, sure, there might be exotic reasons to do that, but come on. Just stick your build environment in a container, and use that. Getting the toolchain from a "vendor" (alpine being one) is not a crime...
[deleted]
Feels like the author was frustrated with agile and took it out in the guise of yocto engineering champion.
I think the author is mostly right. Embedded development does not work well with agile. Hardware does not work well with agile.
You're dealing with vendors, supply chains, hardware revisions, manufacturing pipelines... It's classic waterfall. Agile is a recipe for burnout here.
You're dealing with vendors, supply chains, hardware revisions, manufacturing pipelines... It's classic waterfall. Agile is a recipe for burnout here.
The problem is much more about reasonableness than about "waterfall" or "agile". Having everything "planned" beforehand and forcing it through is also terrible for embedded projects if there are holdups - and there are going to be holdups.
My favorite is always "but we defined 3 years ago that it'll do this-and-this fancy 3D thing on this screen, it's a requirement!" "well then maybe you should have selected hardware that can do fancy 3D. Let's talk about what we can change that still is good." "But its a requirement!".
Or on the hardware side, there is a "plan" that can't cope with the fact that suddenly there is a really good reason to do a hardware revision that nobody planned for. (e.g. because its 2021 and suddenly a bunch of parts just isnt available)
Of course totally freewheeling nothing-planned also causes equivalent problems, as does expectations that anything can change within 2 weeks. On the other hand on plenty projects I've worked on the "agile" requirements on the Yocto person are more among the lines of "we'd like this additional software added to the image please".
My favorite is always "but we defined 3 years ago that it'll do this-and-this fancy 3D thing on this screen, it's a requirement!" "well then maybe you should have selected hardware that can do fancy 3D. Let's talk about what we can change that still is good." "But its a requirement!".
Or on the hardware side, there is a "plan" that can't cope with the fact that suddenly there is a really good reason to do a hardware revision that nobody planned for. (e.g. because its 2021 and suddenly a bunch of parts just isnt available)
Of course totally freewheeling nothing-planned also causes equivalent problems, as does expectations that anything can change within 2 weeks. On the other hand on plenty projects I've worked on the "agile" requirements on the Yocto person are more among the lines of "we'd like this additional software added to the image please".
Had a weird moment reading this where I could not initially determine if "yocto" was a real thing or a made-up stand-in for purposes of the article.
I could use some yocto developers if anyone is interested. Let me know!
The work is challenging and rewarding, but also grueling and frustrating like little else. Embedded wages are miserable. Almost any Yocto developer could immediately improve their quality of life by picking up a Ruby on Rails book and then nearly doubling their salary in a month or two.
That's essentially what I did and I've never looked back.
As long as that holds true, hiring good folks will be challenging.