Microdot: A Web Framework for Microcontrollers(lwn.net)
lwn.net
Microdot: A Web Framework for Microcontrollers
https://lwn.net//Articles/959067/
48 comments
So the entire Microdot web framework is also intended to run on-device? I didn't grok that on my first readthrough
I think so, otherwise what's the point to call it Microdot.
Yeah, I was thinking of something that integrates closely with MicroPython (maybe run the server on RPi and drop a small Microdot comms lib on the device) but it makes a lot more sense to just run everything on-device, given that it's targeting the MicroPython userbase
I hate to be that guy, but I've been writing embedded systems using microcontrollers for decades. The movement to put Python on them makes me cringe.
I understand making them more approachable for developers at a higher level, but the resulting solutions need an order of magnitude more resources to accomplish the same tasks when compared to writing an equivalent system in C.
Assuming these kinds of tools are being used temporarily for proof-of-concept, I think that it is a great fit; however, we all know that there's nothing more permanent than a temporary solution.
I understand making them more approachable for developers at a higher level, but the resulting solutions need an order of magnitude more resources to accomplish the same tasks when compared to writing an equivalent system in C.
Assuming these kinds of tools are being used temporarily for proof-of-concept, I think that it is a great fit; however, we all know that there's nothing more permanent than a temporary solution.
As someone with embedded development experience going back to the 1980’s and across more platforms than I can remember, I agree with your perspective.
Still, even so, I decided to give MicroPython a go for a new design; an industrial product we would ship in the tens of thousands of units.
This will count as one of the many choices I will regret making in my business history. I won’t bore you with details. We shipped 250 units. We are busy porting the code to C as fast as humanly possible.
At least I can say we tried. I have coded embedded systems with assembler, Forth, C and C++. Any one of these would be better than Python.
Yes, I use Python extensively for desktop, data science and web work. It simply isn’t a good fit for embedded work.
Still, even so, I decided to give MicroPython a go for a new design; an industrial product we would ship in the tens of thousands of units.
This will count as one of the many choices I will regret making in my business history. I won’t bore you with details. We shipped 250 units. We are busy porting the code to C as fast as humanly possible.
At least I can say we tried. I have coded embedded systems with assembler, Forth, C and C++. Any one of these would be better than Python.
Yes, I use Python extensively for desktop, data science and web work. It simply isn’t a good fit for embedded work.
I'm curious what made you backpedal after shipping 250 units. I imagine you at least very least made it work at that point.
Fair question. Yes, the product works fine and has been in service in a demanding industrial setting for a year. To make it work we had to write portions of the code in assembler. Extracting performance out of Python is a losing proposition. You are starting out at somewhere around 70x slower than C. We thought MicroPython could pull it off. By the time we understood it could not, there was no way to start over with C.
Things that are trivial to code in a performant way in C are impossible to code in Python. A simple example of this is a cyclic redundancy routine. We had to write those in assembler. Also, garbage collection was problematic, even controlling it manually. If I remember correctly, it took out a 5 us chunk of time.
We had to resort to such things as allocating large buffers globally to avoid allocation/de-allocation issues, etc. This was done mostly by declaring global variables as tuples (due to the simple memory structure) and then accessing them with assembler routines. The immutability of tuples only exists within the confines of Python, outside of that, it’s just memory. This approach resulted in massive performance and memory utilization (reduction) gains.
Beyond that, the entire codebase was a single function running a state machine, this to avoid costly (time and resource) function calls.
Eventually the thing starts to look like a typical bare metal embedded system where you create a round robin scheduler, etc. At that point it is very hard to justify using Python for anything, in fact C is far easier.
Code like that quickly becomes something that is hard to maintain, enhance and evolve. It became very obvious that we had to switch the product to C as soon as possible.
Things that are trivial to code in a performant way in C are impossible to code in Python. A simple example of this is a cyclic redundancy routine. We had to write those in assembler. Also, garbage collection was problematic, even controlling it manually. If I remember correctly, it took out a 5 us chunk of time.
We had to resort to such things as allocating large buffers globally to avoid allocation/de-allocation issues, etc. This was done mostly by declaring global variables as tuples (due to the simple memory structure) and then accessing them with assembler routines. The immutability of tuples only exists within the confines of Python, outside of that, it’s just memory. This approach resulted in massive performance and memory utilization (reduction) gains.
Beyond that, the entire codebase was a single function running a state machine, this to avoid costly (time and resource) function calls.
Eventually the thing starts to look like a typical bare metal embedded system where you create a round robin scheduler, etc. At that point it is very hard to justify using Python for anything, in fact C is far easier.
Code like that quickly becomes something that is hard to maintain, enhance and evolve. It became very obvious that we had to switch the product to C as soon as possible.
Isn’t this an extension of what occurred everywhere in tech, programming becoming bloated/inefficient as a trade off for developer UX via frameworks/abstractions/higher level programming languages. It’s all driven by the low cost of hardware performance.
I’m coming at it from opposite direction. I’ve been learning C to program ESP32 devices and it’s such a frustrating experience how much that language is lacking or has to be DIYed. I could easily use python and move 10x faster, but I’m trying to force myself to learn C in the process, otherwise I have no idea what the benefit would be.
I’m coming at it from opposite direction. I’ve been learning C to program ESP32 devices and it’s such a frustrating experience how much that language is lacking or has to be DIYed. I could easily use python and move 10x faster, but I’m trying to force myself to learn C in the process, otherwise I have no idea what the benefit would be.
> I’ve been learning C to program ESP32 devices and it’s such a frustrating experience ... but I’m trying to force myself to learn C in the process, otherwise I have no idea what the benefit would be.
Keep at it; the benefits to learning C/C++ are immense. They have become the underlying "lingua franca" of the software industry allowing you to study/program applications/libraries/system tools/OS kernels/bare-metal MCUs etc.
For lots of C code to study for MCU programming, see the books (some are available for free) by Michael Pont of SafeTTy Systems here - https://www.safetty.net/publications Though they are for 8051 and not for ESP32 you should be able to understand the techniques involved and use them with your choice of MCU family.
Keep at it; the benefits to learning C/C++ are immense. They have become the underlying "lingua franca" of the software industry allowing you to study/program applications/libraries/system tools/OS kernels/bare-metal MCUs etc.
For lots of C code to study for MCU programming, see the books (some are available for free) by Michael Pont of SafeTTy Systems here - https://www.safetty.net/publications Though they are for 8051 and not for ESP32 you should be able to understand the techniques involved and use them with your choice of MCU family.
There's a pretty direct tradeoff between BOM cost of an ARM microcontroller and ROM/RAM. ESP is a little different with external spi flash, but the tradeoff is mostly there still. So there's a lot of pressure in product development to keep technical costing down by using C/Assembly and choose technologies like UDP vs TCP or CBOR/Binary protocols vs JSON.
As a non-embedded dev, the tooling for embedded stuff has a problematically steep learning curve. I appreciate that sometimes there's a good reason for that, but it drives people to micropython and similar simply because getting them working is so much less painful.
I had an nrf52(840) microcontroller and it took me hours to get "hello world" out of that thing with nRF Connect/Zephyr. Granted, I had no idea what I was doing, but accomplishing the same with equally-foreign-to-me Circuitpython took five minutes - just plug it in to a USB port and drop the bootloader and then your .py into file explorer. I _want_ to do things "properly" and efficiently, but for a small project I can't justify the time investment to get there and so the -pythons provide a tradeoff I'm willing to make.
I had an nrf52(840) microcontroller and it took me hours to get "hello world" out of that thing with nRF Connect/Zephyr. Granted, I had no idea what I was doing, but accomplishing the same with equally-foreign-to-me Circuitpython took five minutes - just plug it in to a USB port and drop the bootloader and then your .py into file explorer. I _want_ to do things "properly" and efficiently, but for a small project I can't justify the time investment to get there and so the -pythons provide a tradeoff I'm willing to make.
> the resulting solutions need an order of magnitude more resources to accomplish the same tasks
Okay, and? It's 2024, even a $2.50 toy devboard had multi-megabit flash & ram. I really don't care if my code takes up 10k or 100k. If Python makes it easier to develop and eliminates entire categories of vulnerabilities in the process, it doesn't make any sense to use bare C instead. I paid for the entire chip, why shouldn't I use the entire chip?
C is fine if you have to do high-speed bitbanging or have to squeeze every last byte out of a chip, but for most applications that's simply not the case anymore. A decent fraction of embedded systems is absolutely trivial on today's hardware. It's time we start treating them like the mini-computers they are, because remaining in a 2000s mindset is going to result in a lot of compromised network-connected devices due to some developer fat-fingering a buffer size.
Okay, and? It's 2024, even a $2.50 toy devboard had multi-megabit flash & ram. I really don't care if my code takes up 10k or 100k. If Python makes it easier to develop and eliminates entire categories of vulnerabilities in the process, it doesn't make any sense to use bare C instead. I paid for the entire chip, why shouldn't I use the entire chip?
C is fine if you have to do high-speed bitbanging or have to squeeze every last byte out of a chip, but for most applications that's simply not the case anymore. A decent fraction of embedded systems is absolutely trivial on today's hardware. It's time we start treating them like the mini-computers they are, because remaining in a 2000s mindset is going to result in a lot of compromised network-connected devices due to some developer fat-fingering a buffer size.
> $2.50 toy devboard had multi-megabit flash & ram
"multi" is "2 or more"
So that's 256KB+ or RAM and 256KB+ of flash. Please send me a link to a $2.50 board with those specs. I could use one at that price point.
"multi" is "2 or more"
So that's 256KB+ or RAM and 256KB+ of flash. Please send me a link to a $2.50 board with those specs. I could use one at that price point.
You can get an S2 mini dev board for a $1 including shipping[1, 2]
[1]: https://imgur.com/HUt2qLY [2]: https://www.aliexpress.us/item/3256804697715980.html?
[1]: https://imgur.com/HUt2qLY [2]: https://www.aliexpress.us/item/3256804697715980.html?
I'm not a fan of that form factor (dual rows of pins on both sides), it's not super breadboard friendly (works ok in perfboard). But, I have just soldered in one row on each side which gives you Vbus, GND, 3.3V, and ~12 GPIOs (with some restrictions). You can draw a fair amount of current from Vbus for ancillary devices (I run a 16x16 LED panel through mine and can reach full brightness, sustained).
You can run the headers 'up' on one side and 'down' on the other if you need more access (you then still need to bridge from 'up' back to down somewhere else but at least you'll have the room to do so.
Yup, that was the exact one I had in mind!
Curious. Cool
I agree. Sadly, it seems like C/C++ isn't popular for teaching the next generation to program microcontrollers (from my limited perspective, at least).
At high school I was taught Arduino and C++ but younger students have rPi Pico's and CircuitPython (Adafruit version of MicroPython), to use the same language as the CS syllabus. I guess the "easy" features of Python (many traps once programs get complex) make it attractive as a first programming language.
One project I've worked on for 3 years with a C++ codebase is going to be passed on to mostly Python programmers... I'll just have to make sure the hardware documentation is good.
At high school I was taught Arduino and C++ but younger students have rPi Pico's and CircuitPython (Adafruit version of MicroPython), to use the same language as the CS syllabus. I guess the "easy" features of Python (many traps once programs get complex) make it attractive as a first programming language.
One project I've worked on for 3 years with a C++ codebase is going to be passed on to mostly Python programmers... I'll just have to make sure the hardware documentation is good.
Why are people teaching kids to program on tiny microcontrollers instead of the beefy laptops they already have?
Very few people are doing microcontroller programming these days, and so beginner-friendly microcontroller resources are far less plentiful than those for mainstream languages. Just teach kids Python on a normal laptop (or in the cloud if you're in the US and give them Chromebooks).
I guess the ability to interact with the physical world (with servos and such) might make the subject more attractive to some, but I'm sure there are ways to plug those into a real computer, or at least a Pi.
Very few people are doing microcontroller programming these days, and so beginner-friendly microcontroller resources are far less plentiful than those for mainstream languages. Just teach kids Python on a normal laptop (or in the cloud if you're in the US and give them Chromebooks).
I guess the ability to interact with the physical world (with servos and such) might make the subject more attractive to some, but I'm sure there are ways to plug those into a real computer, or at least a Pi.
They are getting taught microcontroller programming in addition to programming on PCs - only now they teach the same language for both. I presume it's the school's difference between "Computer Science" and "Engineering" subjects, with more physical electronics aspects to the latter. Personally I think teaching programmers how to interact with the real world beyond a keyboard and screen is pretty important, but I am biased from working on hardware.
There are definitely ways to plug servos, etc. into a normal computer, and some of them are already branded as teaching tools. See SAMLabs, etc.
There are definitely ways to plug servos, etc. into a normal computer, and some of them are already branded as teaching tools. See SAMLabs, etc.
As opposed to the movement to put interpreted BASIC into PIC or Intel 8052 microcontrollers?
1980s embedded engineers please stand up.
-------
But BasicStamp was a thing even decades ago (but after 1980s). There are plenty of advantages to interpreted languages, and they were a bigger deal back when 8052 or whatever we're EPROM and harder to program. I mean, physically harder, with UV to erase the flash.
Having a portion of the code execute out of dataspace thanks to an embedded interpreter was worthwhile, even with all the extra CPU cycles and RAM wasted.
I see Micropython as just an extension of that. Python displaced BASIC a long time ago for good reasons.
1980s embedded engineers please stand up.
-------
But BasicStamp was a thing even decades ago (but after 1980s). There are plenty of advantages to interpreted languages, and they were a bigger deal back when 8052 or whatever we're EPROM and harder to program. I mean, physically harder, with UV to erase the flash.
Having a portion of the code execute out of dataspace thanks to an embedded interpreter was worthwhile, even with all the extra CPU cycles and RAM wasted.
I see Micropython as just an extension of that. Python displaced BASIC a long time ago for good reasons.
Basic can fit into a few kbytes of ROM on an 8-bit CPU, Python definitely cannot.
Even lua needs ~100K IIRC, and that's more than the entire address space (including RAM, ROM and IO) of a 6502.
Even lua needs ~100K IIRC, and that's more than the entire address space (including RAM, ROM and IO) of a 6502.
While that's true, RP2040 is $1 these days. Though it requires external ROM to run, its a popular device and many boards (not just RP2040) are under $5. (ESP32 in particular).
So there's a wide number of under $5 boards available that can run MicroPython, providing an easy start to embedded programmers.
------------
Of course, 8-bits are needed for minimal power consumption and maximum efficiency, with an ability to work off of 128kB of ROM and 8kB of RAM at under $1.
But more importantly, 8-bitters are more complete systems. Better ADCs, often DACs, OpAmps, and other advanced features that RP2040 (and other 32-bit systems) got rid of to make room for RAM or ROM.
That's the thing: 8-bits are more convenient and easier to use not because of code issues, but because they come with all the tools you need on one platform.
---------
But beginners don't know how to use those tools *anyway* and are instead focused on blinking lights or basic I2C / SPI controls. I think grabbing a 32-bit Cortex M0+ (like RP2030) or similar system to run MicroPython isn't a bad thing.
Its inefficient and better designs exist. But beginners shouldn't focus on optimizing yet. Its just like BasicStamp or BASIC-for-8052 back in the day IMO. You need a beginner platform so that young teenagers can experiment with this stuff, and young teenagers aren't going to read the 300+ page manuals that come with an AVR64DD32 to fully understand an 8-bitter's operation.
But even a teenager can plug in a USB port, click a few buttons on an IDE and load Micropython to a RP2040 / ESP32 / whatever.
So there's a wide number of under $5 boards available that can run MicroPython, providing an easy start to embedded programmers.
------------
Of course, 8-bits are needed for minimal power consumption and maximum efficiency, with an ability to work off of 128kB of ROM and 8kB of RAM at under $1.
But more importantly, 8-bitters are more complete systems. Better ADCs, often DACs, OpAmps, and other advanced features that RP2040 (and other 32-bit systems) got rid of to make room for RAM or ROM.
That's the thing: 8-bits are more convenient and easier to use not because of code issues, but because they come with all the tools you need on one platform.
---------
But beginners don't know how to use those tools *anyway* and are instead focused on blinking lights or basic I2C / SPI controls. I think grabbing a 32-bit Cortex M0+ (like RP2030) or similar system to run MicroPython isn't a bad thing.
Its inefficient and better designs exist. But beginners shouldn't focus on optimizing yet. Its just like BasicStamp or BASIC-for-8052 back in the day IMO. You need a beginner platform so that young teenagers can experiment with this stuff, and young teenagers aren't going to read the 300+ page manuals that come with an AVR64DD32 to fully understand an 8-bitter's operation.
But even a teenager can plug in a USB port, click a few buttons on an IDE and load Micropython to a RP2040 / ESP32 / whatever.
When I use python on my microcontroller, it's really just configuring and driving peripherals like the ESP32 RMT. But I have implemented interrupt handlers that handle microsecond-level timing events in python (closed loop camera/light flash/stepper motors). The only reason this works is modern microcontrollers have tons of CPU and memory.
I had a python-based game of life that drove a 16x16 LED panel but the frame rate wasn't very good so I rewrote it in C++.
I had a python-based game of life that drove a 16x16 LED panel but the frame rate wasn't very good so I rewrote it in C++.
I think the balance will shift here a little, now that the RP2040 has made PIO a rather more mainstream concept. Access to really very high-speed, narrowly-focussed reliable IO timing in the context of an interpreted language seems to me to be a good plan.
I gather other microcontrollers have had programmable I/O state machine cores in the past but I'd expect to see a lot more of them (or more documentation and coverage of the ones that exist)
I gather other microcontrollers have had programmable I/O state machine cores in the past but I'd expect to see a lot more of them (or more documentation and coverage of the ones that exist)
I feel like all sane people are with you here mate. Its insane.
Been using esp32-rs recently (esp32 is an admittedly beefy platform/soc) and its been real pleasant.
Been using esp32-rs recently (esp32 is an admittedly beefy platform/soc) and its been real pleasant.
[deleted]
https://lib.rs/crates/picoserve
This should make you happy :)
This should make you happy :)
I'm personally not interested in MicroPython either, but...
> the resulting solutions need an order of magnitude more resources to accomplish the same tasks when compared to writing an equivalent system in C.
I mean, yeah... but that's true when you're writing Python code on a normal computer as well. It's not really an argument against writing Python except in cases where you're actually running into resource constraints.
If you can get the resources you need in a microcontroller you can afford, and you prefer programming in Python... why not?
> the resulting solutions need an order of magnitude more resources to accomplish the same tasks when compared to writing an equivalent system in C.
I mean, yeah... but that's true when you're writing Python code on a normal computer as well. It's not really an argument against writing Python except in cases where you're actually running into resource constraints.
If you can get the resources you need in a microcontroller you can afford, and you prefer programming in Python... why not?
> If you can get the resources you need in a microcontroller you can afford, and you prefer programming in Python... why not?
Particularly when you consider that the "bloat" here has no secondary side-effects: it doesn't waste resources that other software could be using, because no other software is running.
I am a long-time web developer who has always intellectually understood the interest in microcontrollers but never had much of the need. I do now.
I'm not even a fan of Python but MicroPython is a surprisingly productive development experience and I've learned a lot in a very short time with a Pico W.
I will probably use TinyGo at some point in the future, particularly as I try to use smaller devices, and I'm not averse to porting to C, but I really rate the MicroPython ecosystem highly in terms of actual coverage, though mainly just the intent of it, which is unimpeachable IMO.
Particularly when you consider that the "bloat" here has no secondary side-effects: it doesn't waste resources that other software could be using, because no other software is running.
I am a long-time web developer who has always intellectually understood the interest in microcontrollers but never had much of the need. I do now.
I'm not even a fan of Python but MicroPython is a surprisingly productive development experience and I've learned a lot in a very short time with a Pico W.
I will probably use TinyGo at some point in the future, particularly as I try to use smaller devices, and I'm not averse to porting to C, but I really rate the MicroPython ecosystem highly in terms of actual coverage, though mainly just the intent of it, which is unimpeachable IMO.
Not sure why you got down voted.
> it doesn't waste resources that other software could be using, because no other software is running.
While this is interesting observation it is not always true. A lot of devices are low power, and depend on battery.
> it doesn't waste resources that other software could be using, because no other software is running.
While this is interesting observation it is not always true. A lot of devices are low power, and depend on battery.
Oh gosh. You are entirely right :-) If that is why I was downvoted then I'd go along with it for sure.
Doesn't this depend highly on whether that resource usage is worth the tradeoff of developer productivity? Python devs are much more plentiful and easier to hire, and if you now need a $1 microcontroller instead of a $0.5, and you're selling 100k units, that's a $50k cost.
It really depends here. The lower the volume, tighter the iteration time, etc the more sense it makes. I know oil rigs, for example, have a bunch of rpis running on them, because the manufacturer is selling a $100k box and any amount of time spent shaving $10 off the BOM is really not worth it at all.
It really depends here. The lower the volume, tighter the iteration time, etc the more sense it makes. I know oil rigs, for example, have a bunch of rpis running on them, because the manufacturer is selling a $100k box and any amount of time spent shaving $10 off the BOM is really not worth it at all.
You can't just make a firmware developer out of anybody by slapping a more widely used language on a microcontroller, because the language isn't the issue.
"Embedded" development on a highly specced SBC running Linux (like a Raspberry Pi) is a long ways away from what MicroPython is targeting.
"Embedded" development on a highly specced SBC running Linux (like a Raspberry Pi) is a long ways away from what MicroPython is targeting.
Do you think using Python make programmers more productive? I find it's the opposite, due to finding bugs one-by-one as they show up at run time.
Quick honourable mention for Pimoroni's similar tiny framework, phew:
https://github.com/pimoroni/phew
and this rather clever start-as-access-point-to-join-network scheme implemented on it:
https://github.com/simonprickett/phewap
which works a treat!
https://github.com/pimoroni/phew
and this rather clever start-as-access-point-to-join-network scheme implemented on it:
https://github.com/simonprickett/phewap
which works a treat!
Looked in several places, but I can’t figure out exactly what sort of microcontroller they aim to support… are we talking ATmega? ESP32? ARM Cortex M? All of those?
Presumably "anything that micropython (or circuitpython) does" which is vast range. https://github.com/miguelgrinberg/microdot/tree/main/src/mic... is just python code...
Microdot: A web framework for microcontrollers - https://news.ycombinator.com/item?id=39111276 - Jan 2024 (1 comment)
Microdot: Yet another Python web framework - https://news.ycombinator.com/item?id=38834156 - Jan 2024 (26 comments)
Microdot - https://news.ycombinator.com/item?id=38156060 - Nov 2023 (26 comments)
Microdot: Yet another Python web framework - https://news.ycombinator.com/item?id=38834156 - Jan 2024 (26 comments)
Microdot - https://news.ycombinator.com/item?id=38156060 - Nov 2023 (26 comments)
Interesting. Is there a similar micro- or nano- web framework like this but in C?
Looks great. Not sure how apples-to-apples this comparison is [1] but my team has a similar product, pw_web, a Node.js lib that lets you communicate with embedded devices over RPC: https://pigweed.dev/pw_web/
I'm curious to hear what other similar products are out there that others are using?
Edit: Sounds like Microdot runs the entire web framework completely on-device? My initial understanding was wrong in that case; it's an apples-to-oranges comparison
[1] It seems like Microdot handles more of the stack and is probably more hobbyist-friendly based on the fact that it targets MicroPython. Or maybe it's just better all-around; I haven't tried MicroPython so I can't compare.
I'm curious to hear what other similar products are out there that others are using?
Edit: Sounds like Microdot runs the entire web framework completely on-device? My initial understanding was wrong in that case; it's an apples-to-oranges comparison
[1] It seems like Microdot handles more of the stack and is probably more hobbyist-friendly based on the fact that it targets MicroPython. Or maybe it's just better all-around; I haven't tried MicroPython so I can't compare.
I would think Lua or even C based web framework for MCU and restricted embedded boards will be a lot more resource efficient.
Typically you can just run a few cgi scripts with bootstrap as the UI and jquery as the javascript, but a simple MVC will be really cool.