Uf2: USB Flashing Format(github.com)
github.com
Uf2: USB Flashing Format
https://github.com/microsoft/uf2
37 comments
I dunno... I have like a 50% success rate between DFU, UF2, and whatever the Arduino IDE does (varies by chip). Meanwhile, SWD/JTAG works for me 100% of the time, and is faster. I wouldn't use anything else.
The downside is that it requires $6 in hardware (or $20 if you splurge on a J-LINK EDU) and so I guess it takes quite a few encounters with "60 seconds of poking around at reset buttons frantically for what should have been a 1 second operation" before that $6 is worth it. GDB on your microcontroller code is also occasionally useful. I think hobbyists should just give up on stuff designed to "make it easy" (but only works half the time) and use the tools that people use for their day jobs, especially when it's only a few dollars investment.
The downside is that it requires $6 in hardware (or $20 if you splurge on a J-LINK EDU) and so I guess it takes quite a few encounters with "60 seconds of poking around at reset buttons frantically for what should have been a 1 second operation" before that $6 is worth it. GDB on your microcontroller code is also occasionally useful. I think hobbyists should just give up on stuff designed to "make it easy" (but only works half the time) and use the tools that people use for their day jobs, especially when it's only a few dollars investment.
With all these UF2 first boards the first thing I do is solder on the SWD header and figure out what magic sequence of memory read/writes I need to make over jlink to unlock the flash and remove the damn bootloader.
Sometimes I wonder if anyone actually uses a debugger anymore :(
Sometimes I wonder if anyone actually uses a debugger anymore :(
My experience as a hobbyist with an STM32F401CE dev board echoes "DFU doesn't work reliably". For some reason the success rate of getting the integrated DFU bootloader to work varied for the exact same board with different computers. But, given that flashing over SWD worked all the time, it's just easier to use that.
Micropython supported writing python programs as plaintext files to the flash-drive on the device. This was super convenient for hobbyist prototyping.
Micropython supported writing python programs as plaintext files to the flash-drive on the device. This was super convenient for hobbyist prototyping.
Yup, exactly. I don't know how they made it so flaky, but it's flaky every single time. (To some extent, I think it's bootloader-related issues; not a hardware or protocol problem, but a quirk of the bootloaders floating around.)
Once you get Micropython or Circuit Python loaded, it is pretty great. I think if I were teaching someone programming for the first time, that's the platform I'd use. (But every time I try, they say they want to learn React, try doing so for a week, and then give up on programming for good. You've gotta crawl before you can walk!)
Once you get Micropython or Circuit Python loaded, it is pretty great. I think if I were teaching someone programming for the first time, that's the platform I'd use. (But every time I try, they say they want to learn React, try doing so for a week, and then give up on programming for good. You've gotta crawl before you can walk!)
Can second this. Having JTAG/SWD access is so much better than any of these ad-hoc stuff combined. When even SWD doesn't work, you know with certainty that your board/chip is in deep trouble anyway. Also for most of Cortex-M you buy the dongle only once.
Get one of these and you don't even need to populate a connector on the PCB, just buy the programmer once.
https://www.tag-connect.com/
I prefer the style with the little hook legs.
https://www.tag-connect.com/
I prefer the style with the little hook legs.
If you don't feel like spending a lot you can use this https://arcade.makecode.com/hardware/dbg
For the small connector you may need to buy something for a few dollars, and for the big one you almost certainly already have everything on your desk. The small one is smaller, and despite the picture you can also put it in the middle of your board - just make sure to keep taller components our of the yellow area.
For the small connector you may need to buy something for a few dollars, and for the big one you almost certainly already have everything on your desk. The small one is smaller, and despite the picture you can also put it in the middle of your board - just make sure to keep taller components our of the yellow area.
There's also the poor man's version of this where you pull your debugging symbols out to a card edge. Have done that on several design and it worked well. You get to play with what you want on the connector for the design typically, so have thrown ethernet MII on there in addition JTAG and UART.
That is really nifty!
I'm firmly in the "why not both?" camp: scrappy, low-barrier-to-entry UF2 et al for a simple on-ramp that gets people hooked with a single purchase (like a microbit a kid gets given, or someone's Christmas present that they might not have had any input into), then graduating to better tools as it becomes clear that the time investment is worthwhile.
That's the main cost here, it's not the purchase price that matters. It's the uncertainty of "have I bought the right thing? Will it work with tools I already know how to use, or am I going to need to learn a whole bunch more stuff? Do I need to figure out a new docs ecosystem?" and all that jazz.
There's one point I'm unsure of, though, and that's how to make it clear that when people see failures with the simple-onramp system, that those failures would be fixed by moving to different tooling, and not just replaced with new problems to occupy one's brain with.
That's the main cost here, it's not the purchase price that matters. It's the uncertainty of "have I bought the right thing? Will it work with tools I already know how to use, or am I going to need to learn a whole bunch more stuff? Do I need to figure out a new docs ecosystem?" and all that jazz.
There's one point I'm unsure of, though, and that's how to make it clear that when people see failures with the simple-onramp system, that those failures would be fixed by moving to different tooling, and not just replaced with new problems to occupy one's brain with.
That's fair. To me, the reliability exploits the hobbyist's inexperience; they don't know how short a edit/test cycle should be, so they aren't upset by how long it is. "Hardware is hard," they're told.
As other comments mention, the real innovation is not making them flash firmware at all -- things like Circuit Python have worked very well for me, and you can point your editor at a Python file right on the device itself and it starts running from the top when you save. That is probably the ideal situation; nevermind how inefficient it is compared to C++ (people can learn C++ once they understand the basics of programming; no need to start there).
I'll also point out that Amazon has STM32 kits; $18 gets you 2 STM32F103s and a programmer. I have used those and they are probably my favorite microcontroller platform and the price sure is good. There is no doubt about which programmer to use because it's in the box, and the programmer is so cheap that buying one (and a backup microcontroller) is cheaper than most hobbyist MCU boards. ("Made in Italy" ends up being quite a bit pricier than "made in China probably from a grey-market chip".)
As other comments mention, the real innovation is not making them flash firmware at all -- things like Circuit Python have worked very well for me, and you can point your editor at a Python file right on the device itself and it starts running from the top when you save. That is probably the ideal situation; nevermind how inefficient it is compared to C++ (people can learn C++ once they understand the basics of programming; no need to start there).
I'll also point out that Amazon has STM32 kits; $18 gets you 2 STM32F103s and a programmer. I have used those and they are probably my favorite microcontroller platform and the price sure is good. There is no doubt about which programmer to use because it's in the box, and the programmer is so cheap that buying one (and a backup microcontroller) is cheaper than most hobbyist MCU boards. ("Made in Italy" ends up being quite a bit pricier than "made in China probably from a grey-market chip".)
The Raspberry Pi Pico has a "picoprobe" firmware that turns it into an SWD debugger. So, it's a $4 interface now :)
I believe this spec is incompatible with the Linux kernel page cache.
Imagine this scenario:
Someone plugs in a U2F microcontroller and copies a new U2F file to it.
The fat filesystem driver can pick any 512 byte sectors to save the data on. There is no guarantee they are contiguous.
Any 512 byte sector written involves reading a 4k page to populate the page cache.
It is possible for a 4k page to have a mix of 512 byte sectors that are part of the newly written file and those from the current.U2F from the device.
The page cache is written out in an arbitrary order.
When that page containing a mix of sectors is written out to the device, it will become flashed with a mix of the new and old firmware images, and won't work properly.
Imagine this scenario:
Someone plugs in a U2F microcontroller and copies a new U2F file to it.
The fat filesystem driver can pick any 512 byte sectors to save the data on. There is no guarantee they are contiguous.
Any 512 byte sector written involves reading a 4k page to populate the page cache.
It is possible for a 4k page to have a mix of 512 byte sectors that are part of the newly written file and those from the current.U2F from the device.
The page cache is written out in an arbitrary order.
When that page containing a mix of sectors is written out to the device, it will become flashed with a mix of the new and old firmware images, and won't work properly.
That's a good point, but in practice, it works fine on Linux. It looks like the static content of the disk, including CURRENT.UF2, is all at the start, so the only time you could have a mix of CURRENT.UF2 content and the uploaded uf2 in a single page cache entry would be the very last of CURRENT.UF2 and the very start of the new flash content. I'm tempted to wave my hands and say that will work out fine (since that content would be overwritten again when reaching the end of the uploaded uf2) but it does seem like a potential problem and it would be nice to know why it's okay..
The device does not allow reading of flashed data, so this can't happen. When it reads, it reads a static file system image of blocks that do not match the pattern, which it'll write into a black hole.
What's CURRENT.U2F then?
Ok, yes, if that's not 4k aligned & padded you could read data from that (otherwise the OS wouldn't load something from it to then write data to the same blocks) - I had somehow missed that on rereading. But even then it's not in the U2F block format, so you'd have to randomly have a block in your firmware that matches the format correctly for that bug to trigger.
I was under the impression that this is in the U2F block format to allow users to copy the file off one microcontroller and onto another...
Ahh,"randomly".
Frustratingly often correlates with something Mr Sod said which turned into a Law.
Frustratingly often correlates with something Mr Sod said which turned into a Law.
They're not in effect any longer, but Microsoft (promoter of this format, that depends on the FAT filesystem, another Microsoft thing) used to enforce their patents on the FAT filesystem against companies like TomTom (as recently as 2009).
FAT is a wide standard now, but it's garbage, even with the ExFAT extensions. I hope people start doing everything they can to move away from the ecosystems promoted by those who would wield standards as weapons.
http://en.swpat.org/wiki/Microsoft_FAT_patents
Note: they did ultimately pledge not to enforce their ExFAT patents against implementors, but I don't forget their behaviors that quickly. https://www.zdnet.com/article/microsoft-readies-exfat-patent...
> Why is Microsoft doing this considering over the years it's made tens of millions from its FAT patents? Stephen Walli, Microsoft's principal program manager for Azure, explained at Open Source Summit Europe last year that "Open source changed everything. Customers have changed. Fifteen years ago, a CIO would have said, 'we have no open source, they would have been wrong, but that's what they thought.' Now, CIOs know open source's essential … Microsoft has always been a company by, of, and for developers. At this point in history, developers love open source."
Beware of Redmond bearing gifts.
FAT is a wide standard now, but it's garbage, even with the ExFAT extensions. I hope people start doing everything they can to move away from the ecosystems promoted by those who would wield standards as weapons.
http://en.swpat.org/wiki/Microsoft_FAT_patents
Note: they did ultimately pledge not to enforce their ExFAT patents against implementors, but I don't forget their behaviors that quickly. https://www.zdnet.com/article/microsoft-readies-exfat-patent...
> Why is Microsoft doing this considering over the years it's made tens of millions from its FAT patents? Stephen Walli, Microsoft's principal program manager for Azure, explained at Open Source Summit Europe last year that "Open source changed everything. Customers have changed. Fifteen years ago, a CIO would have said, 'we have no open source, they would have been wrong, but that's what they thought.' Now, CIOs know open source's essential … Microsoft has always been a company by, of, and for developers. At this point in history, developers love open source."
Beware of Redmond bearing gifts.
Microsoft's patents were around VFAT long names. This doesn't involve that aspect.
Patents aside are there any real alternatives to FAT that are even remotely widely adopted?
For microcontrollers, https://github.com/littlefs-project/littlefs has some mindshare (although not sure how much there is outside the mbed world).
Closest may be ISO 9660 or UDF, although those are aimed more towards write-once/read-only media.
UDF works fine with read/write media (https://en.wikipedia.org/wiki/Universal_Disk_Format).
I had several USB flash sticks that I formatted as UDF under Linux that I used to transfer files between Linux and W10. You get large files, long file names, and read/write compatibility on Linux and W10 with a UDF format USB stick.
I had several USB flash sticks that I formatted as UDF under Linux that I used to transfer files between Linux and W10. You get large files, long file names, and read/write compatibility on Linux and W10 with a UDF format USB stick.
UDF doesn’t work that well for true R/W storage, it’s more suited for appendable or rewritable storage and from my experience the format is so bloody fragmented version wise that half the time when you format it on one OS you don’t know what will work on the other, sometimes the other would read it just fine but won’t be able to modify it without reformat in some cases modifying it makes it unreadable on other devices, it’s quite a mess...
There are many unencumbered filesystems with free software implementations. Sadly, compatible filesystem innovation seems to have stagnated about twenty years ago; all of the cool filesystems these days are developed by OS teams, for their specific OS. It's a bummer.
ZFS is only 15 years old and continues to be improved (though granted, all of the real innovations have already happened). It works on (in order) Solaris, FreeBSD, Linux, MacOS, and Windows.
ZFS does not reliably work on macOS. If you think otherwise I encourage you to try: I have.
Maybe not (there's a reason I put macos and windows last), but the fact remains that people have put effort into making it portable, and it shows: ZFS runs well on more OSes than most other (non-toy) FSes.
ZFS is way too complex and finicky for these small scale applications.
FAT32 patents are now fully expired, making it the most compatible filesystem for which any new devices should support at a minimum, from this point forward.
Most likely the best approach for NTFS is to standardize at ASTM where Microsoft could contribute as much documentation & volunteer effort as they would like (or not), regardless with other members a full-consensus ASTM standard could then be agreed upon & maintained from that point forward. Should have been done a long tome ago for both FAT32 & NTFS as well as all other file systems which are intended for widespread use.
It's just not something where you want to have a moving target, it basicaly needs to stay the same forever once it gets into widespread use.
Important data is too important for any less than truly standard file sysstems.
Most likely the best approach for NTFS is to standardize at ASTM where Microsoft could contribute as much documentation & volunteer effort as they would like (or not), regardless with other members a full-consensus ASTM standard could then be agreed upon & maintained from that point forward. Should have been done a long tome ago for both FAT32 & NTFS as well as all other file systems which are intended for widespread use.
It's just not something where you want to have a moving target, it basicaly needs to stay the same forever once it gets into widespread use.
Important data is too important for any less than truly standard file sysstems.
NTFS would be a poor choice. This FS is very complex, undocumented, and without a good complete open source implementation (NTFS 3G is far from complete and wasn't updated for many years already).
At ASTM, overcoming those type of problems with complete consensus, then independently publishing and maintaining the resulting Standard Specification into the next century or beyond is just an everyday (everyyear actually) occurrence.
That's exactly the reason for the existence of the Society to begin with.
I would start by calling for action from an appropriate committee, submitting all available documentation from NTFS-3G plus from Paragon. Microsoft, and anyone else who has anything to offer that they would not like to become overlooked.
Then hammer away in committee, especially with the full cooperation of Microsoft and all the great engineers who do have some rare expertise, it should be possible to publish like never before as an international ASTM Standard within a year, and maintain it from that point forward with all revisions rigorously reviewed and documented.
After that engineers everywhere will have the technical reference material they have been lacking when they need to make sure their implementations of NTFS are as compatible with each other as they can be, even better allowing for absolute full compliance by all going forward.
That's exactly the reason for the existence of the Society to begin with.
I would start by calling for action from an appropriate committee, submitting all available documentation from NTFS-3G plus from Paragon. Microsoft, and anyone else who has anything to offer that they would not like to become overlooked.
Then hammer away in committee, especially with the full cooperation of Microsoft and all the great engineers who do have some rare expertise, it should be possible to publish like never before as an international ASTM Standard within a year, and maintain it from that point forward with all revisions rigorously reviewed and documented.
After that engineers everywhere will have the technical reference material they have been lacking when they need to make sure their implementations of NTFS are as compatible with each other as they can be, even better allowing for absolute full compliance by all going forward.
This format is not dependent on FAT, it was designed with FAT in mind. It can be used with any filesystem.
Unless that filesystem sometimes doesn't align writes (NTFS), uses copy on write (btrfs), uses compression, raid, encryption, sometimes rewrites old unallocated data (Linux kernel page cache) etc.
To me this whole thing seems like a massive layer violation which will cause ossification of filesystems.
Future Microsoft:. "We can't deploy upgrades to the FAT driver because the new order of writing partial sectors breaks U2F"
To me this whole thing seems like a massive layer violation which will cause ossification of filesystems.
Future Microsoft:. "We can't deploy upgrades to the FAT driver because the new order of writing partial sectors breaks U2F"
I work with Adafruit on CircuitPython. We use it on basically every microcontroller device that can possibly support it. The recently announced Raspberry Pi Pico / RP2040 microcontroller have given UF2 a boost by placing a UF2 bootloader directly in the ROM of the microcontroller.
With no additional software to install, you do something as easy as "double click the reset button" or "hold down button while plugging in the USB cable", and then just drag a downloaded file into the drive that appears. Almost everyone can handle this step!
It also works well with integrated environments like Arduino and you can craft "make flash" type rules in any build system.
The usual way, with the bootloader as a "first stage" that loads the real program such as CircuitPython or an Arduino sketch is still susceptible to occasional overwrites by buggy software, so it's occasionally necessary for end-users to resort to other techniques like DFU or SWD reloading of the firmware. That's when you really remember how not great the bad old days were. But RP2040 even avoids that, because there's no way to overwrite the UF2 bootloader!