Getting the RK3588 SoC supported in upstream Linux (2023)(kernel-recipes.org)
kernel-recipes.org
Getting the RK3588 SoC supported in upstream Linux (2023)
https://kernel-recipes.org/en/2023/schedule/getting-the-rk3588-soc-supported-upstream/
33 comments
I went down this rabbit hole just the other week:
https://taoofmac.com/space/blog/2024/02/10/2000
I managed to get recovery booting but my rootfs didn't finish loading, and I still haven't figured out why. And I am no novice to these things (I currently have around half a dozen RK3588/3588S boards around my desk as I do some advisory work on IoT projects, and right now my strongest recommendations are all Armbian-compatible).
https://taoofmac.com/space/blog/2024/02/10/2000
I managed to get recovery booting but my rootfs didn't finish loading, and I still haven't figured out why. And I am no novice to these things (I currently have around half a dozen RK3588/3588S boards around my desk as I do some advisory work on IoT projects, and right now my strongest recommendations are all Armbian-compatible).
Great presentation..one of the major issues with the support of RK3588 is that the GPU doesn't have proper drivers, so GPU acceleration relies on patches made by small contributors and these are obviously not updated regularly, so they tend to be obsolete easily.
The situation doesn't seem quite so dire, Collabora is apparently working on driver for this now: https://www.phoronix.com/news/Panthor-DRM-Newer-Mali
They have pretty decent track record in producing these sort of drivers so there is a chance
They have pretty decent track record in producing these sort of drivers so there is a chance
This is not true, the panthor driver (previously pancsf) for mesa is actively being developed and funded, and already works quite well (I had a demo desktop running on prototye MNT Reform Next laptop at 37c3).
I feel like "already" is a bit strong considering the GPU has been out for almost 3 years and it's still seemingly far from upstream.
And the main panfrost developer Alyssa Rosenzweig has moved on, so the pace of development has declined significantly. At this point the Raspberry Pi 5 is more than capable and the Rock 5 has lost the primary advantage it once had.
It's nothing to do with that really. It's much more to do with the gen10 (Gxxx) Mali requiring a completely new kernel driver. Given that nouveau and other drivers required the same thing, this ended up being a long diversion into a lot of new common DRM infrastructure to make things easier for the next drivers who need to do the same thing. By comparison, enabling gen9 in the kernel was more like adding a couple of device IDs. See bbrezillon & dakr's talk here for more details: https://indico.freedesktop.org/event/4/contributions/181/
A story as old as the embedded ARM ecosystem itself :(
AMD is probably the best hope nowadays, they've started licensing their RDNA GPU architecture out to ARM SOC vendors and they already have excellent open source drivers. Mali and Adreno are still much more common though, especially on cheap SBCs which nearly always use Mali.
I hoped so but nope. Exynos with RDNA has closed source user-space, with AMD/Samsung not making public the differences, and no attempt at upstreaming the kernel-side ones too.
The best mobile platform on Arm with open-source GPU drivers rn is Qualcomm...
The best mobile platform on Arm with open-source GPU drivers rn is Qualcomm...
Either that, or using the proprietary Mali drivers which rely on userspace X11 GPU drivers which don't work in current versions of X or current kernels so you end up running 5.10 using a separate x11-legacy package.
ARM GPUs on Linux are truly in a terrible state right now. I hope ARM gets serious about ensuring that upstream supports their GPUs soon after release.
ARM GPUs on Linux are truly in a terrible state right now. I hope ARM gets serious about ensuring that upstream supports their GPUs soon after release.
At least ARM is not completely antagonist anymore and is actually funding some this effort, that is already pretty good step forwards. But yes, there is long way to go for everyone here.
https://newsroom.arm.com/news/arm-expands-open-source-partne...
https://newsroom.arm.com/news/arm-expands-open-source-partne...
I've seen that too, I'm (very) cautiously optimistic. Ideally, ARM would be doing a bunch of the development work and actually ensure that the driver support is ready to be upstreamed on day 1 (like what AMD pretty much does AFAIU).
However, the Mali-G610 (used in the RK3588) is still not supported despite being released in 2021. ARM's "work with Collabora" started before the release of the Mali-G610, so the fact that we're so far from upstream support even after this many years of development seems like a bad sign. I don't understand why future Mali architectures wouldn't also take 4+ years unless ARM really really steps it up.
However, the Mali-G610 (used in the RK3588) is still not supported despite being released in 2021. ARM's "work with Collabora" started before the release of the Mali-G610, so the fact that we're so far from upstream support even after this many years of development seems like a bad sign. I don't understand why future Mali architectures wouldn't also take 4+ years unless ARM really really steps it up.
> I don't understand why future Mali architectures wouldn't also take 4+ years unless ARM really really steps it up.
G610 (and the larger G710) are sort of special since there was a major change to the internal architecture and how work is submitted to the GPUs. Later versions didn't have such a big, breaking change.
G610 (and the larger G710) are sort of special since there was a major change to the internal architecture and how work is submitted to the GPUs. Later versions didn't have such a big, breaking change.
>I hope ARM gets serious about ensuring that upstream supports their GPUs soon after release.
Ironically, Imagination Technologies of PowerVR fame (infamy?) did, with a funded mesa3d driver effort. And embraced RISC-V while at it.
Ironically, Imagination Technologies of PowerVR fame (infamy?) did, with a funded mesa3d driver effort. And embraced RISC-V while at it.
Joshua makes some good improvement and upstreams for this SOC
https://github.com/Joshua-Riek/ubuntu-rockchip
https://github.com/Joshua-Riek/ubuntu-rockchip
Very cool, I wonder if I could get that working for the FriendlyElec CM3588 board I have here. Hm.
Yes, I'm happy cuz I finally got a updated distro.
P.S I have a NanoPi R6C.
I ended up doing surgery and using the provided debian install but moving /usr and /var and /home over to nVME SSD (from SDCard) but still booting from the card. The debian itself seems mostly fine and packages upgradeable etc. But kernel, yeah that's another story
I don't get it.
1. Rockchip sells a chip in some boards (let's focus on one, e.g. in Orange Pi 5)
2. Orange Pi 5 has some software pre-installed (I guess linux right? even if it's android then it's still linux, right?)
3. Some people need to find out by themselves how to make normal upstream linux running on this chip
Questions: A. How the system provided by Orange Pi works on the RK3588? Does it not use all the power/features? B. Why Rockchip only "suggests" how to handle problems instead of fixing it? C. Who and why created the "OEM" linux that has been installed on Orange Pi 5 for selling? C.1. Why the people who created it won't release their work into linux?
Questions: A. How the system provided by Orange Pi works on the RK3588? Does it not use all the power/features? B. Why Rockchip only "suggests" how to handle problems instead of fixing it? C. Who and why created the "OEM" linux that has been installed on Orange Pi 5 for selling? C.1. Why the people who created it won't release their work into linux?
This is a common story in ARM / embedded linux. SoC vendors’ main focus is to make the chip and give it enough BSP support for customers to buy it and build their products around it. The bare minimum for this is typically a set of hacky kernel patches targeting a giver kernel version.
The customer then needs to figure out what they want to do: Take the kernel blob as-is? Get the vendor’s kernel source and try and modify it for their specific needs? Rebase it to a newer kernel? The customer typically wants to just get enough support to ship their product; they’re not in the business of maintaining some other company’s kernel patches. They’ll do it if they have to, but the engineers will grumble about it the entire time.
Thus, getting support landed into Linux upstream is primarily the task of the chip vendor. There’s a good business case for it, as better and widely available software support will help sell the chip. But it’s also incredibly expensive to write all that kernel code and take it through the process of upstreaming. A lot of this stuff won’t meet the standards of the Linux kernel without lots of work.
The ARM SBC market flips this, where you have a bunch of enthusiasts, and then a smaller market of businesses wanting to use them in their products. That can provide enough momentum for a community-driven upstreaming effort.
The customer then needs to figure out what they want to do: Take the kernel blob as-is? Get the vendor’s kernel source and try and modify it for their specific needs? Rebase it to a newer kernel? The customer typically wants to just get enough support to ship their product; they’re not in the business of maintaining some other company’s kernel patches. They’ll do it if they have to, but the engineers will grumble about it the entire time.
Thus, getting support landed into Linux upstream is primarily the task of the chip vendor. There’s a good business case for it, as better and widely available software support will help sell the chip. But it’s also incredibly expensive to write all that kernel code and take it through the process of upstreaming. A lot of this stuff won’t meet the standards of the Linux kernel without lots of work.
The ARM SBC market flips this, where you have a bunch of enthusiasts, and then a smaller market of businesses wanting to use them in their products. That can provide enough momentum for a community-driven upstreaming effort.
> A lot of this stuff won’t meet the standards of the Linux kernel without lots of work.
Can you elaborate on this? I never got the impression that 'the standards of the Linux kernel' were some incredibly high bar.
Can you elaborate on this? I never got the impression that 'the standards of the Linux kernel' were some incredibly high bar.
How does questionably formatted, badly documented, insufficiently tested, and full of Chinese comments sound?
That's what I've come expect from this space. In general, it's the usual: China, great at hardware... software an afterthought
That's what I've come expect from this space. In general, it's the usual: China, great at hardware... software an afterthought
Why exactly is it so difficult to polish it up, document it properly, and translate as necessary, that it can't be accomplished by a few smart engineers in a few months?
Here's one example of a wifi driver for the pinephone:
https://salsa.debian.org/Mobian-team/devices/kernels/sunxi64...
The diff is 13.4 MiB and is over 200,000 lines long. It includes it's own AES and SHA-256 engine. Trying to polish this up would be close to years, and a good kernel dev is not cheap.
https://salsa.debian.org/Mobian-team/devices/kernels/sunxi64...
The diff is 13.4 MiB and is over 200,000 lines long. It includes it's own AES and SHA-256 engine. Trying to polish this up would be close to years, and a good kernel dev is not cheap.
Thanks, that does look substantial.
It's very much doable. You see the big vendors upstream proactively, and they have whole teams supporting upstreaming efforts. But chip bring-up and driver support can be messy affairs. It's not just a matter of a driver working or not- hardware bugs or weird undocumented behavior show up all the time. If customers aren't exercising some feature of the chip, bugs can and will go undetected in that module.
The datasheet only gets you so far, especially for undocumented hardware quirks, and the vendor is in a much better place to understand what's happening and fix them. It's an active process of reporting bugs, investigating them, and getting fixes merged upstream into the vendor's BSP, at the very least. This requires active maintenance, which means engineers on the vendor side. That's where the support contracts kick in.
Upstreaming to Linux on top of all that means that you actually need to support and validate the set of features your chip claims to support. That means more thorough test and validation, code that meets the standards of the kernel, and more engineers to actively maintain the code. Not to mention the language barrier for Chinese chip vendors.
Of course, any upstreaming efforts are great and I think things are trending in a good direction. Just pointing out that there are reasons things are the way they are. Fabbing a chip that isn't completely buggy and people want to buy is already a small miracle- better software support can always wait until the chip becomes a hit.
The datasheet only gets you so far, especially for undocumented hardware quirks, and the vendor is in a much better place to understand what's happening and fix them. It's an active process of reporting bugs, investigating them, and getting fixes merged upstream into the vendor's BSP, at the very least. This requires active maintenance, which means engineers on the vendor side. That's where the support contracts kick in.
Upstreaming to Linux on top of all that means that you actually need to support and validate the set of features your chip claims to support. That means more thorough test and validation, code that meets the standards of the kernel, and more engineers to actively maintain the code. Not to mention the language barrier for Chinese chip vendors.
Of course, any upstreaming efforts are great and I think things are trending in a good direction. Just pointing out that there are reasons things are the way they are. Fabbing a chip that isn't completely buggy and people want to buy is already a small miracle- better software support can always wait until the chip becomes a hit.
If it's doable by a few smart engineers in a few months, assuming there are no show stopping quirks, then that should imply the trustworthy, reliable, manufacturers get their stuff upstreamed, by whomever, and the unreliable ones don't.
To these companies software is just a cost centre. Their product is hardware. They want to make a lot of a thing, sell it to as many people as possible, and then move on to make the next thing they sell a lot of.
The kind of thing that motivates us when we build systems doesn't necessarily motivate them. The finished software product is, in the end, your problem, not theirs. Software engineers are expensive. The more they can push off and avoid paying them, the better.
The kind of thing that motivates us when we build systems doesn't necessarily motivate them. The finished software product is, in the end, your problem, not theirs. Software engineers are expensive. The more they can push off and avoid paying them, the better.
thank you, guys/gals, for the explanation, that's a new plane of problems, orthogonal to what I have been experiencing so far.
They will ship images using a probably old kernel with various patches and drivers which may not be in suitable shape for simply including upstream.
The vendor may ship this initially, but often won't work on upstreaming the changes or porting to newer kernels, so you end up with devices stuck running forks of old kernels unless somebody takes the kind of effort discussed here to upstream the changes.
The vendor may ship this initially, but often won't work on upstreaming the changes or porting to newer kernels, so you end up with devices stuck running forks of old kernels unless somebody takes the kind of effort discussed here to upstream the changes.
or unless vulnerabilities of old kernels will allow someone to take them over
But yes, I ventured into... building the kernel & image ... and like a lot of these systems it was a nightmare of brittle shell & Python scripts, patches, many steps, and multiple, multiple READMEs. And because of its origin in the world of phone/media-box SoCs, the kernel defaults set up include annoying things like overriden highly-reduced VmMalloc limit (I work on my own DB component that uses VM overcommit at very large sizes) which I can't find the right knob to fix, etc. etc.
But really, this is a very promising board. I hope to see more boards like this that lean into more "PC-class" capabilities than the usual RPi type sytems.