Real time Linux has one known issue remaining(spinics.net)
spinics.net
Real time Linux has one known issue remaining
https://www.spinics.net/lists/linux-rt-users/msg25648.html
25 comments
This headline makes no sense. This particular version has one known issue. Real Time Linux in general of course has a myriad of issues, not least the existence of hypervisors, management controllers and so on in a given piece of hardware that can cause interrupts outside of OS control and therefore unpredictable latency.
Are there processors that don't have management controllers these days? I know ARM processors for phones have things like TrustZone and what not, but what about other ARM processors that are suitable for embedded systems where real-time Linux might make sense?
Apple Silicon machines have no TrustZone, SMM, or anything else to steal control away from the main CPU cores. They also have tricks like being able to direct low-priority IRQs away from CPUs running RT tasks, which we might want to figure out how to support on Linux. The CPU frequency switching latencies are also lower than x86 machines as far as I can tell, and the stalls on switching shorter too. I am very much looking forward to testing out RT on them and seeing how stable I can get low latency audio, compared to typical Intel boxes which are... not great.
Will this eventually get merged into the main Linux branch?
Are there any downsides to using the real time kernel for more mainstream tasks?
Are there any downsides to using the real time kernel for more mainstream tasks?
It's already being merged; PREEMPT_RT got merged last year. We're close to the endgame on that front.
There are some performance tradeoffs to using an RT kernel; you trade some throughput for determinism. That said, I've been running RT on my desktop most of the time for years and never had any issues.
There are some performance tradeoffs to using an RT kernel; you trade some throughput for determinism. That said, I've been running RT on my desktop most of the time for years and never had any issues.
After all the patches are merged, will there be work on making PREEMPT_RT a boot-time or run-time choice rather than build-time?
Why did you choose RT for your desktop?
Why did you choose RT for your desktop?
> will there be work on making PREEMPT_RT a boot-time or run-time choice rather than build-time?
Given the performance critical nature of how this changes things I personally think it's unlikely. It does stuff like change the way spinlocks work in the Kernel which would be not only difficult, but expensive without recompilation. The best I think you'll get is a boot-time option to switch between pre-compiled kernels.
That being said, most of the RT Linux patches over the years are actually compiled in all the time anyway so a lot of these improvements have been incorporated for a long time.
Given the performance critical nature of how this changes things I personally think it's unlikely. It does stuff like change the way spinlocks work in the Kernel which would be not only difficult, but expensive without recompilation. The best I think you'll get is a boot-time option to switch between pre-compiled kernels.
That being said, most of the RT Linux patches over the years are actually compiled in all the time anyway so a lot of these improvements have been incorporated for a long time.
It's not impossible; Linux already has a mechanism for this (alternatives, static keys), which allows patching code at runtime. It is used precisely for this kind of thing, where you don't want the performance impact of branching every time to decide what code path to use.
However, while it can practically eliminate code overhead, it can't eliminate data overhead. I'm guessing the RT stuff does add overhead to data structures, and alternatives wouldn't fix that.
However, while it can practically eliminate code overhead, it can't eliminate data overhead. I'm guessing the RT stuff does add overhead to data structures, and alternatives wouldn't fix that.
Id consider RT if I was serious about audio.
> Are there any downsides to using the real time kernel for more mainstream tasks?
Real-time kernels tend to sacrifice anything in order to minimize latency, for being better for use cases where lowest latency is needed. This usually means tradeoffs regarding throughput, power consumption, resource usage, reliability and more, in order to get the lowest possible latency.
In some cases you need really low latency, but in most day-to-day tasks I don't think you need it (disclaimer: I run linux-zen personally, mainly for minimizing audio latency when doing music with bunch of hardware + desktop PC).
> Will this eventually get merged into the main Linux branch?
I'm not too familiar with kernel development so someone else should give you a definitive answer, but considering the tradeoffs it's very unlikely.
Real-time kernels tend to sacrifice anything in order to minimize latency, for being better for use cases where lowest latency is needed. This usually means tradeoffs regarding throughput, power consumption, resource usage, reliability and more, in order to get the lowest possible latency.
In some cases you need really low latency, but in most day-to-day tasks I don't think you need it (disclaimer: I run linux-zen personally, mainly for minimizing audio latency when doing music with bunch of hardware + desktop PC).
> Will this eventually get merged into the main Linux branch?
I'm not too familiar with kernel development so someone else should give you a definitive answer, but considering the tradeoffs it's very unlikely.
Slight nit, real time systems are designed for a maximum worst case latency within some spec, not minimum latency. You can make a system that has a lower best case latency but that doesn't matter if the worst case causes a missed deadline.
And in the context of real time systems, a missed deadline is either equivalent to a catastrophic error (hard real time) or really terrible but non catastrophic error (soft real time). Soft real time applications like audio (no one dies because of a buffer underrun) can sacrifice the maximum worst case for better average case, because the worst case depends on load and can be worked around. Hard real time cares a lot more about the maximum worst case, like say a CNC machine where missing a deadline can cause a mechanical failure and break the part.
And in the context of real time systems, a missed deadline is either equivalent to a catastrophic error (hard real time) or really terrible but non catastrophic error (soft real time). Soft real time applications like audio (no one dies because of a buffer underrun) can sacrifice the maximum worst case for better average case, because the worst case depends on load and can be worked around. Hard real time cares a lot more about the maximum worst case, like say a CNC machine where missing a deadline can cause a mechanical failure and break the part.
> a missed deadline is either equivalent to a catastrophic error (hard real time) or really terrible but non catastrophic error (soft real time)
I think you're conflating soft/hard real time with criticality (although they're often aligned)
Hard real time refers to a total system failure, even if it has trivial consequences. For example if a printer has a timing error at any point the page it's printing is irrecoverably ruined. It's hard real-time but low criticality.
Soft real time is where you can miss deadlines but the performance / experience degrades rather than fails. e.g. frame rate drops in a video game. The frames are still valid when delivered late, but the experience is poor.
> Slight nit, real time systems are designed for a maximum worst case latency within some spec, not minimum latency.
Indeed, in fact at times you don't even want a substantially lower minimum latency because it may cause unreasonable jitter in your output.
I think you're conflating soft/hard real time with criticality (although they're often aligned)
Hard real time refers to a total system failure, even if it has trivial consequences. For example if a printer has a timing error at any point the page it's printing is irrecoverably ruined. It's hard real-time but low criticality.
Soft real time is where you can miss deadlines but the performance / experience degrades rather than fails. e.g. frame rate drops in a video game. The frames are still valid when delivered late, but the experience is poor.
> Slight nit, real time systems are designed for a maximum worst case latency within some spec, not minimum latency.
Indeed, in fact at times you don't even want a substantially lower minimum latency because it may cause unreasonable jitter in your output.
> I'm not too familiar with kernel development so someone else should give you a definitive answer, but considering the tradeoffs it's very unlikely.
I believe most of PREEMPT_RT has already been merged over the years. The expensive bits are all guarded behind a compile time flag.
I believe most of PREEMPT_RT has already been merged over the years. The expensive bits are all guarded behind a compile time flag.
[deleted]
Do you have concrete examples of any of those things?
The only real tradeoff I am aware of is you sacrifice IO throughput so that the kernel can spread time related service tasks more fairly and userland process can interrupt the kernel. But, as modern Linux kernels work, it is not obvious you experience these throughput problems if you don't mark any tasks with realtime priority. You can still get low power consumption, low resource usage, high reliability, etc, with realtime requirements.
The only real tradeoff I am aware of is you sacrifice IO throughput so that the kernel can spread time related service tasks more fairly and userland process can interrupt the kernel. But, as modern Linux kernels work, it is not obvious you experience these throughput problems if you don't mark any tasks with realtime priority. You can still get low power consumption, low resource usage, high reliability, etc, with realtime requirements.
The degree to which the OS deviates from the vanilla kernel behavior depends on the requirements of the processes being run in real-time.
For example, to run hard real-time, high frequency (>10 kHz) tasks with low latency and jitter, one needs to disable CPU frequency scaling (among other things) when compiling the kernel. The result is higher power consumption.
For example, to run hard real-time, high frequency (>10 kHz) tasks with low latency and jitter, one needs to disable CPU frequency scaling (among other things) when compiling the kernel. The result is higher power consumption.
You don't have to do those things, it's just easiest to sometimes. It depends entirely on your hardware and your realtime requirements. If you do in-code synchronization you'll usually have problems with throttling but if you can synchronize against a stable timer/counter you will have fewer issues.
It's also worth noting that some of these tasks people throw out are just not a very good fit for even a real time kernel. The main use on Linux seems to be sound and video, which uses specialized hardware. The specialized hardware for other operations is typically not available on Linux capable SoCs and it's worth communicating with a different chip to do those things.
The premise of some of these tradeoffs is not always valid.
It's also worth noting that some of these tasks people throw out are just not a very good fit for even a real time kernel. The main use on Linux seems to be sound and video, which uses specialized hardware. The specialized hardware for other operations is typically not available on Linux capable SoCs and it's worth communicating with a different chip to do those things.
The premise of some of these tradeoffs is not always valid.
Forgive my ignorance, but could you explain a bit more why frequency scaling needs to be disabled at kernel compile time, and can't be disabled at runtime via sysctl?
I elided a lot detail for the sake of making a simple example about power consumption.
For applications that don't need hard real-time and/or aren't very high frequency, one could set the CPU frequency governor to always run at max frequency. You're right in that this can be done at runtime without a need to disable CONFIG_CPU_FREQ.
To achieve hard real-time at high frequency (roughly >10kHz), the standard kernel and PREEMPT_RT are not sufficient. One approach is to use a microkernel that preempts the Linux kernel. RTAI and Xenomai are two frameworks that do this. Processes run through them cannot be preempted by the kernel, so deterministic timings can be much better guaranteed. It's in this case that one would disable CPU scaling (along with Intel pstates, ACPI power management, and other things I don't remember offhand) when compiling the kernel. Changes in CPU frequency can change the frequency of the time counters on the CPU, and that opens up a vector for the vanilla kernel to interfere with the timers used by the microkernel and potentially break real-time.
Now, is it true in this latter scenario that one, as I said, "needs to" disable CPU scaling entirely? Now that you've got me thinking about it, probably not. If one guarantees that the 'performance' (i.e. max freq) governor is running at boot, leaving CONFIG_CPU_FREQ enabled may be fine. (The default governor will still need to be changed to the performance one in the kernel config, though.) Setting max frequency at runtime, however, maybe not. This is probably an edge case (I haven't tested this), but if the system boots with a dynamic CPU frequency enabled, the standard kernel, which cannot access information about threads controlled by the microkernel, could scale down the CPU a real-time thread is running on and mess up the timings. Changing the frequency mode at runtime after that happens may not fix that issue.
If I were to write that again, I'd use "should" or instead of "needs to." It's much simpler to disable frequency scaling in the menuconfig while disabling everything else than it is to leave it in and worry about it afterward.
None of this changes the point about real-time and power consumption, though.
For applications that don't need hard real-time and/or aren't very high frequency, one could set the CPU frequency governor to always run at max frequency. You're right in that this can be done at runtime without a need to disable CONFIG_CPU_FREQ.
To achieve hard real-time at high frequency (roughly >10kHz), the standard kernel and PREEMPT_RT are not sufficient. One approach is to use a microkernel that preempts the Linux kernel. RTAI and Xenomai are two frameworks that do this. Processes run through them cannot be preempted by the kernel, so deterministic timings can be much better guaranteed. It's in this case that one would disable CPU scaling (along with Intel pstates, ACPI power management, and other things I don't remember offhand) when compiling the kernel. Changes in CPU frequency can change the frequency of the time counters on the CPU, and that opens up a vector for the vanilla kernel to interfere with the timers used by the microkernel and potentially break real-time.
Now, is it true in this latter scenario that one, as I said, "needs to" disable CPU scaling entirely? Now that you've got me thinking about it, probably not. If one guarantees that the 'performance' (i.e. max freq) governor is running at boot, leaving CONFIG_CPU_FREQ enabled may be fine. (The default governor will still need to be changed to the performance one in the kernel config, though.) Setting max frequency at runtime, however, maybe not. This is probably an edge case (I haven't tested this), but if the system boots with a dynamic CPU frequency enabled, the standard kernel, which cannot access information about threads controlled by the microkernel, could scale down the CPU a real-time thread is running on and mess up the timings. Changing the frequency mode at runtime after that happens may not fix that issue.
If I were to write that again, I'd use "should" or instead of "needs to." It's much simpler to disable frequency scaling in the menuconfig while disabling everything else than it is to leave it in and worry about it afterward.
None of this changes the point about real-time and power consumption, though.
> where lowest latency is needed
Not so sure it would even be "the lowest", rather than "low enough, stable and predictable", even if it's higher on average than optimal.
For example, comparing a hashtable lookup to a BST lookup (and ignoring the memory hierarchy for the sake of example), the former would be faster on average, but the latter would produce more predictable lookup times without hiccups.
Not so sure it would even be "the lowest", rather than "low enough, stable and predictable", even if it's higher on average than optimal.
For example, comparing a hashtable lookup to a BST lookup (and ignoring the memory hierarchy for the sake of example), the former would be faster on average, but the latter would produce more predictable lookup times without hiccups.
[1] https://en.wikipedia.org/wiki/RTLinux
[2] https://en.wikipedia.org/wiki/Linux_Foundation#Real-Time_Lin...