Thread Priorities, Starvation, and Real-Time Workloads in Genode

Developer's Corner
1

I have been investigating thread priorities in Genode 26.05 on x86_64 with the NOVA microhypervisor. I would like to share my findings and get feedback on how priorities should be managed in a Genode-based system, particularly to prevent starvation and support time-sensitive workloads such as audio and video playback.

As part of this investigation, I implemented a PriorityThread abstraction that allows a thread’s priority to be specified when it is created. I then traced the priority values through Genode’s NOVA backend to verify that they reach the underlying scheduling contexts. My tests confirmed that the priority values are propagated to NOVA and that changing a thread’s priority affects its execution behavior in a CPU-bound workload.

For example, I tested three workers with different Genode priorities:

  • Worker 0: priority 0

  • Worker 1: priority 2000

  • Worker 2: priority 4000

In this configuration, lower Genode priority values correspond to higher priorities. The observed mapping to NOVA priorities was 64, 63, and 61, respectively.

In a continuous CPU-bound test, the highest-priority worker completed multiple work units during a 10-second observation interval, while the two lower-priority workers completed none.

This result appears consistent with the priority-based scheduling policy described in the NOVA documentation. I would like to make sure I understand this behavior correctly and have not overlooked anything.

Is assigning priorities to individual threads an appropriate way to control execution in Genode, or is the preferred approach based on other mechanisms, such as CPU-session configuration, resource quotas, scheduling contexts, or component architecture?

More specifically:

  • How should priorities be assigned to threads belonging to different components?

  • Is starvation an expected consequence of strict priority-based scheduling with NOVA, and should applications be designed with this behavior in mind?

  • Are there established ways to ensure that lower-priority threads still receive sufficient CPU time?

  • How should priorities and CPU resources be managed across components to prevent a single component from monopolizing CPU resources?

The Main Concern: Real-Time and Multimedia Workloads

My primary concern is the reliable execution of time-sensitive services, particularly audio and video playback, in a Genode-based system.

For example, video playback involves several activities that must make progress within appropriate time limits:

  • Video decoding

  • Audio decoding and playback

  • Display composition and rendering

  • Graphics and display driver activity

  • Storage and network I/O

  • Other system services running in the background

If CPU-bound work in one component prevents another activity from running on time, the result may be audio dropouts, video stuttering, or degraded playback quality.

What is the recommended approach to prioritizing these activities in Genode?

For example:

  • Should audio processing and playback receive higher scheduling priorities than video decoding and background tasks?

  • How should priorities be assigned to drivers and services that support time-sensitive workloads?

  • Are there recommended mechanisms to ensure that audio processing meets its deadlines without starving other components?

  • Does Genode provide an established approach for coordinating scheduling priorities across component boundaries?

  • Are there existing examples or configurations that demonstrate how to configure a system for smooth audio and video playback?

I understand that scheduling priorities alone cannot guarantee real-time behavior. I would also like to understand how CPU allocation, execution budgets, interrupt handling, and multimedia component architecture should be considered together.

Finally, do you consider explicit per-thread priority control a useful addition to Genode, or would you recommend a different approach to this problem?

If per-thread priorities are useful, what policy would you recommend for managing them across components while maintaining responsiveness for time-sensitive workloads and preventing starvation?

Any pointers to relevant documentation, existing implementations, or practical examples would be greatly appreciated.

Thank you for your time and feedback.

1 Like
2

Your results match what I would expect from strict fixed-priority scheduling: if a higher-priority thread never blocks, the lower ones simply never run, so starvation is a property of the policy rather than a bug. Because of that, I’d treat priorities as a coarse tool and keep CPU-bound work in lower bands while only short, event-driven threads such as audio or input handlers sit at the top, since those block quickly and hand the CPU back. Using separate priority bands per component class, set from the init configuration rather than inside each application, also makes it easier to reason about who can monopolize the CPU. It would be interesting to see whether a video pipeline with a bounded-buffer between decoder and display still hits its deadlines when a CPU-bound worker shares the same core. Pinning the real-time components to their own CPU via affinity might be another cheap way to avoid contention.

3

Thank you for your response. Your explanation raised several additional questions for me.

  1. What is your view on NOVA’s strict priority-based scheduling policy?

Given that NOVA is not primarily designed for hard real-time workloads, do you think strict priority scheduling is the best choice, or could a hybrid approach that balances priority with fairness and CPU-time allocation be more suitable for systems running both real-time and general-purpose workloads?

What do you see as the main advantages and trade-offs of these approaches?

  1. Is a microkernel that permits priority-induced starvation a suitable foundation for a general-purpose operating system?

I understand that starvation can be an intentional consequence of strict fixed-priority scheduling rather than a kernel bug. However, in a general-purpose operating system, many components need to make progress concurrently, even when some of them are continuously CPU-bound.

In mainstream operating systems, runnable threads are generally scheduled across available logical CPUs, subject to affinity constraints and the scheduler’s policies. Load balancing can also move threads between CPUs when appropriate. In Genode, however, CPU allocation and scheduling are more explicitly structured, and the scheduling behavior can depend heavily on how scheduling contexts and CPU resources are configured.

Does this make priority-induced starvation a more significant concern in Genode-based systems? How is fairness expected to be achieved when a CPU-bound thread remains continuously runnable at a higher priority than other threads sharing the same CPU?

Does this mean that preventing starvation must be handled entirely through system architecture and application design, rather than by the scheduler itself?

More specifically, what mechanisms or established practices does Sculpt recommend to ensure that one runnable thread or component cannot indefinitely prevent other components from making progress?

I would also like to understand how this approach scales beyond carefully controlled real-time workloads to systems running browsers, storage services, networking, multimedia, and other general-purpose applications simultaneously.

  1. How should component-level priorities prevent storage and networking workloads from interfering with real-time multimedia?

Suppose a Genode-based system assigns relatively high priorities to storage and networking drivers or services because they are considered important system components.

During a large file copy, a storage-related thread might continuously consume CPU time. Similarly, a large download could generate sustained activity throughout the networking stack and its associated services.

If such a workload monopolizes a CPU core, lower-priority threads responsible for video decoding, audio processing, display composition, or other time-sensitive activities could miss their deadlines.

For example, high-resolution online video playback could suffer from audio dropouts or video stuttering while a large download or file-copy operation is running concurrently.

Assigning priorities based on component classes does not appear to solve this problem by itself, because the importance of a component does not necessarily correspond to the urgency of every thread it runs.

How is this situation intended to be handled in Genode and NOVA?

Should storage and networking components use lower priorities for bulk data processing, while latency-sensitive operations receive higher priorities?

Is there a mechanism to limit the CPU time consumed by a component or scheduling context, rather than relying exclusively on priority levels?

When a time-sensitive operation depends on services running at lower priorities, how should priorities and scheduling resources be coordinated across component boundaries?

When sufficient CPU cores are available, is CPU affinity the intended solution? Or is there a recommended approach that also works on single-core systems?

Ultimately, I would like to understand whether the intended solution is to rely on architectural discipline and component design, implement additional scheduling and resource-management policies at the Genode level, or use a different scheduling model for workloads that combine general-purpose computing with real-time multimedia requirements.