Reorganise folder hierarchy

General
1

I am new to Genode, and found the concepts good, and there seems to have been done some good work over the years on the OS and the tools for porting new software, however I got one major complaint about the whole project, everything seems organised like a hobby project.

What I am referring to is how each new “exotic” hardware platform gets its own repository on Github, instead of just an arch folder in the mainline Genode repo, or perhaps just a new branch until it is ready to be merged into mainline.

It is also a problem how the Genode repo contains “repos”, and in it, it contains a mess of files, including hello_tutorial, demo, base, os, pc, etc. This makes it impossible to know what is contained within without scrolling down and getting an explanation in the readme, or opening the folders. This is improper as the naming and ordering should be mostly self-explanatory, instead of jumping back and fourth in the readme and/or manual(s). It also makes no sense to put all of these files in the same folder, and from the readme it is not apparent why “base” and “pc” exists, and it isn’t clearly until entering “pc” that it is heavily tied to x86 platform.

It also gets apparent that the folder hierarchy is a mess when trying to get an idea of how goa works. If you go into goa/examples/hello_make/used_apis, it says genodelabs/api/base. Ok, where does this exist? Where does the posix api exist? There isn’t an api repo in genodelabs with such a name. There isn’t a folder named “api” in goa/share/goa (why is the name reused in the same tree?). There isn’t any apparent location inside genode/repos/base either.

It cannot be that one needs to put on their detective hat to figure out where things are, or get a general idea of their purposes without having to read the whole documentation and remembering it by heart. There is a reason why we don’t just use a random string generator for naming schemes, even if those names are well documented in the manual. So please reorganise and/or rename the folders/files/links so the project is maintainable by others than just people close to whomever wrote it to begin with.

2 Likes
2

What a charming way to enter a room.

4 Likes
3

I know it is fairly harsh way to enter the forum, but I believe it is needed. I mean you could compare your git structure to other projects, see Linux, openBSD, XNU, CMAKE, etc. They attempt to flatten the structure, and keep each repo only containing relevant files to that specific repo. They attempt to make links follow the same folder structure. They attempt to keep the naming schemes human readable and unique in the structure. They attempt to keep the special cases in their own subfolders. Like, repos should just be the base of the repo. base-* probably shouldn’t be there, rather treated similar to different instruction sets - arch folder, specific architecture in subfolder.

4

I apologize if you found this thread already, but offer it as evidence that Genode does put some thought into the structure of its various repositories.

5

I believe you should keep it cleaner, more readable, and thus more maintainable, by moving the ports out of the genodelabs/genode repo, and instead keep them in another, for examples genodelabs/Ports or could called it packages. This way it makes it much more clear what is relevant to the OS framework, and what is not. This is a similar structure to openBSD and freeBSD.

So keep genode-world out of the genodelabs/genode, and stay in genodelabs/genode-world (quite redundant naming to call it genode-world instead of world in the genodelabs repo, but I can live with it).

6

The project also appears way too tightly linked to the authors’ setup. For example base/board contains pc, which sets up the args for virtualisation, for an arbitrary x86 CPU. What if you want to test it on other archs? One would naturally think you should go to arch folder for all arch related things and have the hardware abstracted after that point. It doesn’t tell you it is that specific arch unless you read the file content.
The package manager is designed to link to the authors personal pages and then know by heart who got which packages in which version, because you talked to them in the coffee break, and if you have a lot of packages you scroll through the packages’ names (not categories) as a madman.

7

Have you had a go at rearranging your fork in accordance with the ideas you propose? I have only ever submitted a handful of pull requests (not in Genode), and in those cases only added or edited files and maybe add a folder for my own work, so do not know the practicality of submitting a completely reordered directory structure for merge upstream. But it would still be a useful proof of concept.

Whilst I think you might need to spend a little more time here before you change everything about, I encourage you not to give up exploring Genode. Whilst my volunteering tends to be outside software, my experience is that those willing to volunteer their time into creating order and keeping things tidy are extremely valuable to their projects. Welcome onboard!

8

Dude,

I do not work for Genode Labs, nor am I a moderator for this forum, so I speak only for myself. But I think it’s safe to say that everyone in this forum enjoys the respectful atmosphere that we’ve cultivated over the years in the Genode community. Even when we disagree, we do it with civility and humility.

TL;DR: Please bring your points/questions back without the rudeness and arrogance, and I suspect you will get much better results. I recommend asking why it is designed that way first, then making suggestions for improvement (if applicable).

A few quick points:

  1. Genode is not a hobby project. Genode Labs is a successful company, with customers shipping actual products running on different hardware platforms, all running various configurations of Genode. So the system for maintaining hardware platforms works well.
  2. I can assure you that such key components of the Genode project are well thought-out. The entire project was founded on questioning certain orthodoxies of kernel design, and they have followed this model successfully in many areas over the years, often with exciting results.
  3. To follow on that point, when they find that something is no longer optimal, or stands in the way of progress, they are not afraid to change it (e.g., the build tools, the Genode World re-organization, the display driver redesign, removing C++ exceptions from the code, etc.). Thoughtful design in the first place has made these efforts smoother than they would have been on many other projects.
  4. The documentation is extremely good, and very readable. I also personally find it amazing how much time and effort the Genode Labs team put into supporting and explaining things to the open-source community!

I look forward to following this discussion in the above manner. :slight_smile:

5 Likes
9

While it could have been structured in a more HR way, I do not believe the criticism was destructive, rather constructive as I pointed out why something is a problem, and an alternative solution to it.

Also, I didn’t say it was a hobby project, but it was structured like one, which I do believe is a fair statement. The structure follows due to a small and tightly knit developer base, so what they feel is reasonable, because they’re very familiar with the code base, isn’t good practice.

I also didn’t say their documentation was bad, rather referred to it being good. Same goes for the implementations. However, the structure is very lacking, because it makes it 10x harder to understand the codebase if you don’t already know the codebase. The structure doesn’t keep the codebase compartmentalised nor follow a natural order.

10

I can relate to this topic in a way that indeed the system is unusual and confusing to me.

Maybe I’m too biased towards Unix and Linux systems and thus it’s hard for me to pick up the logical structure of the system.

I already finished almost all Foundations book, played around with the system, but still struggling to navigate through it and understand what is happening under the hood of the Sculpt, Leiptzentrale and package manager in the terms of Foundation book.
I even managed to build my own Sculpt/Base-HW but still not fully understand what I did and how to control it.

So I believe I just need to read something else and tinker more with it to get understanding and have a conclusion about the current folder hierarchy.

2 Likes
11

I even managed to build my own Sculpt/Base-HW […]

Good job! Given that you haven’t even asked a single question along your way, that’s quite an an accomplishment.

[…] but still not fully understand what I did and how to control it.

Don’t worry and let us be realistic. Understanding the architecture, configuration, tooling, integration, and package management of an operating system - regardless of Genode or otherwise - is not a simple feat after all. It is certainly not intuitive. Intuitiveness is too vague a term to be useful in a discussion because intuition is an individual trait. I’m afraid the project structure and tools prioritize the empowering of informed users over the catering of existing preconceptions. To draw an analogy, Vim is certainly inapproachable for people who already have a strong preconception of how a text editor has to behave. They may find it repelling. But it is loved by actual users once after reaching a certain degree of competence in how it works.

You may find the various sub directories under repos/ or the use of multiple Git repositories confusing. But these are deliberate instruments for the clear-cut separation of concerns. It gives customers of Genode Labs a passable way to attain licensing hygiene, keeping proprietary supplements and modifications almost galvanically separated from upsteam Genode. And it enables Genode’s developers and maintainers a way to assert, manage, and limit their individual responsibilities. A measure against becoming burned out by open-source-maintenance fatigue. Without these instruments, Genode would not exist (and persist to exist) in the first place.

To appreciate it, a learning curve must be mastered. It would be wrong to pretend otherwise. Good documentation should be your friend while learning. If you find the documentation lacking in some area, please ask for directions. So we can improve the documentation and help you along the way.

So I believe I just need to read something else and tinker more with it […]

Tinkering, definitely! Assuming that you are already acquainted with Sculpt’s high-level documentation, I think the best way to attain a profound understanding how Sculpt is put together would be to try digesting its construction plan, the sculpt.run script and the accompanied configuration snippets. You may also find the section about special-purpose Sculpt systems of the release documentation helpful. The studying will bring up a lot of questions. Try to investigate them, first your own, or - when you get stuck - by seeking advise here in the forum. :smiley:

3 Likes
12

**“**I’m afraid the project structure and tools prioritize the empowering of informed users over the catering of existing preconceptions.”

How to build other projects:
make

Scripts will then use sane defaults to look for missing tools, and either prompt you how to install them, either from binary or from source, or simply attempt to do either of them. This allows new users to try it out and make small changes without needing to have complete information. Usually there is a config file to pick other platforms then currently detected platforms, and other desired tweaks.

Tries to compile Genode:

make

*fails*

*Looks for explanations in the readme - none*
*Goes to genode.org, finds genode-applications*

*Only explains how to do it for x86 - but you use one of the modern arm64 laptops*

*Goes to https://genode.org/download/tool-chain and follows instructions for compiling from source*

make -f ./tool/toolchain

“Need to have GNAT installed and the GNAT version must match the GCC version (found GCC 15.2.0).”

*Knows it is already installed, and you are using ubuntu as specified in the manual, but remembers current gcc is too new, so follows conventions*

make CC=gcc-14 -f tool/tool_chain

*Still fails*

*Checks manual - nothing*

*Checks readme - nothing*

*Opens tool/tool_chain in text editor*

“HOST_GCC_VERSION := $(shell gcc -dumpfullversion -dumpversion)

GNAT_OK = $(call check_equal_f,$(shell gnatmake --version | sed -n -e ‘s/GNATMAKE //p’),$(HOST_GCC_VERSION),\

        Need to have GNAT installed and the GNAT version must match the GCC version (found GCC $(HOST_GCC_VERSION)).)“

*Dies inside*

*Fixes*

"CC ?= gcc

HOST_GCC_VERSION := $(shell $(CC) -dumpfullversion -dumpversion)

GNAT_OK = $(call check_equal_f,$(shell gnatmake --version | sed -n -e ‘s/GNATMAKE //p’),$(HOST_GCC_VERSION),\

        Need to have GNAT installed and the GNAT version must match the GCC version (found GCC $(HOST_GCC_VERSION)).)”

make CC=gcc-14 -f tool/tool_chain

*Finally some process*

make CC=gcc-14 -f tool/tool_chain aarch64

*While letting it build you check the manual - manual doesn’t describe how to build the system*

*Finds out the description on the home page is wrong, you need “genode-foundations” instead*

*Reads terminal outputs - confused why it says tool_chain-25.05 when 25.11 is released*

*Ignores and accepts the insanity*

./tool/create_builddir aarch64

“tool/create_builddir:52: *** Bad platform argument ‘aarch64’. Stop.”

./tool/create_builddir

*Finds out need to specify it more specifically here*

./tool/create_builddir arm_v8a

cd build/arm_v8a

*Finally finds a make file*

*Cries inside that this insanity should hopefully be done*

make

make run/demo KERNEL=linux BOARD=linux

*Errors*

*Finds this forum and sees main dev essentially says it is a skill issue, even though he violates common conventions for no upside, and says the users should just read their superior documentation*

*Genode claims to be an OS framework and modular but uses one large document for documentation rather than keeping it modular and having each module have a readme - ???*

*Documentation only explains exactly how the devs are using the tools and not anything for other platforms, and no clear documentation for platforms - ???*

*Finds the current structures extremely hostile for new contributors - incentivised to give up*

13

Sounds like you would have gained from getting familiar with the Genode build process on an ‘intel’ platform first, there it’s just a matter of downloading the pre-built toolchain. (At least that’s how I do it ^^)
”Crawl, walk, run”.
After trying it out the ‘easy’ way, I suppose it would make it easier to then try to build for aarch64, since you would have some ‘reference’ build logs for comparison purposes ?

14

What I meant was, it seems needlessly hard to get started. Would be good to have an easy way to make an initial built running with minimal effort. We’re not talking about cross compiling or any modifications to the software. It seems a bit overkill having multiple locations for different configs, and would be nice if more tested and tried conventions were followed. I fail to see how providing some sane defaults with options for further modifications is a negative.

Why not have a makefile in the main repo, which lets you do the most expected events. So make checks the platform you are on, looks if it can find preinstalled toolchain for the platform, tries to compile for that platform, and run a demo. Then you don’t need to deal with figuring out where things are and in which folders they are, and that one place the arg is aarch64 and somewhere else it’s arm_v8a, etc.

The hierarchy seems to needlessly complicate the whole process, and it does require a lot of extra steps to figure out compared to other projects, which makes it seem less intuitive for anyone new.

15

I followed a similar path with GNAT and other requirements actually.

The difference only that I tried to build on Arch Linux (so expected to experience differences with installing the required packages) and I build base_hw for intel.

Actually one question I had in my had about the process:

Why do I need GNAT to compile for x86_64 platform?
As far as I reasoned about the code of the build process it wasn’t actually used. But requirement for compilation is there…

If this is the case it would be nice to adapt build requirements for installed tools depending on the platform :thinking:

16

@G0retZ:

GNAT is needed for ADA/Spark, which we don’t really use anymore. You can build the tool chain without ADA support as follows

tool/tool_chain x86 ENABLE_FEATURES="c c++ gdb"

or download the prebuilt tool chain Genode - Genode tool chain that contains x86/x86_64/riscv/arm/arm64 compiler versions. Without arguments tool_chain prints a help message with the available options. Unfortunately, this doesn’t happen when GNAT is not installed (that should to be fixed).

We update the tool chain every two years, so the version is always the Genode release the tool chain was published with. Currently it is 25.05, the next will be 27.05.

1 Like
17

@SyntaxStorm:

make run/demo KERNEL=linux BOARD=linux

KERNEL=linux BOARD=linux would only work for arm64 if you execute it from an arm64-based Linux, because it will start the scenario on Linux directly.

Please try:
make run/demo KERNEL=hw BOARD=virt_qemu_arm_v8a
This creates an image with our custom kernel and executes it in Qemu with VirtIO enabled.

Regarding the tool chain, I wonder if there is a specific a reason why you have to build it yourselves?

18

Oh, good to know I could avoid tinkering around GNAT installation!

Installing GNAT is kinda a challenge of it’s own, tbh :sweat_smile:

Would be nice to have this information in manual somewhere if the tool chain will stay as it is now for the next 1.5 years.

wdyt?

19

I will answer your question for myself.

I’m an Arch user and don’t know how to install debian or ubuntu packages without breaking my neck in a rabbit hole :upside_down_face:

So I simply followed these compile instructions to get the latest version.

Now I know that tool_chain version is updated only once in 2 years. In this case there’s no point indeed in trying to build from source “the latest one” because it probably would be the same as I can find in AUR :thinking:

I wish I knew this earlier and save some time and efforts :sweat_smile: