I’d like to run Sculpt as NAS on my homelab (Intel Gen 9). And of course I want to have all the features ZFS provides for storage (RAID mirroring, snapshots etc.)
I know that currently there is a commonly used OpenZFS implementaion that is incorporated into FreeBSD system (in the kernel?) and as a loadable kernel module.
My only experience of porting something somewhere is proting pjsip with opus codes to Android circa10 years ago, so I’m not very experienced with this field
Imagine I’d like to bring ZFS support to SculptOS, could you roughly outline what path would I need to walk through in order to port it?
Also as a related question, what journey would challenge me if I’d like to add support to other filesystems like exFAT, XFS or EXT4?
Some quasi stream-of-consciousness items, waiting for someone to give better answers ^^:
I believe there is already support for ext4, possibly in the ‘genode-world’ repo instead of the main ‘genode’ repository
I seem to recall there is also a port of exFAT, but using an old (deprecated) Genode API, so you’d need to adapt it to new APIs before using it (it’s outlined in a genode-world ticket, probably one of the first tickets on the first page – edit : might be this ticket)
I’d start with something smaller first, maybe XFS indeed, before embarking on something like ZFS, which I hear is huge and is an “everything and the kitchen sink” FS, with deep dependancies into the Linux kernel (?), which would thus mean porting not just an FS but a whole lot of “stuff” on top of an FS
in Genode, file system support is provided simply by “FS plug-ins” – for instance a library called “vfs-fat.lib.so” (not the exact name, can’t recall ATM) is loaded into your vfs server, and bam, your vfs server supports fat.
in Genode, the vfs server (and thus its FS plugins) run in userland, making debugging of FS support easier
there is one difficulty when porting an FS though, even a ‘simple’ one like NTFS (let alone ZFS I assume) : the libc is not yet initialized at the time when the vfs server loads its plug-ins… So when porting e.g. XFS, you’d have to “re route” POSIX calls (like open(), close(), fstat(), read(), write()…).
That’s what was done to add NTFS support as an FS plug-in in the style of the new API (by opposed to the old Genode FS API)
edit : oh, there’s also the whole can of worms re., which FS interface you want to use, to interface with Genode. Some FS ports use the FUSe (File User in Userspace) interface, heck, I’ve even been known to port BFS, using a ‘port’ of the BeOS kernel file-system interface, so that I would then (in theory) get others Be FS ports “for free”, including their port of XFS… But I’m probably going into the weeds here… FUSe is a solid standard, for its part.
As @ttcoder’s list nicely demonstrates, there are a few considerations to make when dealing with file-systems on Genode (although that is also true on other OSes because there is always some form of VFS involved).
So let’s start with a brief overview of the architecture (that will contain a decent amount of technical details to give you starting points for exploration):
There is the File system session (os/include/file_system_session) that is implemented by the various server components, most prominently the vfs (os/src/server/vfs) server and used by client components to access a given file-system.
Internally this server makes use of the VFS vfs.lib.so (os/include/vfs and os/src/lib/vfs) provided as shared-library to assemble the file-system it exports to its clients. This library is also used for providing component-local VFS instances, like in components making use of the C library.
The VFS itself already contains internal plugins, e.g. fs that can be used to access another File system server, but can be extended through external plugins (also shared-libraries) as well, like vfs_rump.lib.so (dde_rump/src/lib/vfs/rump) for providing ext2 support via NetBSD’s rumpkernel. vfs_rump in return uses the Block session directly (dde_rump/src/lib/rump/io.cc) to access the underlying storage device.
I used lwext4 to implement the vfs_lwext4 plugin but at this point I would consider it to be in “maintainance-mode”. That being said, in contrast to vfs_rump it accesses the Block session via the VFS block plugin.
As you may have noticed, one of the purposes of the VFS is to provide access to Genode’s services through file I/O mechanics, e.g. /dev/block serves the same purpose as /dev/sda1 on other OSes.
So… to bring support for a new file-system to Genode—your mission, should you choose to accept it—would be to write a new VFS plugin. As the context in which a VFS plugin operates is constrained, making use of the C library file I/O, as is commonly done in FUSE based file-systems, not possible (as of now) and porting such file-system implementation can be difficult (more on that later).
(As a long-time ZFS user myself, having it on Genode would be awesome—so you are not alone with that desire .)
Well, porting ZFS will be quite involved, as in “it is a marathon not a sprint”. On the one hand you have to port the file-system driver, i.e., the kmod (implement the SPL (Solaris Porting Layer) to interface with the OpenZFS code), and on the other hand you have to port the tools, like zdb, zpool and zfs, and have to make them work with each other.
Which brings my to the second part of your quote:
Since the upstream implementation of ext4 is maintained within Linux (for obvious reasons), using the DDE Linux approach (providing a Genode C API for the File system session and “carving” out Linux’ VFS-layer plus ext4) to port it over feels natural and is the direction I would take. The added benefit is (potentially) easier access to other file-systems as well in the future, i.e., when the first ported Linux file-systems works sufficiently, getting additional ones should require less effort. That would also include ZFS: we would ride piggy-back on the SPL implementation of OpenZFS on Linux.
That being said, this is still involved, which is why I would not recommend that as a spare time project. However, I found creating a minimal FUSE implementation (still available in world/legacy but more or less left untouched besides some kneading by @ttcoder) for Genode and porting a few selected FUSE file-systems over some years ago rewarding.
If you want to get your toes wet in that regard, I think looking into, for example, fuse-exfat and creating a vfs_exfat plugin based on libexfat might be a more approachable exercise. IIRC the API is similar to lwext4 and thus vfs_lwext4 could serve as a blueprint. Looking into fuse-exfat should demonstrate what libexfat API calls you would have to make in the VFS plugin’s implementation. The block I/O calls warrant some consideration but these could be implemented within the plugin (again, similar to how vfs_lwext4 provides some C library functions).
In the interm, running ZFS in a VM on top of Sculpt OS might be a workable stop-gap solution and could play nicely in the headless-server Sculpt OS variant others talked about during the roadmap discussion.
.)
) for Genode and porting a few selected FUSE file-systems over some years ago rewarding.
