Trying to compile and run `core` with a custom compiler

Developer's Corner
1

Hi everyone,
I am currently trying to add a fuzzer to Genode in order to fuzz different sessions. Fuzzers come in many different flavours, but it is generally most efficient to instrument the existing source code. Further the fuzzer needs to support GCC. So AFL++ seemed like a good candidate. It is widely used, has many optimizations/features and comes with built in GCC support. In order for the fuzzer to work properly, the code to be tested needs to be instrumented with custom gcc/g++ compilers.

There is one challenge I am stuck at, which lead to this forum post.

TL;DR:
When compiling with custom compilers and running core, the needed memory (stack, bss…) never gets assigned. Core then tries to write to the unassigned memory and thus crashes.

In more detail:

Currently core is being built and run like this:

make core CUSTOM_CC=/path/to/afl-gcc-fast CUSTOM_CXX=/path/to/afl-g++-fast
./core/linux/core-linux

This leads to an error Segmentation fault (core dumped).

After executing the binary in gdb we get a bit more information. Here _start refers to the program entry-point:

Starting program: /home/g-user/code/genode/build/x86_64/core/linux/core-linux 

Program received signal SIGSEGV, Segmentation fault.
_start () at /home/g-user/code/genode/repos/base/src/lib/startup/spec/x86_64/crt0.s:61
        /* init_rtld relocates the linker */
61		call init_rtld

Upon further investigation it turns out that call init_rtld tries to write to the memory region .bss but the stack and memory has never been assigned any memory, as seen in the process information (/proc/<id>/maps). The write therefore leads to a crash.

Additional information:

  • Genode is being run and a Linux x86 Debian VM.
  • The compiler requires many libc functionalities. In order to have a successful build, I created my own implementation that is not completely finished. This is then statically linked to the custom compilers. This should probably not matter, as the execution crashes so early.

Exactly pin-pointing the issue is quite difficult as I’m fairly new to Genode. My current best guess is, that the problem lies with the linker. Maybe the AFL compiler does not link to the correct functionality that core needs.

Does anyone have any ideas or hunches how to solve this problem?

Cheers and thanks for your time

1 Like
2

Hi,

at this point the dynamic linker cannot access nor allocate any memory. The .bss section is part of the ELF (ld.lib.so) and should be accessible once the ELF is loaded. .bss should also be zeroed by the ELF loader. These are the assumptions at this point. Unfortunately I have no idea what goes wrong in your case.

3

One additional thought: It can also be that the global offset table (GOT) contains all zeros and no valid pointers. I had a lot of trouble with this on RISC-V. This would cause the call to init_rtld to dereference as null pointer. You can check this by inspecting the .got section of ld.lib.so and see if there is any data present.

4

While I generally agree that the ELF loader has to prepare a zero’d BSS, I disagree that ld.lib.so is of any relevance as core-linux crasshes. Please correct me if I’m mistaken.

5

@chelmuth: You are right. Since core is a static binary init_rtld calls init_cxx_guard which constructs Blockers via. unmanaged_singleton (looks like in BSS). Something could go wrong with the compiler there. @smhm: Did you check that a basic scenario runs with our tool chain on your test machine?

6

Yes, running something like make run/log works perfectly.
EDIT: And by default this then uses the genode toolchain.

7

@smhm could you send the output of

objdump -ph bin/core-linux

of the broken version?

8

Gladly!

 $ objdump -ph bin/core-linux

bin/core-linux:     file format elf64-x86-64

Program Header:
    LOAD off    0x0000000000001000 vaddr 0x0000000001000000 paddr 0x0000000001000000 align 2**12
         filesz 0x00000000001dbf44 memsz 0x00000000001dbf44 flags r-x
    LOAD off    0x00000000001dd000 vaddr 0x00000000011dc000 paddr 0x00000000011dc000 align 2**12
         filesz 0x0000000000043320 memsz 0x00000000055f93cc flags rw-
    LOAD off    0x0000000000000120 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**12
         filesz 0x0000000000000000 memsz 0x0000000000000000 flags r--
    LOAD off    0x0000000000000000 vaddr 0x0000000040000000 paddr 0x0000000040000000 align 2**12
         filesz 0x0000000000000000 memsz 0x0000000010000000 flags ---

Dynamic Section:
  HASH                 0x000000000120c5a8
  STRTAB               0x000000000120c070
  SYMTAB               0x000000000120bb18
  STRSZ                0x0000000000000535
  SYMENT               0x0000000000000018
  DEBUG                0x0000000000000000
  PLTGOT               0x00000000011e3838
  PLTRELSZ             0x0000000000012ab0
  PLTREL               0x0000000000000007
  JMPREL               0x000000000120c730
  RELA                 0x000000000120c730
  RELASZ               0x0000000000012a98
  RELAENT              0x0000000000000018
  FLAGS_1              0x0000000008000000
  RELACOUNT            0x0000000000000c6f

Sections:
Idx Name          Size      VMA               LMA               File off  Algn
  0 .text         001d796e  0000000001000000  0000000001000000  00001000  2**5
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  1 .plt          00000020  00000000011d7970  00000000011d7970  001d8970  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  2 .ctors        00000018  00000000011d7990  00000000011d7990  001d8990  2**3
                  CONTENTS, ALLOC, LOAD, DATA
  3 .eh_frame_hdr 0000459c  00000000011d79a8  00000000011d79a8  001d89a8  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  4 .data         00006e38  00000000011dc000  00000000011dc000  001dd000  2**5
                  CONTENTS, ALLOC, LOAD, DATA
  5 .got          00000a00  00000000011e2e38  00000000011e2e38  001e3e38  2**3
                  CONTENTS, ALLOC, LOAD, DATA
  6 .got.plt      00000020  00000000011e3838  00000000011e3838  001e4838  2**3
                  CONTENTS, ALLOC, LOAD, DATA
  7 .eh_frame     0001e79c  00000000011e3858  00000000011e3858  001e4858  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  8 .gcc_except_table 00009ad9  0000000001201ff8  0000000001201ff8  00202ff8  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  9 .interp       0000001c  000000000120bad1  000000000120bad1  0020cad1  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 10 .note.gnu.build-id 00000024  000000000120baf0  000000000120baf0  0020caf0  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 11 .dynsym       00000558  000000000120bb18  000000000120bb18  0020cb18  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 12 .dynstr       00000535  000000000120c070  000000000120c070  0020d070  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 13 .hash         00000180  000000000120c5a8  000000000120c5a8  0020d5a8  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 14 .stapsdt.base 00000001  000000000120c728  000000000120c728  0020d728  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 15 .rela.dyn     00012ab0  000000000120c730  000000000120c730  0020d730  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 16 .dynamic      00000140  000000000121f1e0  000000000121f1e0  002201e0  2**3
                  CONTENTS, ALLOC, LOAD, DATA
 17 .tbss._ZZN12_GLOBAL__N_110get_globalEvE6global 00000010  000000000121f320  000000000121f320  00220320  2**4
                  ALLOC, THREAD_LOCAL
 18 .tbss         00000064  000000000121f320  000000000121f320  00220320  2**4
                  ALLOC, THREAD_LOCAL
 19 .bss          055b5fcc  000000000121f400  000000000121f400  00220320  2**8
                  ALLOC
 20 .stack_area   10000000  0000000040000000  0000000040000000  00221000  2**0
                  ALLOC
 21 .note.stapsdt 000000e8  0000000000000000  0000000000000000  00220320  2**2
                  CONTENTS, READONLY
 22 .gnu_debuglink 00000018  0000000000000000  0000000000000000  00220408  2**2
                  CONTENTS, READONLY
9

@smhm: Your core-linux is a dynamic binary (not statically linked). This cannot work.

1 Like
10

Some notes:

  • you need to use genode.ld for linking core not genode_dyn.ld
  • for core-linux you also need stack_area.ld
  • we do not support position independent executables (pie) for Genode on Linux
  • you can verify your core-linux with the objdump output, in case .dynamic, .dynsym, .dynstr and the “Dynamic Section: …” all disappear, you are good
3 Likes
11

Interesting. Thanks a lot!

I will give an update if/when I find a solution for future readers.

2 Likes
12

Update 1:

TL;DR:
Add LD_CMD += -no-pie -Wl,--build-id=none when linking.

Using file showed me, that the binary is apparently already statically linked. But the dynamic linked sections still showed up. I further noticed that PIE is also being enabled default enabled. See the memory maps for the binaries:

Process mappings from genode toolchain compiled binary:

           0x10c4000                       0xc4000     0x1000  r-xp   /home/g-user/code/linux-genode-build/core-linux
           0x10c4000          0x10ee000    0x2a000    0xc5000  rw-p   /home/g-user/code/linux-genode-build/core-linux
           0x10ee000          0x64a4000  0x53b6000        0x0  rw-p   
          0x40000000         0x50000000 0x10000000        0x0  rw-p   [heap]
      0x7ffff7ff9000     0x7ffff7ffd000     0x4000        0x0  r--p   [vvar]
      0x7ffff7ffd000     0x7ffff7fff000     0x2000        0x0  r-xp   [vdso]
      0x7ffffffde000     0x7ffffffff000    0x21000        0x0  rw-p   [stack]

Process mappings from afl-compiler compiled binary:

      0x7fffa7e01000     0x7fffa7fdd000   0x1dc000     0x1000  r-xp   /home/g-user/code/genode/build/x86_64/core/linux/core-linux
      0x7fffa7fdd000     0x7fffa8021000    0x44000   0x1dd000  rw-p   /home/g-user/code/genode/build/x86_64/core/linux/core-linux
      0x7fffa8021000     0x7fffad5d7000  0x55b6000        0x0  rw-p   
      0x7fffe6e01000     0x7ffff6e01000 0x10000000        0x0  rw-p   [heap]
      0x7ffff7ff9000     0x7ffff7ffd000     0x4000        0x0  r--p   [vvar]
      0x7ffff7ffd000     0x7ffff7fff000     0x2000        0x0  r-xp   [vdso]
      0x7ffffffde000     0x7ffffffff000    0x21000        0x0  rw-p   [stack]

As we can see, the first entry in the memory mappings for the afl-compiler are different compared to the toolchain compiled binary. It could indicate the usage of PIE and ASLR, which is a key feature for fuzzing scenarios, where deterministic memory layouts could lead to predictable results and reduce test coverage. So this makes sense that this happens. Adding -no-pie removes this issue.

Adding this flag did not completely solve the issue, namely the build-id section (.note.gnu.build-id) is added and now did not receive any special memory location and was mapped to 0x0. This breaks everything again. Adding -Wl,--build-id=none to LD_FLAGS removes this section and finally core does not stop at call init_rtld.

Now another issue arose: for some reason the the following assembly code gets generated:

        /genode/repos/base/src/include/base/internal/unmanaged_singleton.h:83
        if (!object_constructed) {
        ...
   0x00000000012085d7 <+69>:	je     0x1208648 <unmanaged_singleton<Genode::Registry<Genode::Registered_no_delete<Genode::Semaphore> > >()+182>
   0x00000000012085d9 <+71>:	mov    $0xffffffffffffff90,%rax
=> 0x00000000012085e0 <+78>:	mov    %fs:(%rax),%eax
   ...

At this instruction it crashes again, as the the memory at address $0xffffffffffffff90 cannot be accessed, there is no memory mapped to it.

Any ideas here?

This is probably not directly related to genode and an AFL-compiler problem.

13

%fs contains the thread-local storage (TLS) pointer on Linux. The 0xffff...90 above is probably just a negative offset in rax. None of the microkernels we support offers TLS in %fs. On the other hand: Why is %fs not valid for Genode on Linux? I have no answer for that at the moment. I would try to disable TLS in case AFL has an option for that.

14

The crash at $0xffffffffffffff90 at least implies that %fs is 0.

1 Like
15

While the AFL++ port is not yet finished, I can give an solve the issue here.

I realized that using a custom compiler would turn out to be quite the challenge. So I took a step back and saw that what I was actually working with was a GCC Plugin. What I missed was, that AFL++ adds this plugin dynamically and not from a Makefile/CMD line.

This means the default Genode compiler setup can be used and by adding -fplugin=/path/to/afl-gcc-pass.so the code gets instrumented as desired!

3 Likes