Taking NetBSD kernel bug roast to the next level: Kernel Fuzzers (quick A.D. 2019 overview)

Presenter Notes

krolik

Presenter Notes

netbsd

EuroBSDcon 2019

Author: Kamil Rytarowski

E-mail: kamil@netbsd.org

Speaker: Maciej Grochowski

Date: September 21st 2018

Place: Lillehammer, Norway

Presenter Notes

Topics

User space fuzzing

  • MKSANITIZER

Kernel space fuzzing

  • Kernel Sanitizers
  • Kernel Fuzzers
  • Miscellaneous

Presenter Notes

MKSANITIZER

Sanitizer - is an open source programming tool that detects program bugs. Sanitizer is based on compiler instrumentation and runtime library.

MKSANITIZER is a distinct feature of NetBSD that allows the whole distribution sanitization.

See: MeetBSDCa 2018 slides.

sanitizer

Presenter Notes

MKSANITIZER

Sanitizers:

  • Address Sanitizer (Asan) - Finds invalid address usage bugs.
  • Undefined Behavior Sanitizer (UBSan) - Finds unspecified code semantics bugs.
  • Thread Sanitizer (TSan) - Finds threading bugs.
  • Memory Sanitizer (MSan) - Finds uninitialized memory read.
  • Leak Sanitizer (LSan) - Finds resources leak [awaits upgrade to LLVM 9+]

Additionally:

  • libFuzzer - Library fuzzer
  • SafeStack - Stack hardening
  • XRay - Function call tracer

Presenter Notes

MKSANITIZER

MKSANITIZER merged into NetBSD-current (and NetBSD-9).

LLVM sanitizers pass as of now 95% of the upstream tests on NetBSD.

There are hundreds of issues left in programs, libraries and the sanitizer runtime before executing all ATF regression tests. There are likely some issues to be fixed on the kernel/userland level.

Presenter Notes

MKSANITIZER kernel code fuzzing

Prerequisites:

  • NetBSD-current/-9 with MKSANITIZER
  • honggfuzz
  • rumpkernel code

Presenter Notes

MKSANITIZER kernel code fuzzing

Step 1: Build the distribution with MKSANITIZER and libFuzzer code coverage.

1 ./build.sh \
2     -V MKCOMPAT=no \ # No 32-bit support in libFuzzer
3     -V MKDEBUGLIB=yes -V MKDEBUG=yes \ # Add debug symbols
4     -V MKSANITIZER=yes \ # Enable sanitization of the userland
5     -V USE_SANITIZER=fuzzer-no-link \ # libFuzzer code coverage
6     -V MKLLVM=yes -V MKGCC=no -V HAVE_LLVM=yes \ # Clang/LLVM distro
7     -O /public/netbsd.fuzzer \ # Example destination work dir
8     distribution

GCC MKSANITIZER is possible, but currently there was no work on it.

Presenter Notes

MKSANITIZER kernel code fuzzing

Step 2: Install devel/honggfuzz from pkgsrc.

honggfuzz is code coverage aided evolutionary fuzzer, with ptrace(2) based monitoring.

honggfuzz supports persistent fuzzing mode (without forks, execs).

Presenter Notes

MKSANITIZER kernel code fuzzing

Step 3: Mount useful directories into destdir

mount -t null /dev /public/netbsd.fuzzer/destdir.amd64/dev
mount -t null /dev/pts /public/netbsd.fuzzer/destdir.amd64/dev/pts
mount -t null /tmp /public/netbsd.fuzzer/destdir.amd64/tmp
mkdir -p /public/netbsd.fuzzer/destdir.amd64/usr/pkg
mount -t null /usr/pkg /public/netbsd.fuzzer/destdir.amd64/usr/pkg

Presenter Notes

MKSANITIZER kernel code fuzzing

Step 4: chroot into the new environment.

chroot /public/netbsd.fuzzer/destdir.amd64
cd /tmp

Presenter Notes

MKSANITIZER kernel code fuzzing

Step 5: Write rumpkernel code.

Example: Testing mount(2) + unmount(2) of FFS.

http://netbsd.org/~kamil/rump/rump_pub_etfs_register_buffer.c

Presenter Notes

MKSANITIZER kernel code fuzzing

Step 6: Start the fuzzer.

1 hfuzz-clang -lrumpvfs -lrump -lrumpfs_ffs -lrumpuser -lrumpdev_disk \
2     -lrumpdev -pthread rump_pub_etfs_register_buffer.c
3 mkdir corpus
4 newfs -F -s 10000 corpus/ffs.img
5 honggfuzz -P -f corpus/ -- ./a.out

Presenter Notes

MKSANITIZER kernel code fuzzing

Step 7: Observe crashes...

 1 ------------------------[  0 days 01 hrs 00 mins 00 secs ]-------
 2   Iterations : 367,977 [367.98k]
 3         Mode : [2/2] Feedback Driven Mode
 4       Target : ./a.out
 5      Threads : 4, CPUs: 8, CPU%: 0% [0%/CPU]
 6        Speed : 25/sec [avg: 102]
 7      Crashes : 7817 [unique: 7817, blacklist: 0, verified: 0]
 8     Timeouts : 13 [10 sec]
 9  Corpus Size : 203, max size: 5,120,000 bytes, init dir: 361 files
10   Cov Update : 0 days 00 hrs 00 mins 29 secs ago
11     Coverage : edge: 28 pc: 206 cmp: 104,650
12 ---------------------------------- [ LOGS ] ----/ honggfuzz 1.7 /-

Status after 60 minutes of fuzzing.

Presenter Notes

MKSANITIZER kernel code fuzzing

Step 8: Observe crashes...

 1 Persistent mode: Launched new persistent PID: 18581
 2 [   1.0400090] panic: kernel diagnostic assertion "newsize != VSIZ
 3 ENOTSET && newsize >= 0" failed: file "/usr/src/lib/librumpvfs/../
 4 ../sys/rump/../uvm/uvm_vnode.c", line 353 
 5 [   1.0400090] rump kernel halting...
 6 halted
 7 [   1.0300090] ufs_inactive: unlinked ino 2 on "/mnt" has non zero
 8  size 0 or blocks 817b2ac6 with allerror 0
 9 [   1.0300090] panic: ufs_inactive: dirty filesystem?
10 [   1.0300090] rump kernel halting...
11 halted
12 Crash: saved as './SIGABRT.PC.0.STACK.badbad183b661ac0.CODE.-5.ADD
13 R.0x0.INSTR.jb_0x7f7ff4d79acd.2019-08-31.01:57:47.18384.fuzz'
14 [2019-08-31T01:57:47+0200][W][5] arch_checkWait():230 Persistent m
15 ode: PID 18384 exited with status: SIGNALED, signal: 6 (Abort trap)
16 Persistent mode: Launched new persistent PID: 10281

Presenter Notes

MKSANITIZER kernel code fuzzing

Step 9: Log rumpkernel crashes into a file.

 1 [   1.0900090] panic: kernel diagnostic assertion "size + offset <
 2 = flen" failed: file "/usr/src/lib/librumpvfs/../../sys/rump/libru
 3 mp/rumpvfs/rumpblk.c", line 292
 4 [   1.0900090] rump kernel halting...
 5 halted
 6 [   1.4300090] /devdisk: file system not clean (fs_clean=0x4); ple
 7 ase fsck(8)
 8 [   1.4300090] /devdisk: lost blocks 0 files 0
 9 [   1.4500090] /devdisk: file system not clean (fs_clean=0x8); ple
10 ase fsck(8)
11 [   1.4500090] /devdisk: lost blocks 0 files 0
12 [   1.6300090] ufs_inactive: unlinked ino 2 on "/mnt" has non zero
13  size 0 or blocks 1b8b53de with allerror 0
14 [   1.6300090] panic: ufs_inactive: dirty filesystem?
15 [   1.6300090] rump kernel halting...
16 halted
17 [   1.4200090] panic: kernel diagnostic assertion "newsize != VSIZ
18 ENOTSET && newsize >= 0" failed: file "/usr/src/lib/librumpvfs/../
19 ../sys/rump/../uvm/uvm_vnode.c", line 353

Parse thousands of similar lines.

Presenter Notes

MKSANITIZER kernel code fuzzing

Step 10: Fix bugs, repeat.

There is enough work before need to make the fuzzing process quicker.

Presenter Notes

Kernel Sanitizers

NetBSD supports right now:

  • Kernel UBSan ([in theory] all ports)
  • Kernel ASan (amd64, evbarm64)
  • Kernel LEAK (amd64) - finds kernel memory leak to userland (to be obsoleted by kMSan)

Sanitizers still not merged into mainline:

  • Kernel MSan (amd64) - awaits upgrade of Clang/LLVM in base to a newer version (8.0+)
  • Kernel TSan (amd64) - catches the first kernel bugs, development in progress

Credit: Maxime Villard (maxv@)

Presenter Notes

Kernel Fuzers - Syzkaller

Syzkaller + syzbot 24/7 fuzzing with KCOV assisted coverage driver

syzbot

http://blog.netbsd.org/tnf/entry/enhancing_syzkaller_support_for_netbsd http://blog.netbsd.org/tnf/entry/enchancing_syzkaller_support_for_netbsd http://blog.netbsd.org/tnf/entry/enchancing_syzkaller_support_for_netbsd1

GSoC 2019 by Siddharth Muralee

Presenter Notes

Kernel Fuzers - TriforceAFL

triforceafl_2019_06_26.png

http://blog.netbsd.org/tnf/entry/adapting_triforceafl_for_netbsd_part http://blog.netbsd.org/tnf/entry/adapting_triforceafl_for_netbsd_part1 http://blog.netbsd.org/tnf/entry/adapting_triforceafl_for_netbsd_part2

GSoC 2019 by Akul Pillai

Presenter Notes

Miscellaneous

 1 Modified Files:
 2     src/etc: MAKEDEV.tmpl
 3     src/sys/arch/amd64/conf: GENERIC
 4     src/sys/conf: files majors
 5 Added Files:
 6     src/sys/dev/usb: vhci.c
 7 
 8 Log Message:
 9 Add vHCI, a driver which allows to send and receive USB packets
10 directly from userland via /dev/vhci. Using this, it becomes
11 possible to test and fuzz the USB stack and all the USB drivers
12 without having the associated hardware.
13 
14 The vHCI device has four ports independently addressable.
15 
16 For each xfer on each port, we create two packets: a setup packet
17 (which indicates mostly the type of request) and a data packet
18 (which contains the raw data). These packets are processed by read
19 and write operations on /dev/vhci: userland poll-reads it to fetch
20 usb_device_request_t structures, and dispatches the requests
21 depending on bRequest and bmRequestType.

vHCI added by Maxime Villard on 2019-09-14.

Presenter Notes

Miscellaneous

Hardware assisted virtualization is now available for NetBSD/amd64 through NVMM and HAXM.

A prerequisite for Qemu-assisted fuzzing (Syzkaller, ...).

Old school Xen for old school use-cases.

Presenter Notes

Epilogue

Further reading

  • The NetBSD Foundation blog: blog.netbsd.org

Action needed

  • Validate your kernel code with a sanitizer
  • Contribute to the project making sanitizers/MKSANITIZER support more complete

Future directions

  • Switch basesystem from LLVM 7 to LLVM 8+
  • Merge kMSan and kTSan into mainline kernel
  • TriforceAFL 24/7 fuzzing
  • MKSANITIZER on releng machines executing ATF tests
  • integrate fuzzing (rumpkernel, utilities, libraries) code into src/

Presenter Notes