Taking NetBSD kernel bug roast to the next level: Kernel Sanitizers

Presenter Notes

netbsd

EuroBSDcon 2018

Author: Kamil Rytarowski

E-mail: kamil@netbsd.org

Date: September 23th 2018

Place: Bucharest, Romania

Presenter Notes

Bio

Kamil Rytarowski (born 1987)

Krakow, Poland

NetBSD user since 6.1.

The NetBSD Foundation member since 2015.

Work areas: kernel, userland, pkgsrc.

Interest: NetBSD on desktop and in particular NetBSD as a workstation.

The current activity in 3rd party software:

  • LLVM committer.
  • GDB & binutils committer.
  • NetBSD maintainer in qemu.

Presenter Notes

Topics

  • What are sanitizers?
  • Types of kernel sanitizers
  • Sanitizers vs other tools
  • Kernel Undefined Behavior Sanitizer
  • Kernel Address Sanitizer

Presenter Notes

Types of sanitizers

Sanitizer is a programming tool that detects computer program bugs. There are 4 fundamental sanitizers:

  • Address Sanitizer (Asan) - Finds invalid address usage bugs.
  • Undefined Behavior Sanitizer (UBSan) - Finds unspecified code semantics bugs.
  • Thread Sanitizer (TSan) - Finds threading bugs.
  • Memory Sanitizer (MSan) - Finds uninitialized memory read.

sanitizer

Presenter Notes

Types of kernel sanitizers

Sanitizers supported in NetBSD:

  • Kernel Address Sanitizer - kASan
  • Kernel Undefined Behavior Sanitizer - kUBSan

Work-in-progress kernel sanitizers researched by Linux kernel developers:

  • Kernel Thread Sanitizer (KTSan).
  • Kernel Memory Sanitizer (KMSan).

Presenter Notes

Sanitizers vs other tools

Kernel Sanitizers (kUBSan and kASan):

  • Compile-time instrumentation used as it is (however we use kASan with tuned Address Sanitizer defaults for kernels)
  • Slowdown 2x (usually smaller)
  • Finds bugs by triggering them, false positives are usually a bug in a sanitizer runtime
  • Detects: out-of-bounds, use-after-free, use-after-return, undefined-behavior

Presenter Notes

Sanitizers vs other tools

Static analyzers (builtin in compilers, Coverity, cppcheck and others)

  • Static source code analysis
  • No impact of the kernel execution
  • Finds potential bugs, practically impossible to eliminate false positives (false alarm) and false negatives (missed bugs)
  • Detects: all potential programming errors

Presenter Notes

Sanitizers vs other tools

Dynamic Binary analysis tools (Valgrind)

  • Not available in the kernel
  • A restricted fallback available to run it in usermode (NetBSD/usermode) or as a standalone library (rumpkernel)

Presenter Notes

Sanitizers vs other tools

Available the NetBSD kernel diagnostics:

  • DIAGNOSTIC - inexpensive kernel consistency checks
  • DEBUG - expensive debugging checks/support
  • LOCKDEBUG - expensive locking checks/support
  • KMEM_POISON - detects modify-after-free (removed after introduction of kASan)
  • KMEM_GUARD - very expensive; detects overflows, invalid pointer/size passed at free, underflow at free, use-after-free
  • KMEM_REDZONE - detects overrun bugs (removed after introduction of kASan)

They are usually expensive and detect logical kernel bugs in certain subsystems or routines only.

Presenter Notes

Kernel Undefined Behavior Sanitizer

Presenter Notes

Undefined Behavior Sanitizer

Detects:

  • Signed integer overflow (optionally unsigned integer overflow as an extension)
  • Float cast overflow (userland only)
  • Load invalid value for type
  • Invalid builtin usage (ctz, clz..)
  • Function type mismatch and dynamic type mismatch (C++ specific)
  • Missing return
  • Non-null/Nullability argument/attribute violation (compiler extension)
  • Out-of-bounds usage
  • Pointer overflow
  • Invalid shift
  • VLA bound not positive
  • Control-Flow-Integrity violation (LLVM CFI integration)

Presenter Notes

Kernel Undefined Behavior Sanitizer

µUBSan - independent NetBSD runtime:

  • Clean room independent and self-contained implementation (1300 LOC)
  • Implemented within a single C file (ubsan.c) with minimal dependencies (mostly printing, support for variable argument lists and optional aborting the execution)
  • The same runtime is reused in MKLIBCSANITIZER and linked into libc
  • Used as a standalone library The runtime (and compiler instrumentation) is verified with regression ATF tests (checking both C and C++)

Presenter Notes

Kernel Undefined Behavior Sanitizer

µUBSan - independent NetBSD runtime:

  • Designed to be portable to any reasonable 32-bit and 64-bit CPU (restrictions are mostly due to handling of floating point numbers)
  • No Undefined Behavior triggered in the runtime (contrary to alternatives), this implies self-sanitizing
  • With a minimal shim known ports to FreeBSD (arm, aarch64) and XNU kernels (x86?)
  • No TODO lists, considered as feature-complete

Presenter Notes

Detected and fixed kernel bugs

  • sys/sys/wait.h sys/external/bsd/drm2/dist/drm/i915/i915_reg.h sys/netinet6/in6.c sys/kern/kern_descrip.c sys/kern/kern_lwp.c sys/kern/sys_mqueue.c sys/dev/scsipi/scsipiconf.h sys/kern/subr_pool.c sys/ufs/ffs/ffs_subr.c sys/sys/mman.h sys/dev/pci/pciide_piix_reg.h sys/arch/x86/x86/intr.c sys/kern/kern_descrip.c common/lib/libutil/snprintb.c common/lib/libc/inet/inet_addr.c common/lib/libc/sys/cpuset.c sys/fs/msdosfs/msdosfs_fat.c sys/fs/udf/ecma167-udf.h ...

... and more

Mostly:

  • unportable bit shift (mostly harmless in modern CPUs)
  • unaligned memory access (reports in ACPICA, IP stack, MD specific code; RISC CPUs are sensitive to this)
  • signed integer overflow (usually means either bad design or real bugs)

Presenter Notes

Kernel Address Sanitizer

Presenter Notes

Kernel Address Sanitizer

Primary author of the port: Maxime Villard (maxv@NetBSD.org).

Initial porting by Siddharth Muralee (during Google Summer of Code).

Quick overview by myself.

Presenter Notes

Kernel Address Sanitizer

Detects unauthorized memory access (unallocated or already freed) - use-after-free, out-of-bound access

The NetBSD port functional with ASan ABI v6 (GCC 6.x) and v8 (GCC 7.x, Clang/LLVM 6.x).

Presenter Notes

Kernel Address Sanitizer

Requirements:

  • Shadow buffer allocation of at least 1/8 of the size of the kernel heap
  • Machine Dependent initialization on early boot and before any instrumented kernel heap access
  • ELF constructor to register global variables (data section)
  • Interceptors for utility routines, especially for compiler builtins (memset(), memcpy(), memcmp(), strcpy(), strcmp(), strlen()...).

Presenter Notes

Kernel Address Sanitizer

kASan supported on the following ports:

  • NetBSD/amd64
  • NetBSD/aarch64 (to be merged with mainline)

Nice to have for bug detecting purposes:

  • a port to a performant 32-bit CPU emulated with a hardware assisted virtualization (NetBSD/i386 is a good candidate)

Presenter Notes

Detected bugs #1

sys/net/rtsock.c

 1 Fix buffer overflow, detected by kASan.
 2 
 3     ifconfig gif0 create
 4     ifconfig gif0 up
 5 
 6 [   50.682919] kASan: Unauthorized Access In 0xffffffff80f22655: \
 7 Addr 0xffffffff81b997a0 [8 bytes, read]
 8 [   50.682919] #0 0xffffffff8021ce6a in kasan_memcpy <netbsd>
 9 [   50.692999] #1 0xffffffff80f22655 in m_copyback_internal <netbsd>
10 [   50.692999] #2 0xffffffff80f22e81 in m_copyback <netbsd>
11 [   50.692999] #3 0xffffffff8103109a in rt_msg1 <netbsd>
12 [   50.692999] #4 0xffffffff8159109a in compat_70_rt_newaddrmsg1 <n
13 [   50.692999] #5 0xffffffff81031b0f in rt_newaddrmsg <netbsd>
14 [   50.692999] #6 0xffffffff8102c35e in rt_ifa_addlocal <netbsd>
15 [   50.692999] #7 0xffffffff80a5287c in in6_update_ifa1 <netbsd>
16 [   50.692999] #8 0xffffffff80a54149 in in6_update_ifa <netbsd>
17 [   50.692999] #9 0xffffffff80a59176 in in6_ifattach <netbsd>
18 [   50.692999] #10 0xffffffff80a56dd4 in in6_if_up <netbsd>
19 [   50.692999] #11 0xffffffff80fc5cb8 in if_up_locked <netbsd>
20 [   50.703622] #12 0xffffffff80fcc4c1 in ifioctl_common <netbsd>
21 [   50.703622] #13 0xffffffff80fde694 in gif_ioctl <netbsd>
22 [   50.703622] #14 0xffffffff80fcdb1f in doifioctl <netbsd>

Presenter Notes

Detected bugs #2

sys/dev/pci/if_msk.c

 1 Fix buffer overflow, detected by kASan.
 2 
 3 [    1.044878] kASan: Unauthorized Access In 0xffffffff804ec7e2: \
 4 Addr 0xffffffff818a51e4 [2 bytes, read]
 5 [    1.044878] #0 0xffffffff804ec7e2 in mskc_probe <netbsd>
 6 [    1.044878] #1 0xffffffff80e92a77 in mapply <netbsd>
 7 [    1.044878] #2 0xffffffff80e92e5f in config_search_loc <netbsd>
 8 [    1.044878] #3 0xffffffff80e93fb5 in config_found_sm_loc <netbsd>
 9 [    1.044878] #4 0xffffffff802ca9ea in pci_probe_device <netbsd>
10 [    1.044878] #5 0xffffffff802cad97 in pci_enumerate_bus <netbsd>
11 [    1.044878] #6 0xffffffff802caf00 in pcirescan <netbsd>
12 [    1.044878] #7 0xffffffff802cb1ee in pciattach <netbsd>
13 [    1.044878] #8 0xffffffff80e93e5b in config_attach_loc <netbsd>
14 [    1.044878] #9 0xffffffff80e93fce in config_found_sm_loc <netbsd>
15 [    1.044878] #10 0xffffffff80271212 in mp_pci_scan <netbsd>
16 [    1.044878] #11 0xffffffff8022d9ee in mainbus_attach <netbsd>
17 [    1.044878] #12 0xffffffff80e93e5b in config_attach_loc <netbsd>
18 [    1.044878] #13 0xffffffff8021e38b in cpu_configure <netbsd>
19 [    1.044878] #14 0xffffffff814a7068 in main <netbsd>

Presenter Notes

Epilogue

Further reading

Action needed

Future directions

  • kcov(4) - fuzzer-oriented coverage integration with the kernel
  • NetBSD VMM - hardware assisted virtualization
  • syzkaller - multithreaded coverage-guided kernel fuzzer
  • rumpkernel sanitizing and fuzzing - research and innovations with userland tools

Presenter Notes