NetBSD kernel sanitizers

Presenter Notes

netbsd

MeetBSDCa 2018

Author: Kamil Rytarowski

E-mail: kamil@netbsd.org

Date: October 20th 2018

Place: Intel Santa Clara Campus, California, USA

Presenter Notes

Bio

Kamil Rytarowski (born 1987)

Krakow, Poland

NetBSD user since 6.1.

The NetBSD Foundation member since 2015.

Work areas: kernel, userland, pkgsrc.

Interest: NetBSD on desktop and in particular NetBSD as a workstation.

The current activity in 3rd party software:

  • LLVM committer.
  • GDB & binutils committer.
  • NetBSD maintainer in qemu.

Presenter Notes

Topics

  • Example of Address Sanitizer
  • µUBSan
  • Kernel Undefined Behavior Sanitizer
  • Kernel Address Sanitizer

Presenter Notes

Example of Address Sanitizer

1 int main(int argc, char **argv) {
2     int buf[10];
3     buf[10+argc] = 1; // buffer overflow (argc == 1)
4     return 0;
5 }

Presenter Notes

Example of Address Sanitizer

 1 0000000000400c22 <main>:
 2   ......:
 3   400c84: 0f 9d c1             setge  %cl
 4   400c87: 44 21 c1             and    %r8d,%ecx
 5   400c8a: 84 c9                test   %cl,%cl
 6   400c8c: 74 08                je     400c96 <main+0xd4>
 7   400c8e: 48 89 d7             mov    %rdx,%rdi
 8   400c91: e8 7a fb ff ff       callq  400810 <__asan_report_store4@plt>
 9   400c96: 48 63 d7             movslq %edi,%rdx
10   400c99: c7 44 96 a0 01 00 00 movl   $0x1,-0x60(%rsi,%rdx,4)
11   400ca0: 00 
12   400ca1: ba 00 00 00 00       mov    $0x0,%edx
13   400ca6: 49 39 dc             cmp    %rbx,%r12
14   400ca9: 74 2b                je     400cd6 <main+0x114>
15   ......:

Presenter Notes

Example of Address Sanitizer

 1 $ ./a.out
 2 =================================================================
 3 ==2622==ERROR: AddressSanitizer: stack-buffer-overflow on address \
 4     0x7f7fff346b4c at pc 0x000000400c96 bp 0x7f7fff346ae0 \
 5     sp 0x7f7fff346ad8
 6 WRITE of size 4 at 0x7f7fff346b4c thread T0
 7     #0 0x400c95 in main (/tmp/./a.out+0x400c95)
 8     #1 0x400af2 in ___start (/tmp/./a.out+0x400af2)
 9 
10 Address 0x7f7fff346b4c is located in stack of thread T0 at \
11     offset 76 in frame
12     #0 0x400bd1 in main (/tmp/./a.out+0x400bd1)
13 
14   This frame has 1 object(s):
15     [32, 72) 'buf' <== Memory access at offset 76 overflows this \
16     variable
17 ....

Presenter Notes

µUBSan

µUBSan - independent NetBSD runtime:

  • Clean room independent and self-contained implementation (1300 LOC)
  • Implemented within a single C file (ubsan.c) with minimal dependencies (mostly printing, support for variable argument lists and optional aborting the execution)
  • The same runtime is reused in userland and kernel
  • Used as a standalone library, the runtime (and compiler instrumentation) is verified with the ATF regression tests (checking both C and C++)

Presenter Notes

µUBSan

  • Designed to be portable to any reasonable 32-bit and 64-bit CPU (restrictions are mostly due to handling of floating point numbers)
  • No Undefined Behavior triggered in the runtime (contrary to alternatives), this implies self-sanitizing
  • With a minimal shim known ports to FreeBSD (arm, aarch64) and XNU kernels (x86?)
  • No TODO lists, considered as feature-complete - just track upstream for new reports (like signed/unsigned integer truncation)

Presenter Notes

Sanitizers in the NetBSD kernel

Presenter Notes

Kernel sanitizers vs other tools

Available the NetBSD kernel diagnostics:

  • DIAGNOSTIC - inexpensive kernel consistency checks
  • DEBUG - expensive debugging checks/support
  • LOCKDEBUG - expensive locking checks/support
  • KMEM_POISON - detects modify-after-free (removed after introduction of kASan)
  • KMEM_GUARD - very expensive; detects overflows, invalid pointer/size passed at free, underflow at free, use-after-free
  • KMEM_REDZONE - detects overrun bugs (removed after introduction of kASan)

They are usually expensive and detect logical kernel bugs in certain subsystems or routines only.

Presenter Notes

Kernel Undefined Behavior Sanitizer

Shares runtime with the userland (µUBSan).

Presenter Notes

Kernel Undefined Behavior Sanitizer

Detected and fixed kernel bugs

  • sys/sys/wait.h sys/external/bsd/drm2/dist/drm/i915/i915_reg.h sys/netinet6/in6.c sys/kern/kern_descrip.c sys/kern/kern_lwp.c sys/kern/sys_mqueue.c sys/dev/scsipi/scsipiconf.h sys/kern/subr_pool.c sys/ufs/ffs/ffs_subr.c sys/sys/mman.h sys/dev/pci/pciide_piix_reg.h sys/arch/x86/x86/intr.c sys/kern/kern_descrip.c common/lib/libutil/snprintb.c common/lib/libc/inet/inet_addr.c common/lib/libc/sys/cpuset.c sys/fs/msdosfs/msdosfs_fat.c sys/fs/udf/ecma167-udf.h ...

... and more

Mostly:

  • unportable bit shift (mostly harmless in modern CPUs)
  • unaligned memory access (reports in ACPICA, IP stack, MD specific code; RISC CPUs are sensitive to this)
  • signed integer overflow (usually means either bad design or real bugs)

Presenter Notes

Kernel Address Sanitizer

Primary author of the port: Maxime Villard (maxv@NetBSD.org).

Initial porting by Siddharth Muralee (during Google Summer of Code).

Presenter Notes

Kernel Address Sanitizer

Detects unauthorized memory access (unallocated or already freed) - use-after-free, out-of-bound access, etc.

The NetBSD port functional with ASan ABI v6 (GCC 6.x) and v8 (GCC 7.x, Clang/LLVM 6.x).

Presenter Notes

Kernel Address Sanitizer

kASan supported on the following ports:

  • NetBSD/amd64
  • NetBSD/aarch64 (to be merged with mainline)

Nice to have for bug detecting purposes:

  • a port to a performant 32-bit CPU emulated with a hardware assisted virtualization (NetBSD/i386 is a good candidate)

Presenter Notes

Detected bugs #1

sys/net/rtsock.c

 1 Fix buffer overflow, detected by kASan.
 2 
 3     ifconfig gif0 create
 4     ifconfig gif0 up
 5 
 6 [   50.682919] kASan: Unauthorized Access In 0xffffffff80f22655: \
 7 Addr 0xffffffff81b997a0 [8 bytes, read]
 8 [   50.682919] #0 0xffffffff8021ce6a in kasan_memcpy <netbsd>
 9 [   50.692999] #1 0xffffffff80f22655 in m_copyback_internal <netbsd>
10 [   50.692999] #2 0xffffffff80f22e81 in m_copyback <netbsd>
11 [   50.692999] #3 0xffffffff8103109a in rt_msg1 <netbsd>
12 [   50.692999] #4 0xffffffff8159109a in compat_70_rt_newaddrmsg1 <n
13 [   50.692999] #5 0xffffffff81031b0f in rt_newaddrmsg <netbsd>
14 [   50.692999] #6 0xffffffff8102c35e in rt_ifa_addlocal <netbsd>
15 [   50.692999] #7 0xffffffff80a5287c in in6_update_ifa1 <netbsd>
16 [   50.692999] #8 0xffffffff80a54149 in in6_update_ifa <netbsd>
17 [   50.692999] #9 0xffffffff80a59176 in in6_ifattach <netbsd>
18 [   50.692999] #10 0xffffffff80a56dd4 in in6_if_up <netbsd>
19 [   50.692999] #11 0xffffffff80fc5cb8 in if_up_locked <netbsd>
20 [   50.703622] #12 0xffffffff80fcc4c1 in ifioctl_common <netbsd>
21 [   50.703622] #13 0xffffffff80fde694 in gif_ioctl <netbsd>
22 [   50.703622] #14 0xffffffff80fcdb1f in doifioctl <netbsd>

Presenter Notes

Detected bugs #2

sys/dev/pci/if_msk.c

 1 Fix buffer overflow, detected by kASan.
 2 
 3 [    1.044878] kASan: Unauthorized Access In 0xffffffff804ec7e2: \
 4 Addr 0xffffffff818a51e4 [2 bytes, read]
 5 [    1.044878] #0 0xffffffff804ec7e2 in mskc_probe <netbsd>
 6 [    1.044878] #1 0xffffffff80e92a77 in mapply <netbsd>
 7 [    1.044878] #2 0xffffffff80e92e5f in config_search_loc <netbsd>
 8 [    1.044878] #3 0xffffffff80e93fb5 in config_found_sm_loc <netbsd>
 9 [    1.044878] #4 0xffffffff802ca9ea in pci_probe_device <netbsd>
10 [    1.044878] #5 0xffffffff802cad97 in pci_enumerate_bus <netbsd>
11 [    1.044878] #6 0xffffffff802caf00 in pcirescan <netbsd>
12 [    1.044878] #7 0xffffffff802cb1ee in pciattach <netbsd>
13 [    1.044878] #8 0xffffffff80e93e5b in config_attach_loc <netbsd>
14 [    1.044878] #9 0xffffffff80e93fce in config_found_sm_loc <netbsd>
15 [    1.044878] #10 0xffffffff80271212 in mp_pci_scan <netbsd>
16 [    1.044878] #11 0xffffffff8022d9ee in mainbus_attach <netbsd>
17 [    1.044878] #12 0xffffffff80e93e5b in config_attach_loc <netbsd>
18 [    1.044878] #13 0xffffffff8021e38b in cpu_configure <netbsd>
19 [    1.044878] #14 0xffffffff814a7068 in main <netbsd>

Presenter Notes

Sanitizers on NetBSD

Further reading

Action needed

Future directions

  • kcov(4) and syzkaller - multithreaded coverage-guided kernel fuzzer
  • rumpkernel sanitizing and fuzzing - research and innovations

Presenter Notes