NetBSD and userland & kernel SANITIZERS

Presenter Notes

netbsd

Google Summer of Code 2018 Mentor Summit

Author: Kamil Rytarowski

E-mail: kamil@netbsd.org

Date: October 13rd 2018

Place: Googleplex, Mountain View, California, USA

Presenter Notes

Bio

Kamil Rytarowski (born 1987)

Krakow, Poland

NetBSD user since 6.1.

The NetBSD Foundation member (== developer) since 2015.

Work areas: kernel, userland, pkgsrc.

Interest: NetBSD on desktop and in particular NetBSD as a workstation.

The current activity in 3rd party software:

  • LLVM committer.
  • GDB & binutils committer.
  • NetBSD maintainer in qemu.

The first time mentor during GSoC 2018.

Presenter Notes

Topics

  • Lightning introduction to NetBSD
  • NetBSD and Google Summer of Code
  • Sanitizers
  • Userland sanitizers
  • Kernel sanitizers

Presenter Notes

Lightning introduction to NetBSD

Presenter Notes

Lightning introduction to NetBSD

netbsd

NetBSD is a free, fast, secure, and highly portable Unix-like Open Source operating system. It is available for a wide range of platforms, from large-scale servers and powerful desktop systems to handheld and embedded devices.

Presenter Notes

Lightning introduction to NetBSD

PowerPC, Alpha, SPARC, MIPS, SH3, ARM, amd64, i386, m68k, VAX, ...

Of course it runs NetBSD.

Presenter Notes

Lightning introduction to NetBSD

Cross-building is possible from most UNIX-like operating systems.

1 ./build.sh \
2     distribution

Additional build information available in the BUILDING file.

Presenter Notes

Lightning introduction to NetBSD

Binaries.

Presenter Notes

Lightning introduction to NetBSD

Testing your NetBSD system with Automated Test Framework (ATF).

1 cd /usr/tests; atf-run | atf-report

Presenter Notes

Lightning introduction to NetBSD

Community support.

Presenter Notes

Lightning introduction to NetBSD

Fetch the latest sources.

To fetch the main CVS repository:

1 cvs -d anoncvs@anoncvs.NetBSD.org:/cvsroot checkout -P src

GitHub mirror:

1 git clone https://github.com/netbsd/src

Alternatively use snapshots, Mercurial or Fossil mirrors.

Presenter Notes

Lightning introduction to NetBSD

For additional introduction check The NetBSD Guide.

https://www.netbsd.org/docs/guide/en/

Presenter Notes

NetBSD and Google Summer of Code

Presenter Notes

NetBSD and Google Summer of Code

NetBSD participated successfully in the following Google's Summer of Code programs: 2005-2013 and 2016-2018.

Presenter Notes

NetBSD and Google Summer of Code

In 2018 there were 9 project slots for The NetBSD Foundation.

Together with Christos Zoulas, I've mentored 3 students this year.

Two of them finished the coding periods successfully.

  • Yang Zheng (China) "Integrate libFuzzer with the basesystem"
  • Siddharth Muralee (India) "Kernel Address Sanitizer"

Check http://blog.netbsd.org/ for their reports.

Presenter Notes

NetBSD and Google Summer of Code

Other successful projects mentored by the NetBSD developers.

  • Keivan Motavalli "Configuration files versioning in pkgsrc"
  • Nizar Benshaqi "SQL Database for ATF tests results with online query and statistics page"

Presenter Notes

Sanitizers

Presenter Notes

What are sanitizers

Sanitizer is a programming tool that detects computer program bugs such as:

  • buffer overflows,
  • signed integer overflow,
  • uninitialized memory read,
  • data races etc.

sanitizer

Presenter Notes

Types of sanitizers

The fundamental four types of sanitizers:

  • Address Sanitizer (Asan) - Finds invalid address usage bugs.
  • Undefined Behavior Sanitizer (UBSan) - Finds unspecified code semantics bugs.
  • Thread Sanitizer (TSan) - Finds threading bugs.
  • Memory Sanitizer (MSan) - Finds uninitialized memory read.

Presenter Notes

Types of sanitizers in NetBSD

All of them main four userland sanitizers are supported on NetBSD.

  • ASan (amd64, i386)
  • UBSan (all ports)
  • MSan (amd64)
  • TSan (amd64)

There are two sanitizers available in the NetBSD kernel.

  • kASan (amd64, aarch64)
  • kUBSan (all ports)

Presenter Notes

Characteristics of sanitizers

  • Checks are performed dynamically in runtime.
  • Compiler (Clang, GCC) emits checks inlined into the generated code.
  • Runtime handles non-trivial validation and reporting of bugs.

Presenter Notes

Sanitizers vs Valgrind

Sanitizers:

  • Compile-time instrumentation
  • Slowdown 2x
  • Decent portability
  • Detects: out-of-bounds heap, out-of-bounds stack, out-of-bounds globals, use-after-free, use-after-return, uninitialized-memory-read, leaks, undefined-behavior, data races

Valgrind:

  • Dynamic-binary instrumentation
  • Slowdown 20x
  • Difficult porting to new platforms and OSes
  • Detects: out-of-bounds heap, use-after-free, uninitialized-memory-read, leaks, data races

Presenter Notes

Example of Address Sanitizer

1 int main(int argc, char **argv) {
2     int buf[10];
3     buf[10+argc] = 1; // buffer overflow (argc == 1)
4     return 0;
5 }

Presenter Notes

Example of Address Sanitizer

 1 0000000000400c22 <main>:
 2   ......:
 3   400c84: 0f 9d c1             setge  %cl
 4   400c87: 44 21 c1             and    %r8d,%ecx
 5   400c8a: 84 c9                test   %cl,%cl
 6   400c8c: 74 08                je     400c96 <main+0xd4>
 7   400c8e: 48 89 d7             mov    %rdx,%rdi
 8   400c91: e8 7a fb ff ff       callq  400810 <__asan_report_store4@plt>
 9   400c96: 48 63 d7             movslq %edi,%rdx
10   400c99: c7 44 96 a0 01 00 00 movl   $0x1,-0x60(%rsi,%rdx,4)
11   400ca0: 00 
12   400ca1: ba 00 00 00 00       mov    $0x0,%edx
13   400ca6: 49 39 dc             cmp    %rbx,%r12
14   400ca9: 74 2b                je     400cd6 <main+0x114>
15   ......:

Presenter Notes

Example of Address Sanitizer

 1 $ ./a.out
 2 =================================================================
 3 ==2622==ERROR: AddressSanitizer: stack-buffer-overflow on address \
 4     0x7f7fff346b4c at pc 0x000000400c96 bp 0x7f7fff346ae0 \
 5     sp 0x7f7fff346ad8
 6 WRITE of size 4 at 0x7f7fff346b4c thread T0
 7     #0 0x400c95 in main (/tmp/./a.out+0x400c95)
 8     #1 0x400af2 in ___start (/tmp/./a.out+0x400af2)
 9 
10 Address 0x7f7fff346b4c is located in stack of thread T0 at \
11     offset 76 in frame
12     #0 0x400bd1 in main (/tmp/./a.out+0x400bd1)
13 
14   This frame has 1 object(s):
15     [32, 72) 'buf' <== Memory access at offset 76 overflows this \
16     variable
17 ....

Presenter Notes

Sanitizers in the NetBSD userland

Presenter Notes

Sanitizers in the NetBSD userland

The base distribution (HEAD version).

  • GCC-style distribution: ASan, UBSan, LSan (scratch)
  • LLVM-style distribution: shipping with the distribution coming soon

Externally prebuilt standalone toolchain.

Presenter Notes

MKSANITIZER

Build and use almost all of the userland with a selected sanitizer.

1 ./build.sh \
2     -V MKLLVM=yes \
3     -V MKGCC=no \
4     -V HAVE_LLVM=yes \
5     -V MKSANITIZER=yes \
6     -V USE_SANITIZER="address,undefined" \
7     distribution

Unsanitized exceptions: kernel, loadable kernel modules, ramdisks, static libraries, static programs, base libraries (libc, libm, libpthread, librt).

Functional chroot environment: ASan, UBSan, MSan.

Bootable distribution into functional shell: ASan, UBSan.

As of now requires external and patched Clang/LLVM toolchain.

Presenter Notes

µUBSan

µUBSan - independent NetBSD runtime:

  • Clean room independent and self-contained implementation (1300 LOC)
  • Implemented within a single C file (ubsan.c) with minimal dependencies (mostly printing, support for variable argument lists and optional aborting the execution)
  • The same runtime is reused in userland and kernel
  • Used as a standalone library, the runtime (and compiler instrumentation) is verified with the ATF regression tests (checking both C and C++)

Presenter Notes

µUBSan

  • Designed to be portable to any reasonable 32-bit and 64-bit CPU (restrictions are mostly due to handling of floating point numbers)
  • No Undefined Behavior triggered in the runtime (contrary to alternatives), this implies self-sanitizing
  • With a minimal shim known ports to FreeBSD (arm, aarch64) and XNU kernels (x86?)
  • No TODO lists, considered as feature-complete - just track upstream for new reports (like signed/unsigned integer truncation)

Presenter Notes

MKLIBCSANITIZER

Build and use almost all of the userland with a selected sanitizer linked with libc.

1 ./build.sh \
2     -V MKLIBCSANITIZER=yes \
3     distribution

Unsanitized exceptions: ramdisks, static libraries, static programs.

Restricted to UBSan with a specialized homegrown runtime (µUBSan).

Presenter Notes

A selection of fixed programs

ASan: sh(1), sysinst(8), heimdal krb5, libutil(3), man(1), installboot(8), passwd(8), ...

UBSan: tmux(1), expr(1), ksh(1), ifconfig(8), libc, [gnu]grep(1), gzip(1), [n]awk(1), [n]vi(1), disklabel(8), ...

MSan: sh(1), top(1), ...

... and others that were forgotten to mention.

Presenter Notes

Sanitizers in the NetBSD kernel

Presenter Notes

Sanitizers vs other tools

Available the NetBSD kernel diagnostics:

  • DIAGNOSTIC - inexpensive kernel consistency checks
  • DEBUG - expensive debugging checks/support
  • LOCKDEBUG - expensive locking checks/support
  • KMEM_POISON - detects modify-after-free (removed after introduction of kASan)
  • KMEM_GUARD - very expensive; detects overflows, invalid pointer/size passed at free, underflow at free, use-after-free
  • KMEM_REDZONE - detects overrun bugs (removed after introduction of kASan)

They are usually expensive and detect logical kernel bugs in certain subsystems or routines only.

Presenter Notes

Kernel Undefined Behavior Sanitizer

Shares runtime with the userland (µUBSan).

Presenter Notes

Kernel Undefined Behavior Sanitizer

Detected and fixed kernel bugs

  • sys/sys/wait.h sys/external/bsd/drm2/dist/drm/i915/i915_reg.h sys/netinet6/in6.c sys/kern/kern_descrip.c sys/kern/kern_lwp.c sys/kern/sys_mqueue.c sys/dev/scsipi/scsipiconf.h sys/kern/subr_pool.c sys/ufs/ffs/ffs_subr.c sys/sys/mman.h sys/dev/pci/pciide_piix_reg.h sys/arch/x86/x86/intr.c sys/kern/kern_descrip.c common/lib/libutil/snprintb.c common/lib/libc/inet/inet_addr.c common/lib/libc/sys/cpuset.c sys/fs/msdosfs/msdosfs_fat.c sys/fs/udf/ecma167-udf.h ...

... and more

Mostly:

  • unportable bit shift (mostly harmless in modern CPUs)
  • unaligned memory access (reports in ACPICA, IP stack, MD specific code; RISC CPUs are sensitive to this)
  • signed integer overflow (usually means either bad design or real bugs)

Presenter Notes

Kernel Address Sanitizer

Primary author of the port: Maxime Villard (maxv@NetBSD.org).

Initial porting by Siddharth Muralee (during Google Summer of Code).

Presenter Notes

Kernel Address Sanitizer

Detects unauthorized memory access (unallocated or already freed) - use-after-free, out-of-bound access, etc.

The NetBSD port functional with ASan ABI v6 (GCC 6.x) and v8 (GCC 7.x, Clang/LLVM 6.x).

Presenter Notes

Kernel Address Sanitizer

kASan supported on the following ports:

  • NetBSD/amd64
  • NetBSD/aarch64 (to be merged with mainline)

Nice to have for bug detecting purposes:

  • a port to a performant 32-bit CPU emulated with a hardware assisted virtualization (NetBSD/i386 is a good candidate)

Presenter Notes

Detected bugs #1

sys/net/rtsock.c

 1 Fix buffer overflow, detected by kASan.
 2 
 3     ifconfig gif0 create
 4     ifconfig gif0 up
 5 
 6 [   50.682919] kASan: Unauthorized Access In 0xffffffff80f22655: \
 7 Addr 0xffffffff81b997a0 [8 bytes, read]
 8 [   50.682919] #0 0xffffffff8021ce6a in kasan_memcpy <netbsd>
 9 [   50.692999] #1 0xffffffff80f22655 in m_copyback_internal <netbsd>
10 [   50.692999] #2 0xffffffff80f22e81 in m_copyback <netbsd>
11 [   50.692999] #3 0xffffffff8103109a in rt_msg1 <netbsd>
12 [   50.692999] #4 0xffffffff8159109a in compat_70_rt_newaddrmsg1 <n
13 [   50.692999] #5 0xffffffff81031b0f in rt_newaddrmsg <netbsd>
14 [   50.692999] #6 0xffffffff8102c35e in rt_ifa_addlocal <netbsd>
15 [   50.692999] #7 0xffffffff80a5287c in in6_update_ifa1 <netbsd>
16 [   50.692999] #8 0xffffffff80a54149 in in6_update_ifa <netbsd>
17 [   50.692999] #9 0xffffffff80a59176 in in6_ifattach <netbsd>
18 [   50.692999] #10 0xffffffff80a56dd4 in in6_if_up <netbsd>
19 [   50.692999] #11 0xffffffff80fc5cb8 in if_up_locked <netbsd>
20 [   50.703622] #12 0xffffffff80fcc4c1 in ifioctl_common <netbsd>
21 [   50.703622] #13 0xffffffff80fde694 in gif_ioctl <netbsd>
22 [   50.703622] #14 0xffffffff80fcdb1f in doifioctl <netbsd>

Presenter Notes

Detected bugs #2

sys/dev/pci/if_msk.c

 1 Fix buffer overflow, detected by kASan.
 2 
 3 [    1.044878] kASan: Unauthorized Access In 0xffffffff804ec7e2: \
 4 Addr 0xffffffff818a51e4 [2 bytes, read]
 5 [    1.044878] #0 0xffffffff804ec7e2 in mskc_probe <netbsd>
 6 [    1.044878] #1 0xffffffff80e92a77 in mapply <netbsd>
 7 [    1.044878] #2 0xffffffff80e92e5f in config_search_loc <netbsd>
 8 [    1.044878] #3 0xffffffff80e93fb5 in config_found_sm_loc <netbsd>
 9 [    1.044878] #4 0xffffffff802ca9ea in pci_probe_device <netbsd>
10 [    1.044878] #5 0xffffffff802cad97 in pci_enumerate_bus <netbsd>
11 [    1.044878] #6 0xffffffff802caf00 in pcirescan <netbsd>
12 [    1.044878] #7 0xffffffff802cb1ee in pciattach <netbsd>
13 [    1.044878] #8 0xffffffff80e93e5b in config_attach_loc <netbsd>
14 [    1.044878] #9 0xffffffff80e93fce in config_found_sm_loc <netbsd>
15 [    1.044878] #10 0xffffffff80271212 in mp_pci_scan <netbsd>
16 [    1.044878] #11 0xffffffff8022d9ee in mainbus_attach <netbsd>
17 [    1.044878] #12 0xffffffff80e93e5b in config_attach_loc <netbsd>
18 [    1.044878] #13 0xffffffff8021e38b in cpu_configure <netbsd>
19 [    1.044878] #14 0xffffffff814a7068 in main <netbsd>

Presenter Notes

Sanitizers on NetBSD

Further reading

Action needed

Future directions

  • kcov(4) and syzkaller - multithreaded coverage-guided kernel fuzzer
  • rumpkernel sanitizing and fuzzing - research and innovations

permalink: http://netbsd.org/~kamil/gsoc2018_mentor_summit.html

Presenter Notes