NetBSD BoF

Author: Kamil Rytarowski

E-mail: kamil@netbsd.org

Date: March 21st, 2019

Location: Tokyo, Japan

Kamil Rytarowski (born 1987)

Krakow, Poland

NetBSD user since 6.1.

NetBSD Foundation member since 2015.

Work areas: kernel, userland, pkgsrc.

Interest: NetBSD on desktop and in particular NetBSD as a workstation.

Current activity in 3rd party software:

• LLVM committer.
• GDB & binutils committer.
• NetBSD maintenance in qemu.

• LLVM projects
• Kernel sanitizing and fuzzing
• Debuggers
• Virtualization

Builds and executes regression tests on the NetBSD buildbot (thanks TNF for hosting it) in the LLVM buildfarm:

• LLVM
• Clang
• libc++ (flaky in POSIX timed mutexes -- looking forward to futexes)
• libc++ABI
• OpenMP
• polly
• LLD (sometimes flaky and needs NetBSD specific downstream patches)
• LLDB (still flaky on the bot, works for tracing applications with 1 thread and examining core(5) files with 1 thread, x86_64 only)
• libunwind (no backtracing through NetBSD signal handlers)

Contains low-level utility code for Operating Systems.

There are components (among others):

• builtins - like libgcc from GNU
• profile - library which is used to collect coverage information
• BlocksRuntime - a target-independent implementation of Apple "Blocks" runtime interfaces
• sanitizer runtimes - runtime libraries that are required to run the code with sanitizer instrumentation
• libFuzzer - fuzzer for libraries
• scudo - hardened allocator
• SafeStack - stack hardening
• XRay - DTrace-like compiler instrumentation

New features whenever there is need for generic NetBSD distribution changes (especially the kernel part), target NetBSD 9.0.

NetBSD supports the most important ones:

• Address Sanitizer (finds unauthorized memory access)
• Thread Sanitizer (finds threading bugs)
• Memory Sanitizer (finds uninitialized memory reads)
• Undefined Behavior Sanitizer (finds unspecified semantics in runtime)

TODO:

• Leak Sanitizer (finds memory leaks) - pending ptrace(2) stabilization
• Dataflow Sanitizer (generic sanitizer framework for dataflow) - should work almost out of the box but use-case is not clear (libFuzzer assist?)
• Hardware assisted Address Sanitizer - aarch64 only

• Micro UBSan - linked into libc (MKLIBCSANITIZER)

Supported options:

• Kernel ASan (thanks Siddharth for initial scratch and Maxime Villard for actual port)
• Kernel UBSan
• KLEAK (Maxime Villard)
• KCOV - kernel coverage suited for fuzzers (syzkaller..) and sanitizers (like KLEAK, but KLEAK uses homegrown code for KCOV)

Kernel fuzzers:

• syzkaller https://syzkaller.appspot.com/#netbsd

24h/7d kernel fuzzing in GCE (hosted by Google, thanks!)

TODO:

• Kernel MSan - Linux accomplished its support and Clang/LLVM, pending to port to NetBSD (by Maxime) - waiting for basesystem upgrade to LLVM 8.0
• Kernel TSan - stalled for Linux

Ongoing port of LLDB to NetBSD sponsored by TNF.

Michal Gorny (mgorny) formally joined the project since Feburary 2019.

Two tracks of the porting effort:

• Addressing remaining kernel failures in ptrace(2) & related interfaces (kamil@)
• LLDB porting and maintaining the NetBSD buildbot (mgorny@)

ptrace(2) in NetBSD-8.0 improved hugely since NetBSD-7.0. There were a lot of panics, almost every operation was broken, how did it ever worked for anything?

ptrace(2) is now feature complete (equivalent to Linux and FreeBSD) in terms of API features. It's still not there yet in terms of correctness.

Remaining major ptrace(2)-related issues:

• fork(2)/vfork(2) related signaling correctness
• threading (LWP) synchronization bugs
• fork1(9) use-after-free violations
• XSAVE, XSAVEOPT ptrace(2) support for NetBSD/amd64

Why to not work on GDB directly as there is some level of NetBSD support?

It's easier to get clean room implementation from scratch in the cotext of LLDB. LLDB mandates modern kernel support and strips legacy approaches used in old kernels of the age of year 1995 (retained in GDB).

The GDB support needs to be reworked with Remote Process plugin framework (gdb-server support, no other BSD is there).

It's easier to not break existing users of GDB and keep the focus on kernel & ptrace.

Once LLDB will be done, we will work on revamping GDB.

Additionally LLDB is mandatory for .NET, Swift. Today some environments need GDB, others LLDB. We need both. (The same like GCC and Clang)

NetBSD is the only BSD supported in the moderm framework (lldb-server). NetBSD is catching up after Linux now.

There is support:

• for tracing programs with 1 thread (NetBSD/amd64).
• investigation of core(5) files (NetBSD/amd64) with 1 (LLVM 8.0) or more threads (LLVM HEAD).

Major pending tasks in the contracted work:

• address flaky tests in the LLDB test-suite.
• support for multiple threads in live applications.
• XSAVE, XSAVEOPT support for NetBSD/amd64 live applications and core(5) files.
• NetBSD/i386 and NetBSD/aarch64 support
• support for backtracing through signal handlers

Expected final goal: tracing firefox-like applications flawlessly with a debugger. this browser uses multiple processes, multiple threads and crash signal masking.

Beyond the TNF contract:

• kgdb support
• ipkgdb support (?)
• qemu gdb-server & lldb interoperability
• expanding NetBSD support to all the other possible CPUs (PPC, MIPS, ARM 32-bit, ...)

Nice to have projects:

• light NetBSD homegrown lldb-server for low-end devices implemented in C

Intel Hardware Accelerated eXecution Manager - multi-kernel hypervisor for Intel x86_64 CPUs has been ported to the NetBSD kernel.

• Keep improving support of kernel fuzzers
• Finalize .NET support (major soft blockers: robust POSIX mutexes, finishing LLDB)
• VirtualBox for NetBSD as host (major blockers: missing features in kevent(2) and stability fixes)
• XRay-as-DTrace in the kernel - instrumentation from the kernel + light dedicated runtime (few kLOC)
• Valgrind port to NetBSD/amd64
• NetBSD/amd64 MULTIBOOT (support for qemu -kernel)
• namespaces and cgroups on NetBSD (adapted for our internal specifics)