The LLDB Debugger on NetBSD

Presenter Notes

netbsd

EuroBSDcon 2017

Author: Kamil Rytarowski

E-mail: kamil@netbsd.org

Date: September 24th 2017

Place: Paris, France

Presenter Notes

Bio

Kamil Rytarowski (born 1987)

Krakow, Poland

NetBSD user since 6.1.

The NetBSD Foundation member since 2015.

Work areas: kernel, userland, pkgsrc.

Interest: NetBSD on desktop and in particular NetBSD as a workstation.

The current activity in 3rd party software:

  • LLVM committer.
  • GDB & binutils committer.
  • NetBSD maintainer in qemu.

Presenter Notes

Topics

  • Elementary terminology
  • Goals and benefits of porting the LLDB Debugger to NetBSD.
  • What to expect when using the new debugger on NetBSD.
  • Impact of this port on the base distribution.
  • The process tracing interface.
  • New kernel features to host the new debugger.
  • Regression tests for the ptrace(2) system call.
  • Tracking LLDB's trunk.
  • Sanitizers.
  • Pending tasks, known bugs and missing features.

Presenter Notes

Elementary terminology

Presenter Notes

Toolchain (software)

Toolchain - A set of programming development tools for creating a software product.

A basic UNIX toolchain consists of: a compiler, a linker, libraries, and a debugger.

A more complex toolchain may consist of additional elements like disassemblers, profilers, sanitizers etc. An advanced toolchain may ship with 3D-graphics modelers, audio synthetizers etc.

toolbox

[cclasa 2.5, Erik Strandberg, 2006; wikimedia]

Presenter Notes

Debugger

A computer program that can test and trace a computer program in order to assist detection of computer program bugs.

In this presentation we will focus on the system-level debuggers (referring to it as a plain "debugger") designed for general-purpose Operating Systems like UNIX, executing native code with an optional involvement of a simulator.

ddd

[GPL, Peter Wainwright, 2010; wikimedia]

Presenter Notes

Debugger

Debugging is the process of using a tracer and searching for a solution to issues.

A usual workflow using a debugger:

  • Attempt to reproduce the issue.
  • Isolation of the issue.
  • Detection of the source of the issue.
  • Removal of the programming error.
  • Verification that the problem is gone.

Presenter Notes

Debugger

Tracing involves starting & stopping of a process (and threads) and process introspection into memory, register context, thread information etc.

The user-visible result of the process introspection is the state of threads, current line of the code in the program-counter register, status of local variables, stack of function calls with the passed optional arguments etc.

 1 (gdb) b fun3
 2 Breakpoint 1 at 0x4007e7: file main.c, line 3.
 3 (gdb) r
 4 Starting program: /tmp/a.out
 5 
 6 Breakpoint 1, fun3 (a=1) at main.c:3
 7 3       return a + 1;
 8 (gdb) bt
 9 #0  fun3 (a=1) at main.c:3
10 #1  0x0000000000400804 in fun2 (a=1) at main.c:8
11 #2  0x000000000040081b in fun (a=1) at main.c:13
12 #3  0x0000000000400836 in main (argc=1, argv=0x7f7fff270fb0) at \
13 main.c:18

Debuggers should support the debugged program's language like C, C++ or Fortran.

Presenter Notes

Debugger

A program for better debugging experience should be built with additional debuginfo information and ship with available source code.

1 #include <stdio.h>
2 int main(int argc, char **argv)
3 {
4     printf("Hello world\n");
5 
6     return 0;
7 }

Compilation with debug symbols:

1 gcc -g -Og program.c -o program

Presenter Notes

Debugger

A debugger runs as a regular process and traces another entity being another process or process image stored on disk (a core file).

Program:

1 $ file /usr/pkg/bin/firefox
2 /usr/pkg/bin/firefox: symbolic link to /usr/pkg/lib/firefox/firefox
3 
4 $ file /usr/pkg/lib/firefox/firefox
5 /usr/pkg/lib/firefox/firefox: ELF 64-bit LSB shared object, x86-64, \
6 version 1 (SYSV), dynamically linked, interpreter                   \
7 /usr/libexec/ld.elf_so, for NetBSD 8.99.2, PaX: -mprotect,          \
8 BuildID[sha1]=74d0f5fe231dcc4148656e613fb9c9b45cb9183d,             \
9 with debug_info, not stripped

Core file:

1 $ file firefox.core
2 firefox.core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), \
3 NetBSD-style, from 'firefox', pid=1407, uid=1000, gid=100,       \
4 nlwps=10, lwp=1 (signal 11/code 32767)

Presenter Notes

Debugger

A debugger reuses Operating System and hardware specific features exported in Machine-Dependent style to userspace programs through kernel API.

Unsupported hardware features by a certain (or all) hardware can be reimplemented with a simulator, while the rest executes underlying hardware capabilities.

lupa

[Public Domain, Julo, 2007; wikimedia]

Presenter Notes

lldb_logo

LLDB is a next generation, high-performance debugger. It is built as a set of reusable components which highly leverage existing libraries in the larger LLVM Project, such as the Clang expression parser and LLVM disassembler.

LLDB is the default debugger in Xcode on Mac OS X and supports debugging C, Objective-C and C++ on the desktop and iOS devices and simulator.

All of the code in the LLDB project is available under the standard LLVM License, an open source "BSD-style" license.

-- Source: http://lldb.llvm.org/

Presenter Notes

Goals and benefits of porting the LLDB Debugger to NetBSD

Presenter Notes

LLDB

Goals and benefits of porting the LLDB Debugger to NetBSD:

  • LLDB is the natural choice for language runtimes based on LLVM. Some of them, like Swift or .NET, don't work on the GCC+GDB tandem.
  • Relatively low cost to gain support for language frontends and hardware backends thanks to shared code with LLVM and Clang. This includes the newest C++ standards.
  • Tradeoff of requiring advanced operating system facilities shipping legacy-free advanced solutions (C++11 and beyond, kernel introspection facilities etc).

The last point enforced the NetBSD kernel to ship reliable and functional features comparable to Windows, Darwin, Linux (&Android) and FreeBSD. NetBSD 8(beta) is now there in terms of the generic Machine-Independent kernel capabilities, and still not there in terms of reliability and availability of Machine-Dependent parts.

Presenter Notes

LLDB

Goals and benefits of porting the LLDB Debugger to NetBSD:

  • Modernization and verification of the process tracing interfaces - ptrace(2).
  • Indirect improvement of the GDB correctness and room for future upgrading to newer API capabilities.
  • LLVM clean basesystem distribution without the GNU toolchain parts (LLD porters wanted).
  • Positive impact on catching up with other debugging software like DTrace (libproc), strace etc and debugging capabilities like kernel debugging (restoring kgdb).

The GNU and LLVM toolchain tools can be treated as two legs on which a modern Operating System like NetBSD should stand. There is competition, but no obsolescence of one because of the other. One set of 3rd party tools will require the former, another set will require the latter.

Presenter Notes

LLDB

Goals and benefits of porting the LLDB Debugger to NetBSD:

  • Reusing shared components with other systems, this involves: ELF+DWARF parsers, core(5) framework, networking and filesystem capabilities.
  • Restricting (this is ongoing effort) of the NetBSD specific bits to NetBSD specific files.
  • Native support for cross-platform (cross-hardware, cross-OS etc) debugging of NetBSD programs.
  • Another tool of verification of correctness of LLVM and Clang on NetBSD.
  • Visibility in the LLVM project (and beyond it).
  • Making NetBSD the better generic purpose Operating System as a professional workstation (daily desktop driver for a developer).

Presenter Notes

What to expect when using the new debugger on NetBSD

Presenter Notes

LLDB 5.0.0

What to expect when using the new debugger on NetBSD.

Today in the mainline code and the 5.0.0 release:

  • Only NetBSD/amd64 support.
  • Support for tracing single-threaded applications.
  • Support for investigating core(5) files of applications with a single thread.
  • Client-Server model - modern process plugin framework shared with Linux (&Android).
  • Discovering external debuginfo data in NetBSD specific paths (/usr/libdata/debug).
  • Unfortunately, breakage of runtime stability in the late process of 5.0.0 development.

Presenter Notes

LLDB 5.0.0

What to expect when using the new debugger on NetBSD.

Local patches:

  • Support for core(5) files with multiple threads.

Presenter Notes

Impact of this port on the base distribution

Presenter Notes

Basesystem impact

LLDB depends on correct and modern support in: Clang, LLVM, userland libraries and kernel.

Everything is supported in the context of certain hardware - that cannot be programically fixed or improved - it must be supported as it is.

lldb

(the stack of dependencies of the LLDB debugger)

Presenter Notes

Basesystem impact - kernel

kernel

Presenter Notes

Basesystem impact - userland

userland

Presenter Notes

Basesystem impact - LLVM and Clang

While it might be unrelated to LLDB porting effort, I attempt to follow the approach of zero-problems on NetBSD/amd64

LLVM and Clang are important in the process of LLDB/NetBSD porting because:

  • Build problems propagade to other projects.
  • Bugs in LLVM and Clang can propagate to other projects.
  • Code-flow exchange between the LLVM projects family.

Presenter Notes

The process tracing interface

Presenter Notes

The process tracing interface

ptrace(2) - process tracing and debugging facility

By using process trace one process can take over another, inspect and manipulate its register context, address space and control flow.

Underneath this interface is using the reparenting mechanism. The tracer is a new parent, the tracee is a child and userland processes mostly don't notice it.

This interface is used in BSD operating systems and GNU/Linux (kernel) and they are relatively closely related.

Presenter Notes

Introspection interfaces

The difference between ptrace(2), sysctl(2) and kvm(3):

  • ptrace(2) - tracer-oriented introspection; privileged; with taking over the process and reparenting
  • sysctl(2) - generic purpose introspection including self-introspection; unprivileged; no reparenting
  • kvm(3) - kernel virtual memory interface; privileged for live-kernel reads, unprivileged for basic introspection (wraps internally sysctl(2)) and crash dump examination

Presenter Notes

Security

A process must have the same real UID as the tracing process, and also it must not be executing a setuid or setgid executable. (If the tracing process is running as root: these restrictions do not apply.)

Three restrictions apply to all tracing processes.

  • First, no process may trace a system process.
  • Second, no process may trace the process running init(8) (unless running as root and "Insecure" kernel mode).
  • Third, if a process has its root directory set with chroot(2), it may not trace another computer program unless that process' root directory is at or below the tracing process' root.

In order to put a trap by a tracer into the tracee's program, the debugger must violate the PaX MPROTECT restrictions.

1 $ sysctl -d security.pax.mprotect.ptrace
2 security.pax.mprotect.ptrace: When enabled, allow ptrace(2) to \
3 override mprotect permissions on traced processes

Presenter Notes

ptrace(2) reparenting

reparenting

Presenter Notes

Evolution of NetBSD's ptrace(2)

ptrace

Simplified evolution of the NetBSD's ptrace(2)
NetBSD's flag color: #F26711

Presenter Notes

ptrace(2) - function prototypes

NetBSD

1 int
2 ptrace(int request, pid_t pid, void *addr, int data);

Linux

1 long
2 ptrace(enum __ptrace_request request,
3        pid_t pid, void *addr, void *data);

FreeBSD

1 int
2 ptrace(int request, pid_t pid, caddr_t addr, int data);

OpenBSD

1 int
2 ptrace(int request, pid_t pid, caddr_t addr, int data);

Presenter Notes

ptrace(2) - included headers

NetBSD

1 #include <sys/types.h>
2 #include <sys/ptrace.h>

Linux

1 #include <sys/ptrace.h>

FreeBSD

1 #include <sys/types.h>
2 #include <sys/ptrace.h>

OpenBSD

1 #include <sys/types.h>
2 #include <sys/ptrace.h>

Presenter Notes

NetBSD's ptrace(2) reference

Resources:

  • the ptrace(2) man-page
  • ATF tests src/tests/lib/libc/sys/t_ptrace*
  • "Porting ptrace(2) software to NetBSD", Kamil Rytarowski, February 26th 2017

Presenter Notes

New kernel features to host the new debugger

Presenter Notes

ptrace(2) news in NetBSD 8(beta)

Development policies:

  • Do not break any known programs.
  • Clean design.
  • Holistic look at the existing API.
  • Feature-parity with Linux and FreeBSD.
  • Organic evolution of existing interface, reuse of them.
  • Reuse FreeBSD ideas when applicable.
  • Reuse Linux ideas when applicable.
  • Easy to use in a tracer.
  • Lack of ambiguity of the tracee's state.

Presenter Notes

ptrace(2) news in NetBSD 8(beta)

Future-ready policies:

  • Efficient tracing of processes with a massive number of threads.
  • Efficient tracing of multiprocess applications.
  • Overloading the existing API for in-process tracing of other threads?

Presenter Notes

ptrace(2) news in NetBSD 8(beta)

Major changes in the Machine-Independent part:

  • Event monitor through PT_GET_SIGINFO.
  • vfork(2) events.
  • Thread birth and termination trap.
  • Thread control (suspend/resume).
  • Per-thread stepping.
  • Syscall entry/exit trap.
  • exec(3) trap (unless syscall tracing).

Presenter Notes

ptrace(2) news in NetBSD 8(beta)

Major changes in the Machine-Independent part:

  • Debug Registers (amd64 and i386).

Presenter Notes

Regression tests for the ptrace(2) system call

Presenter Notes

ATF ptrace(2) tests

atf

Presenter Notes

Tracking LLDB's trunk

Presenter Notes

Tracking LLDB's trunk

Local development through pkgsrc-wip:

  • llvm-git, clang-git, lldb-git - track continuously upstream's HEAD
  • llvm-netbsd, clang-netbsd, lldb-netbsd - develop LLDB and surroundings against specified SVN revision

Presenter Notes

Tracking LLDB's trunk

NetBSD buildbot

buildbot

Presenter Notes

Sanitizers

Presenter Notes

Sanitizers

LLVM ships with sanitizers:

  • asan - address sanitizer - detects use-after-free, buffer overflow (C,C++)
  • ubsan - UB sanitizer - detects undefined behavior (C,C++)
  • msan - memory sanitizer - detects uninitialized memory read (C,C++)
  • tsan - thread sanitizer - detects data races (C,C++, Go)

asan and ubsan are already functional on NetBSD and fully upstreamed

msan and tsan are work-in-progress

Presenter Notes

Sanitizers - ubsan

Example source code:

1 int
2 main(int argc, char *argv[])
3 {
4         int abc[2];
5 
6         abc[argc + 3] = 123;
7 
8         return 0;
9 }

Execution of ubsan:

1 $ /usr/local/bin/clang -fsanitize=undefined 1.c -o 1.undef
2 $ ./1.undef
3 1.c:6:2: runtime error: index 4 out of bounds for type 'int [2]'

Presenter Notes

Sanitizers - asan

Execution of asan:

 1 $ /usr/local/bin/clang -fsanitize=address 1.c -o 1.addr    
 2 $ ./1.addr                                              
 3 =================================================================
 4 ==25664==ERROR: AddressSanitizer: stack-buffer-overflow on address \
 5 0x7f7fffffe650 at pc 0x00000040a359 bp 0x7f7fffffe610 sp 0x7f7fffffe608
 6 WRITE of size 4 at 0x7f7fffffe650 thread T0
 7     #0 0x40a358 in main (/tmp/1.addr+0x40a358)
 8     #1 0x40a13a in ___start (/tmp/1.addr+0x40a13a)
 9 
10 Address 0x7f7fffffe650 is located in stack of thread T0 at offset 48 \
11 in frame
12     #0 0x40a22f in main (/tmp/1.addr+0x40a22f)
13 
14   This frame has 1 object(s):
15     [32, 40) 'abc' <== Memory access at offset 48 overflows this \
16                        variable
17 [... memory dump here, hints etc ...]

Presenter Notes

LLVM libFuzzer and SafeStack

libFuzzer and SafeStack have been ported to NetBSD and upstreamed.

SafeStack is a software security hardening technique that creates two stacks: one for data that needs to be kept safe, such as return addresses and function pointers; and an unsafe stack for everything else.

LLVM libFuzzer is an in-process, coverage-guided, evolutionary fuzzing engine. As of now libFuzzer requires for more sanitizers to get aboard to NetBSD, especially lsan (Leak Sanitizer).

Presenter Notes

Pending tasks, known bugs and missing features.

Presenter Notes

Roadmap

  1. Make tsan+msan functional.
  2. Sort out lldb user-space bugs, desynced threads in user-space program LLDB. Restore the state before the breakage. Reuse sanitizers in this task.
  3. Sort out ptrace(2) bugs with debugees with more than 1 thread.
  4. Resume LLDB porting......
  5. ... sorting out bugs on the way of LLDB porting the elementary functionality in LLDB/NetBSD...
  6. Enable LLVM+Clang+LLDB tests on the buildbot.
  7. Productize sanitizers for NetBSD. Support GCC as a possible compiler?
  8. Enable compiler-rt tests on the NetBSD buildbot.

Presenter Notes

Unordered tasks

  • GDB restoration on NetBSD and catch up after Linux
  • kgdb in LLDB
  • revisit ipkdb - in-kernel IP-based remote kernel debugging (currently only supported NIC: NE2000)
  • kernel-asan porting to NetBSD
  • Machine-Dependent interfaces in ptrace(2) like XSAVE and XSAVEOPT accessors on amd64
  • new ptrace(2) interface to handle StopTheProcess() in lsan to self-introspect the process?
  • !amd64 dimension in LLDB/NetBSD

Presenter Notes

Credits

The NetBSD Foundation - for sponsoring the project.

The NetBSD developers and in particular Christos Zoulas, Martin Husemann, Thomas Klausner, K.Robert Elz, Nicolas Joly, Paul Goyette for help with code fixes, feedback and bug reports.

Presenter Notes

The LLDB Debugger on NetBSD

Navigare necesse est, vivere non est necesse

"to sail is necessary; to live is not necessary"

-- Gnaeus Pompeius Magnus minor

Presenter Notes