Operating systems can be called monitors as they handle system calls from userland processes. A similar task is performed by debuggers as they implement monitors for traced applications and interpret various events that occurred in tracees and are messaged usually with signals to their tracers. During this month I have started a new Process Plugin within LLDB to incept NativeProcessNetBSD - copied from NativeProcessLinux - implementing basic functionality and handling all the needed events in the MonitorCallback() function. To achieve these tasks, I had to add a bunch of new ptrace(2) interfaces in the kernel to cover all that is required by LLDB monitors. The current Process Plugin for NetBSD is capable to start a process, catch all the needed events correctly and if applicable resume or step the process.

What has been done in NetBSD

1. Verified the full matrix of combinations of wait(2) and ptrace(2) in the following test-cases

2. GNU libstdc++ std::call_once bug investigation

The LLVM toolchain internally uses several std::call_once calls, within LLVM and LLDB codebase. The GNU libstdc++ library implements C++11 std::call_once with wrapping pthread_once(3) and a function pointer passed as a TLS (Thread Local Storage) variable. Currently this setup isn't functional with shared linking on NetBSD and apparently on few other systems and hardware platforms. This issue is still unresolved and its status is traced in PR 51139. As of now, there is a walkaround originally prepared for NetBSD, and adapted for others - namely llvm::call_once.

3. Improving documentation and other minor system parts

4. Documentation of ptrace(2) and explanation how debuggers work

5. Introduction of new siginfo(2) codes for SIGTRAP

6. New ptrace(2) interfaces

There were added new interfaces to the native ptrace(2) NetBSD API.

Interface to introspect and fake signal information

The PT_GET_SIGINFO interface is designed to read signal information sent to tracee. A program can also fake a signal to be passed to a debugged process, with included destination thread or the whole process. A debugger requires detailed signal information in order to distinguish the exact trap reason easily and precisely, for example whether a program stopped on a software defined breakpoint or a single step trap.

This interface is modeled after the Linux specific calls: PTRACE_GETSIGINFO and PTRACE_SETSIGINFO, but adapted for the NetBSD use-case. FreeBSD currently has no way to fake signal information.

I have modeled this interface to be most efficient in terms of determination what exact thread received the signal. Thanks to it, a process does not need to examine each thread separately and performs only a single ptrace(2) call. This makes a significant difference in a massively threaded software.

The signal information accessors introduce a new structure ptrace_siginfo: 

/*
 * Signal Information structure
 */
typedef struct ptrace_siginfo {
       siginfo_t       psi_siginfo;    /* signal information structure */
       lwpid_t         psi_lwpid;      /* destination LWP of the signal
                                        * value 0 means the whole process
                                        * (route signal to all LWPs) */
} ptrace_siginfo_t;
I've included in the <sys/ptrace.h> header required file to define the siginfo_t type - <sys/siginfo.h>. This makes sure that no existing software will break during build due to a missing type definition.

Interface to monitor vfork(2) operations

Forking a process is one of the fundamental design features of a unix operating system. There are two basic types of forks: fork(2) and vfork(2) ones. The fork(2) one is designed to spawn a mostly independent child from parent's memory layout, however this operation is costy. In order to address it and optimize the forking operation there is vfork(2) which shares address space with its parent. From the ptrace(2) point of view, the difference between fork(2) and vfork(2) calls is whether a parent giving birth to its child is suspended waiting on child termination or exec() operation or not.

Currently there is an interface to monitor fork(2) operations - PTRACE_FORK - however NetBSD missed the vfork(2) ones. I've added two new functions: PTRACE_VFORK and PTRACE_VFORK_DONE. The former is supposed to notify why a parent gives a vfork(2)-like birth to its child and later when a child is born with vfork(2) from its parent - perfoming a handshake with two SIGTRAP signals emitted from the kernel. Once the parent is resuming it gets a notification for vfork(2) done.

These events throw SIGTRAP signal with the TRAP_CHLD property. Currently PTRACE_VFORK is a stub and it's planned to be fully implemented later.

Interface to monitor thread operations

A debugger requires an interface to monitor thread birth and termination. This is needed in order to properly track the thread list of a process and ensure that every thread has for example applied watchpoints.

I've added two events:

This interface reuses the EVENT_MASK and PROCESS_STATE interface. It means that it shares these calls with PTRACE_FORK, PTRACE_VFORK and PTRACE_VFORK_DONE.

To achieve this goal, I've changed the following structure: 

typedef struct ptrace_state {
        int     pe_report_event;
        pid_t   pe_other_pid;
} ptrace_state_t;

to 

typedef struct ptrace_state {
        int     pe_report_event;
        union {
                pid_t   _pe_other_pid;
                lwpid_t _pe_lwp;
        } _option;
} ptrace_state_t;

#define pe_other_pid    _option._pe_other_pid
#define pe_lwp          _option._pe_lwp
This change keeps the size of ptrace_state_t unchanged as both pid_t and lwpid_t are defined as an int32_t-like integer. New struct form should not break existing software and be source and binary compatible with it.

I've introduced a new SIGTAP type for thread events: TRAP_LWP.

Hardware assisted watchpoints

I've introduced a few changes to the current interface. One of them is allowing to mix single-step operation with enabled hardware assisted watchpoints. The other one was added new extension pw_type to the ptrace_watchpoint structure.

6. Updated doc/TODO.ptrace entries

The current state of TODO.ptrace - after several updates - is as follows:

Features in ELF, DWARF, CTF, DTrace are out of scope for the above list.

7. Future directions

After research and testing the current watchpoint interface, I've realized that it's impossible (impractically complicated) to pretend to have a "safe" watchpoint interface inside the kernel, as the current one isn't safe from undefined behavior even on stock amd64. I've decided to revert this code and introduce PT_GETDBREGS and PT_SETDBREGS restricted to INSECURE secure level mode. The good side of this change is that there is already part of the code needed in the kernel, I have a local draft introducing this interface and it will be easier to integrate with LLDB, as Linux and FreeBSD keep having the same interface.

What has been done in LLDB

I work on the LLDB port inside the pkgsrc-wip repository, in the lldb-netbsd package.

To summarize the changes, there were so far 84 commits in this directory. The overall result is a list of 26 patched or added files. The overall diff's length is 3539 lines. 

$ wc -l patches/patch-*                                                               
      12 patches/patch-cmake_LLDBDependencies.cmake
      17 patches/patch-cmake_modules_AddLLDB.cmake
      14 patches/patch-include_lldb_Host_netbsd_HostThreadNetBSD.h
      30 patches/patch-include_lldb_Host_netbsd_ProcessLauncherNetBSD.h
      12 patches/patch-source_CMakeLists.txt
      12 patches/patch-source_Host_CMakeLists.txt
      60 patches/patch-source_Host_common_Host.cpp
      13 patches/patch-source_Host_common_NativeProcessProtocol.cpp
      21 patches/patch-source_Host_netbsd_HostThreadNetBSD.cpp
     175 patches/patch-source_Host_netbsd_ProcessLauncherNetBSD.cpp
      23 patches/patch-source_Host_netbsd_ThisThread.cpp
      24 patches/patch-source_Initialization_SystemInitializerCommon.cpp
     803 patches/patch-source_Plugins_Platform_NetBSD_PlatformNetBSD.cpp
     141 patches/patch-source_Plugins_Platform_NetBSD_PlatformNetBSD.h
      12 patches/patch-source_Plugins_Process_CMakeLists.txt
      13 patches/patch-source_Plugins_Process_NetBSD_CMakeLists.txt
    1392 patches/patch-source_Plugins_Process_NetBSD_NativeProcessNetBSD.cpp
     188 patches/patch-source_Plugins_Process_NetBSD_NativeProcessNetBSD.h
     393 patches/patch-source_Plugins_Process_NetBSD_NativeThreadNetBSD.cpp
      92 patches/patch-source_Plugins_Process_NetBSD_NativeThreadNetBSD.h
      13 patches/patch-tools_lldb-mi_MICmnBase.cpp
      13 patches/patch-tools_lldb-mi_MICmnBase.h
      13 patches/patch-tools_lldb-mi_MIDriver.cpp
      13 patches/patch-tools_lldb-mi_MIUtilString.cpp
      13 patches/patch-tools_lldb-mi_MIUtilString.h
      27 patches/patch-tools_lldb-server_CMakeLists.txt
    3539 total

1. Native Process NetBSD Plugin

I've created the initial code for the Native Process NetBSD Plugin.

2. The MonitorCallback function

The MonitorCallback function supports now the following events:

3. Other LLDB code, out of the NativeProcessNetBSD Plugin

During this work segment I've completed the following tasks:

4. Automated LLDB Test Results Summary

The number of passing tests increased by 45% between devel/lldb 3.9.1 and lldb-netbsd 2017-01-21.

The above graphs renders test results for:

Example LLDB sessions

It's demo time!

Breakpoint interrupt

In this example, I'm calling a hello world application that is triggering a software breakpoint. It's implemented by embedding int3 call on amd64. The debugger is capable of catching this and resuming till correct process termination. 

$ lldb ./int3 
(lldb) target create "./int3"
Current executable set to './int3' (x86_64).
(lldb) r
Hello world!
Process 29578 launched: './int3' (x86_64)
Process 29578 stopped
* thread #1, stop reason = signal SIGTRAP
    frame #0:
(lldb) c
Process 29578 resuming
Process 29578 exited with status = 0 (0x00000000)
(lldb)

Thread monitor interrupt

In this example we set trap on thread events - creation and termination. The executed program incepts a thread and terminates afterwards, this is caught by the MonitorCallback with appropriate thread list update. After the end, program terminates correctly and passes proper exit status to the debugger. 

$ lldb ./lwp_create
(lldb) target create "./lwp_create"
Current executable set to './lwp_create' (x86_64).
(lldb) r
Process 27331 launched: './lwp_create' (x86_64)
Hello world!
Process 27331 stopped
* thread #1, stop reason = SIGTRAP has been caught with Process LWP Trap type
    frame #0:
  thread #2, stop reason = SIGTRAP has been caught with Process LWP Trap type
    frame #0:
(lldb) thread list 
Process 27331 stopped
* thread #1: tid = 0x0001, stop reason = SIGTRAP has been caught with Process LWP Trap type
  thread #2: tid = 0x0002, stop reason = SIGTRAP has been caught with Process LWP Trap type
(lldb) c
Process 27331 resuming
Process 27331 stopped
* thread #1, stop reason = SIGTRAP has been caught with Process LWP Trap type
    frame #0:
(lldb) thread list
Process 27331 stopped
* thread #1: tid = 0x0001, stop reason = SIGTRAP has been caught with Process LWP Trap type
(lldb) c
Process 27331 resuming
It works
Process 27331 exited with status = 0 (0x00000000)
(lldb)

Plan for the next milestone

I've listed the following goals for the next milestone.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue to fund projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:

http://netbsd.org/donations/#how-to-donate