Coverage of signal routines in the kernel in the context of ptrace(2)

During the past month I have been working on coverage of various corner cases in the signal subsystem in the kernel. I have also spent some time on improvements in the land of sanitizers. As a mentor I was able to, thanks to the fulltime focus on NetBSD work, actively help three Google Summer of Code students. Not every question would be answered by myself without code reading but at least I am available for active collaboration, especially when it's to improve code that I have already authored, like sanitizers. At the end of the month we have managed to catch two uninitialized memory reads in the top(1) utility, using the Memory Sanitizer feature and rebuilt part of the basesystem (i.e. library dependencies: libterminfo, libkvm, libutil) with dedicated sanitization flags.

ptrace(2) and related distribution changes

I am actively working on handling of processes, forks/vforks, signals and threads that is reliable and fully functional under a debugger. This is a process and the situation is actively improving. For the end-user this means that we are achieving the state when a developer will be able to trace an application like Firefox using modern tools and save time detecting the issues quickly.

I am using the Test-Driven Development approach in my work. I keep extending the Automatic Test Framework with new tests, covering sets of scenarios handled by debuggers and related code. This is followed by kernel fixes. Thanks to the tests, I can more confidently introduce changes to critical routines inside the Operating System, test new changes quickly for regressions and keep covering new verifable scenarios.

Titles of the merged commits with the main tree of NetBSD:

LLVM compiler-rt features

I have helped the GSoC student to prepare for LLVM libfuzzer integration with the NetBSD base system. We have managed to get down to the following results for the test target in the upstream repository:

$ check-fuzzer-default

  Expected Passes    : 105
  Unsupported Tests  : 8
  Unexpected Failures: 2

$ check-fuzzer

  Expected Passes    : 105
  Unsupported Tests  : 8
  Unexpected Failures: 2

$ check-fuzzer-unit

  Expected Passes    : 35

The remaining two failures appear to be false positives and specific to the differences between the NetBSD setup difference and other supported Operating Systems (including Linux). I have decided not to investigate them and instead to move on to more urgent tasks.

While there, I have been working on restoring a good state to userland LLVM sanitizers in the upstream repository, in order ship them in the NetBSD distribution along with the libfuzzer utility. While there, I've been working on restoring a good state with userland LLVM sanitizers in the upstream repository, in order ship them into the NetBSD distribution, along the libfuzzer utility.

A number of patches were merged upstream:

There is also at least a single pending upstream patch that is worth to note: Introduce CheckASLR() in sanitizers

At least the ASan, MSan, TSan sanitizers require disabled ASLR on a NetBSD.

Introduce a generic CheckASLR() routine, that implements a check for the
current process. This flag depends on the global or per-process settings.

There is no simple way to disable ASLR in the build process from the
level of a sanitizer or during the runtime execution.

With ASLR enabled sanitizers that operate over the process virtual address
space can misbehave usually breaking with cryptic messages.

This check is dummy for !NetBSD.

The current results for test targets in the compiler-rt features are as follows:

$ make check-builtins

  Expected Passes    : 343
  Expected Failures  : 4
  Unsupported Tests  : 36
  Unexpected Failures: 5

$ check-interception

-- Testing: 0 tests, 0 threads --

$ check-lsan

  Expected Passes    : 6
  Unsupported Tests  : 60
  Unexpected Failures: 106

$ check-ubsan

  Expected Passes    : 229
  Expected Failures  : 1
  Unsupported Tests  : 32
  Unexpected Failures: 2

$ check-cfi

  Unsupported Tests  : 232

$ check-cfi-and-supported

BaseException: Tests unsupported

$ make check-sanitizer

  Expected Passes    : 576
  Expected Failures  : 13
  Unsupported Tests  : 206
  Unexpected Failures: 31

$ check-asan

  Expected Passes    : 852
  Expected Failures  : 4
  Unsupported Tests  : 440
  Unexpected Failures: 16

$ check-asan-dynamic

  Expected Passes    : 394
  Expected Failures  : 3
  Unsupported Tests  : 440
  Unexpected Passes  : 1
  Unexpected Failures: 222

$ check-msan

  Expected Passes    : 102
  Expected Failures  : 1
  Unsupported Tests  : 30
  Unexpected Failures: 4

$ check-tsan

  Expected Passes    : 288
  Expected Failures  : 1
  Unsupported Tests  : 84
  Unexpected Failures: 8

$ check-safestack

  Expected Passes    : 7
  Unsupported Tests  : 1

$ check-scudo

  Expected Passes    : 14
  Unexpected Failures: 28

$ check-ubsan-minimal

  Expected Passes    : 6
  Unsupported Tests  : 2

$ check-profile

  Unsupported Tests  : 116

$ check-xray

  Expected Passes    : 21
  Unsupported Tests  : 1
  Unexpected Failures: 21

$ check-shadowcallstack

  Unsupported Tests  : 4

Sanitization of userland and the kernel

I am helping to setup the process for shipping a NetBSD userland that is prebuilt with a desired sanitizer. This involves consulting the Google Summer of Code student, fixing known issues, reviewing patches etc.

There were two new uninitialized memory read bugs detected in the top(1) program:

Fix unitialized signal mask passed to sigaction(2) in top(1)

Detected with Memory Sanitizer during the integration of sanitizers with
the NetBSD basesystem.

Reported by <Yang Zheng>

Fix read of uni[ni]tialized array elements in top(1)

The cp_old array is allocated with malloc(3) and its pointer is passed to
percentages64().

In this function there happens a calculation of total_change, which value
depends on the value inside the unitialized cp_old[] array.

==26662==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x268a2c in percentages64 /usr/src/external/bsd/top/bin/../dist/machine/m_netbsd.c:1341:6
#1 0x26748b in get_system_info /usr/src/external/bsd/top/bin/../dist/machine/m_netbsd.c:478:6
#2 0x25518e in do_display /usr/src/external/bsd/top/bin/../dist/top.c:507:5
#3 0x253038 in main /usr/src/external/bsd/top/bin/../dist/top.c:975:2
#4 0x21cad1 in ___start (/usr/bin/top+0x1cad1)
SUMMARY: MemorySanitizer: use-of-uninitialized-value /usr/src/external/bsd/top/bin/../dist/machine/m_netbsd.c:1341:6 in percentages64
Exiting

Fix this issue by chang[]ing malloc(3) with calloc(3).

Detected with Memory Sanitizer during the integration of sanitizers with
the NetBSD basesystem.

Reported by <Yang Zheng>

As similar process happens with two kernel sanitizer GSoC tasks: kernel-ubsan and kernel-asan.

Thanks to the involvement to The NetBSD Foundation tasks, I can be reachable for students (although not always in all cases) for active feedback and collaboration.

Summary

The number of ATF ptrace(2) tests cases has been significantly incremented, however there is still a susbstantial amount of work to be done and a number of serious bugs to be resolved.

With fixes and addition of new test cases, as of today we are passing 1,206 (last month: 961) ptrace(2) tests and skipping 1 (out of 1,256 total; last month: 1,018 total). No counted here tests that appeared outside the ptrace(2) context.

Plan for the next milestone

Cover with regression tests remaining elementary scenarios of handling crash signals. Fix known bugs in the NetBSD kernel.

Follow up the process with the remaining fork(2) and vfork(2) scenarios.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:

http://netbsd.org/donations/#how-to-donate