The LLVM Memory Sanitizer support work in progress
<p>
In the past 31 days, I've managed to get the core functionality of MSan to work.
This is an uninitialized memory usage detector.
MSan is a special sanitizer because it requires knowledge of every entry to the basesystem library and every entry to the kernel through public interfaces.
This is mandatory in order to mark memory regions as initialized.
<p>
Most of the work has been done directly for MSan. However, part of the work helped generic features in compiler-rt.
<p>
<h1>Sanitizers</h1>
<p>
Changes in the sanitizer are listed below in chronological order.
Almost all of the changes mentioned here landed upstream.
A few small patches were reverted due to breaking non-NetBSD hosts and are rescheduled for further investigation.
I maintain these patches locally and have moved on for now to work on the remaining features.
<p>
<ul>
<li>Move __tsan::Vector to the __sanitizer namespace. This move allows reuse of the homegrown Vector container in sanitizers other than TSan.
I've implemented atexit(3) and __cxa_atexit() handling in MSan using this class.</li>

<li>I've added function renaming of the NetBSD libc calls to the common code of sanitizer interceptors (sanitizer_common_interceptors.inc).
There have been 37 renames registered.
In general, I don't intend to support compat code and focus entirely on newly compiled features, taking NetBSD-8 as the basesystem release.
Therefore I support only the newest function version that is basically always used in new code.
<p>
<ul>
<li>clock_gettime -> __clock_gettime50</li>
<li>clock_getres -> __clock_getres50</li>
<li>clock_settime -> __clock_settime50</li>
<li>setitimer -> __setitimer50</li>
<li>getitimer -> __getitimer50</li>
<li>opendir -> __opendir30</li>
<li>readdir -> __readdir30</li>
<li>time -> __time50</li>
<li>localtime_r -> __localtime_r50</li>
<li>gmtime_r -> __gmtime_r50</li>
<li>gmtime -> __gmtime50</li>
<li>ctime -> __ctime50</li>
<li>ctime_r -> __ctime_r50</li>
<li>mktime -> __mktime50</li>
<li>getpwnam -> __getpwnam50</li>
<li>getpwuid -> __getpwuid50</li>
<li>getpwnam_r -> __getpwnam_r50</li>
<li>getpwuid_r -> __getpwuid_r50</li>
<li>getpwent -> __getpwent50</li>
<li>glob -> __glob30</li>
<li>wait3 -> __wait350</li>
<li>wait4 -> __wait450</li>
<li>readdir_r -> __readdir_r30</li>
<li>setlocale -> __setlocale50</li>
<li>scandir -> __scandir30</li>
<li>sigtimedwait -> __sigtimedwait50</li>
<li>sigemptyset -> __sigemptyset14</li>
<li>sigfillset -> __sigfillset14</li>
<li>sigpending -> __sigpending14</li>
<li>sigprocmask -> __sigprocmask14</li>
<li>shmctl -> __shmctl50</li>
<li>times -> __times13</li>
<li>stat -> __stat50</li>
<li>getutent -> __getutent50</li>
<li>getutxent -> __getutxent50</li>
<li>getutxid -> __getutxid50</li>
<li>getutxline -> __getutxline50</li>
</ul>
</li>
<li>I've disabled intercepting absent functions on NetBSD, located in the MSan specific code: mempcpy, __libc_memalign, malloc_usable_size, stpcpy, gcvt, wmempcpy, fcvt.</li>
<li>I've added handling of symbol aliases of certain POSIX threading library functions in MSan interceptors, namely __libc_thr_keycreate -> pthread_key_create.</li>
<li>I've reused the Linux/x86_64 memory layout for NetBSD/amd64. There is a requirement to disable ASLR to get a functional mapping of memory regions on NetBSD. 
The NetBSD PaX ASLR implementation is too aggressive for the MSan needs, and this is similar to the TSan/NetBSD case.</li>
<li>I've switched the strerror_r(3) interceptor for NetBSD from the GNU-specific to the POSIX one. They behave differently and are not interchangeable.</li>
<li>I've disabled unsupported MSan tests on NetBSD: ftime and pvalloc and tsearch. They ship with unportable function calls that are absent on NetBSD.</li>
<li>I've corrected execution of the ifaddrs, textdomain and iconv tests on NetBSD. They were adapted for the NetBSD specific behavior.</li>
<li>I've finished handling in the interceptors the TLS block of the main program.
This is a continuation of the past month's effort, the new solution is now corrected and has been made machine independent.
It's worth noting that NetBSD is the only OS that is handling the main thread TLS block with generic interfaces.</li>
<li>I've introduced dlopen(3) support fixes. I've corrected the retrieval of link_map pointer from the dlopen(3) opaque handler structure.</li>
<li>I've fixed the list of intercepting functions for NetBSD: disabling getpshared-specific functions from POSIX pthreading library and switching from UTMP-like to UTMPX-like interceptors.</li>
<li>I've disabled a MSan test verifying fgetgrent_r() on NetBSD, as this function call is absent.</li>
<li>By mistake, I've disabled handling of fstatat()/MSan on NetBSD, and I've enabled it again.</li>
<li>MSan is supporting all GLIBC strtol-like function symbols in the GNU C library. I've adopted this for the NetBSD case intercepting only the _l variations.</li>
</ul>
<p>
<h1>NetBSD syscall hooks</h1>
<p>
I wrote a large patch (815kb!) adding support for NetBSD syscall hooks for use with sanitizers.
I wrote the following description on the still pending patch for <a href="https://reviews.llvm.org/D41588">review</a>:
<p>
<blockquote>
Implement the initial set of NetBSD syscall hooks for use with sanitizers.
<p>
Add a script that generates the rules to handle syscalls<br>
on NetBSD: generate_netbsd_syscalls.awk. It has been written<br>
in NetBSD awk(1) (patched nawk) and is compatible with gawk.<br>
<p>
Generate lib/sanitizer_common/sanitizer_platform_limits_netbsd.h<br>
that is a public header for applications, and included as:<br>
&lt;sanitizer_common/sanitizer_platform_limits_netbsd.h&gt;.
<p>
Generate sanitizer_netbsd_syscalls.inc that defines all the<br>
syscall rules for NetBSD. This file is modeled after the Linux<br>
specific file: sanitizer_common_syscalls.inc.
<p>
Start recognizing NetBSD syscalls with existing sanitizers:<br>
ASan, ESan, HWASan, TSan, MSan, TSan.
<p>
Update the list of platform (NetBSD OS) specific structs<br>
in lib/sanitizer_common/sanitizer_platform_limits_netbsd.
<p>
This patch does contain the most wanted structs<br>
and handles the most wanted syscalls as of now, the rest<br>
of them will be implemented in future when needed.
<p>
This patch is 815KB, therefore I will restrict the detailed<br>
description to a demo:
<p>
<code> <pre>
$ uname -a
NetBSD chieftec 8.99.9 NetBSD 8.99.9 (GENERIC) #0: Mon Dec 25 12:58:16 CET 2017  root@chieftec:/public/netbsd-root/sys/arch/amd64/compile/GENERIC amd64
$ cat s.cc                                                                                                                   
#include &lt;assert.h&gt;
#include &lt;errno.h&gt;
#include &lt;glob.h&gt;
#include &lt;stdio.h&gt;
#include &lt;string.h&gt;

#include &lt;sanitizer/netbsd_syscall_hooks.h&gt;

int main(int argc, char *argv[]) {
  char buf[1000];
  __sanitizer_syscall_pre_recvmsg(0, buf - 1, 0);
  // CHECK: AddressSanitizer: stack-buffer-{{.*}}erflow
  // CHECK: READ of size {{.*}} at {{.*}} thread T0
  // CHECK: #0 {{.*}} in __sanitizer_syscall{{.*}}recvmsg
  return 0;
}
$ ./a.out   
=================================================================
==18015==ERROR: AddressSanitizer: stack-buffer-underflow on address 0x7f7fffe9c2ff at pc 0x000000467798 bp 0x7f7fffe9c2d0 sp 0x7f7fffe9ba90
WRITE of size 48 at 0x7f7fffe9c2ff thread T16777215
    #0 0x467797 in __sanitizer_syscall_pre_impl_recvmsg /public/llvm/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_netbsd_syscalls.inc:394:3
    #1 0x4abeb2 in main (/public/llvm-build/./a.out+0x4abeb2)
    #2 0x419bba in ___start (/public/llvm-build/./a.out+0x419bba)

Address 0x7f7fffe9c2ff is located in stack of thread T0 at offset 31 in frame
    #0 0x4abd7f in main (/public/llvm-build/./a.out+0x4abd7f)

  This frame has 1 object(s):
    [32, 1032) 'buf' &lt;== Memory access at offset 31 partially underflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
      (longjmp and C++ exceptions *are* supported)
SUMMAR.Y: AddressSanitizer: stack-buffer-underflow /public/llvm/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_netbsd_syscalls.inc:394:3 in __sanitizer_syscall_pre_impl_recvmsg
Shadow bytes around the buggy address:
  0x4feffffd3800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x4feffffd3810: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x4feffffd3820: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x4feffffd3830: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x4feffffd3840: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=&gt;0x4feffffd3850: 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 f1[f1]
  0x4feffffd3860: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x4feffffd3870: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x4feffffd3880: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x4feffffd3890: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x4feffffd38a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==18015==ABORTING
</pre></code>
</blockquote>
<p>
<h1>NetBSD ioctl(2) hooks</h1>
<p>
Similar to the syscall hooks, there is need to handle every ioctl(2) call.
I've created the needed patch, this time shorter - for less than 300kb.
This code is still pending in upstream <a href="https://reviews.llvm.org/D41636">review</a>:
<p>
<blockquote>
Introduce handling of 1200 NetBSD specific ioctl(2) calls.<br>
Over 100 operations are disabled as unavailable or conflicting<br>
with the existing ones (the same operation number).
<p>
Add a script that generates the rules to detect ioctls on NetBSD.<br>
The generate_netbsd_ioctls.awk script has been written<br>
in NetBSD awk(1) (patched nawk) and is compatible with gawk.
<p>
Generate lib/sanitizer_common/sanitizer_netbsd_interceptors_ioctl.inc<br>
with the awk(1) script.
<p>
Update sanitizer_platform_limits_netbsd accordingly to add the needed<br>
definitions.
</blockquote>
<p>
<h1>New patches still pending for upstream review</h1>
<p>
There are two corrections that I've created, and they are still pending upstream for review:
<ul>
<li><a href="https://reviews.llvm.org/D41637">Add MSan interceptor for fstat(2)</a></li>
<li><a href="https://reviews.llvm.org/D41502">Correct the setitimer interceptor on NetBSD</a></li>
</ul>
<p>
I've got a few more local patches that require cleanup before submitting to review.
<p>
<h1>NetBSD basesystem corrections</h1>
<p>
I've introduced few corrections in the NetBSD codebase:
<ul>
<li>Rename of a local function uname() in ps(1) to usrname(). This removes a name clash with libc.
This situation is legal in the C language, but unwanted in the POSIX environment and it makes harder to reuse sanitizers with such code.</li>
<li>Removal of three unused and unimplemented legacy syscalls: sstk (stack section size change), sbrk (change data segment size) and vadvise (specify system paging behaviour).
The sbrk syscall could be removed as the functionality is implemented in libc brk(2) in assembly.</li>
<li>I've detected and corrected a bug in the pipe2() syscall, as it was returning incorrectly two integers instead of the one with the status of operation.
This was leftover from the pipe(2) system call behavior and implementation detail. I've introduced refactoring to address this and make the code cleaner.</li>
<li>Committed and fixed the jobs variable usage in sh(1) - by Christos Zoulas and K. Robert Elz - the first real (but harmless) bug detected and squashed thanks to MSan.</li>
</ul>
<p>
<h1>Sanitizers in Go</h1>
<p>
I've prepared a scratch <a href="https://netbsd.org/~kamil/llvm/0001-Add-initial-TSan-NetBSD-support.patch">port</a> of TSan and MSan to the Go language environment.
This code mostly works. However, there are remaining bugs that must be fixed.
<p>
Results of <code><a href="https://netbsd.org/~kamil/llvm/go-tsan-2017-12-06.txt">./race.bash</a></code>:
<pre>
Passed 340 of 347 tests (97.98%, 0+, 7-)
0 expected failures (0 has not fail)
</pre>
<p>
<h1>The MSan state as of today</h1>
<p>
Although, I've managed to pass more than 90% of tests in the check-msan target within approximately one week, the gap between passing most of the test and sanitizing
real world applications - even the small ones like cat(1) - is large.
This pushed me towards supporting most of the important NetBSD syscalls and NetBSD ioctls, and demands from me to support most of the libc, libutil, librt, libm and libkvm entry calls.
<p>
<pre>
********************
Testing: 0 .. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.. 
Testing Time: 27.92s
********************
Failing Tests (7):
    MemorySanitizer-X86_64 :: chained_origin_with_signals.cc
    MemorySanitizer-X86_64 :: dtls_test.c
    MemorySanitizer-X86_64 :: ioctl_custom.cc
    MemorySanitizer-X86_64 :: sem_getvalue.cc
    MemorySanitizer-X86_64 :: signal_stress_test.cc
    MemorySanitizer-X86_64 :: textdomain.cc
    MemorySanitizer-X86_64 :: tzset.cc

  Expected Passes    : 97
  Expected Failures  : 1
  Unsupported Tests  : 27
  Unexpected Failures: 7
</pre>
<p>
I can already execute cat(1) under sanitizers, and this milestone was achieved just at the end of the passed month.
<p>
Most other from the NetBSD base programs are not usable, few examples:
<p>
<ul>
<li>test(1) - bug with handling isspace(3), still not understood.</li>
<li>sh(1) - bug with signal-specific function calls.</li>
<li>ksh(1) - clash with a libc symbol.</li>
<li>ps(1) - lack of libkvm(3) handling.</li>
</ul>
<p>
There are also general stability problems with signals and forks.
This all makes MSan not ready for larger applications like LLDB.
<p>
<h1>Solaris support in sanitizers</h1>
<p>
I've helped the Solaris team to bring the basic support for Solaris in sanitizers.
I've helped the Solaris team add basic support for Sanitizers (ASan, UBsan).
This does not help NetBSD directly, however indirectly it improves the overall support for non-Linux hosts and helps to catch more Linuxisms in the code.
<h1>Plan for the next milestone</h1>
<p>
I plan to continue the work on MSan and correct sanitizing of the NetBSD basesystem utilities.
This mandates me to iterate over the basesystem libraries implementing the missing interceptors and correcting the current support of the existing ones.
My milestone is to build all src/*bin* programs against Memory Sanitizer and when possible execute them cleanly.
<p>
<h2>This work was sponsored by The NetBSD Foundation.</h2>
<p>
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:
<p>
<a href="http://netbsd.org/donations/#how-to-donate">http://netbsd.org/donations/#how-to-donate</a>