Finishing leftover tasks from Google Summer of Code
<p>
Over the past month, I was coordinating and coding the remaining post-GSoC tasks.
This mostly covers work around honggfuzz and sanitizers.
<p>
<h1>honggfuzz ptrace(2) features</h1>
<p>
I've introduced new ptrace(2) tests verifying attaching to a stopped process.
This is an important scenario in debuggers, the ability to call a ptrace(2)
operation with the PT_ATTACH argument with a process id (PID) of a process entity that is stopped.
In typical circumstances PT_ATTACH causes an executing process to stop and emit SIGSTOP for the tracer.
An already stopped process is a special case as we cannot stop it again.
Not every UNIX-like kernel can handle this scenario in a sensible way, and the modern solution is to keep the process stopped
(rather than e.g... resumed) and emit a new signal SIGSTOP to the debugger (rather than e.g. not emitting anything).
There used to be complex workarounds for mainstream kernels in debuggers such as GDB to workaround kernel bugs
of attaching to a stopped process.
<p>
honggfuzz is a security oriented, feedback-driven, evolutionary, easy-to-use fuzzer with interesting analysis options.
This piece of software is developed by a Google employee, however the product is not an official Google software.
honggfuzz uses on features platforms ptrace(2) to monitor crash signals in traced processes.
I've implemented a new backed in the fuzzer for NetBSD using its ptrace(2) API.
The backend is designed to follow the existing scenarios and features in Linux & Android:
<ul>
<li>Attaching to a manually stopped process, optionally spawned by the fuzzer.</li>
<li>Option to attach to a selected process id over PID.</li>
<li>Crash instruction decoding with aid of a disassembler (capstone for NetBSD).</li>
<li>Ability to monitor multiple processes with an atribtrary number of threads.</li>
<li>Monitor forks(2) and vforks(2) events, however unused in the current fuzzing model.</li>
<li>Concurrent execution of multiple processes.</li>
<li>Timout of longlasting (hanging?) processes in persistent mode (0.25[sec]).</li>
</ul>
<p>
There are few missing features:
<ul>
<li>Intel BTS - hardware assisted tracing</li>
<li>Intel PT - hardware assisted tracing</li>
<li>hp libunwind - unwinding stack for better detection of unique crashes</li>
</ul>
<p>
<h1>Sanitizers</h1>
<p>
I've started researching Kernel Address Sanitizer, checking the runtime internals and differences between its version ABIs.
My intention was to join efforts with Siddharth (GSoC student) and head with a sanitzier for EuroBSDCon 2018 in Romania.
However, Maxime Villard decided to join the efforts a little bit earlier and he managed to get quickly a funtional bare version for NetBSD/amd64.
In the end we have decided to leave the kASan work to Maxime for now and let Siddharth to work on a kCov (SanitizerCoverage) device.
SanCov is a feature of compilers, designed as an aid for fuzzers to ship interesting information from a fuzzing point of view of a number of
function calls, comparisons, divisions etc.
Successful userland fuzzers (such as AFL, honggfuzz) use this feature as an aid in determining of new code-paths.
It's the same with a renowed kernel fuzzer - syzkaller.
<p>
While, I'm helping Siddharth to port a kcov(4) device to NetBSD, I've switched to the remaining pending tasks in userland sanitizers.
I've managed to switch the sanitizers from syscall(2) and __syscall(2) - indirect system call API - calls to libc routines.
The approach of using an indirect generic interface didn't work well in the NetBSD case, as there is the need to handle multiple ABIs,
Endians, CPU architectures, and the C language ABI is not a good choice to serialize and deserialize arbitrary function arguments
with various types through a generic interface.
The discission on the rationale is perhaps not the proper place, and every low-level C developer is well aware of the problems.
It's better to restrict the discussion to the statement that it's not possible (not trivial) to call in a portable way
all the needed syscalls, without the aid of per-case auxilary switches and macros.
There are also some cases (such as pipe(2)) when is is not possible to express the system call semantics
with syscall(2)/__syscall(2).
<p>
I've switched these routines to use internal libc symbols when possible.
In the remaining cases I've used a fallback to libc's versions of the routines, with aid of indirect function pointers.
I'm trying to detect the addresses of real functions with dlsym(3) calls.
In the result, I've switched all the uses of syscall(2) and __syscall(2) and observed no regressions in tests.
<p>
I'm also in the process of deduplication of local patches to sanitizers.
My current main focus is to finish switching syscall(2) and __syscall(2) to libc routines (patch pending upstream),
introduce a new internal version of sysctl(3) that bypasses interceptors (partially merged upstream) and
introduces new interceptors for sysctl(3) calls.
This is a convoluted process in the internals with the goal to make the sanitizers more reliable across NetBSD targets and manage to sanitize
less trivial examples such as rumpkernels.
The RUMP code uses internally a modified and private versions of sysctl*() operations and we still must keep the internals in order and properly
handl the RUMP code.
<p>
<h1>Merged commits</h1>
<p>
The NetBSD sources:
<p>
<ul>
<li>Merge FreeBSD improvements to the man-page of timespec_get(3)</li>
<li>Remove unused symbols from sys/sysctl.h</li>
<li>Add a new ATF ptrace(2) test: child_attach_to_its_stopped_parent</li>
<li>Add await_stopped() in t_ptrace_wait.h</li>
<li>Add a new ATF test parent_attach_to_its_stopped_child</li>
<li>Add a new ATF ptrace(2) test: tracer_attach_to_unrelated_stopped_process</li>
<li>Drop a duplicate instruction line [libpthread]</li>
<li>Mark kernel-asan as done (by maxv)</li>
<li>TODO.sanitizers: Mark switch of syscall(2)/__syscall(2) to libc done</li>
</ul>
<p>
The LLVM sources:
<p>
<ul>
<li>Introduce new type for inteceptors UINTMAX_T</li>
<li>Add internal_sysctl() used by FreeBSD, NetBSD, OpenBSD and MacOSX</li>
<li>Improve portability of internal_sysctl()</li>
<li>Try to fix internal_sysctl() for MacOSX</li>
<li>Try to unbreak internal_sysctl() for MacOSX</li>
</ul>
<p>
<h1>Summary</h1>
<p>
I'm personally proud of the success of the reliability of the ptrace(2) backend in the renowed honggfuzz fuzzer.
NetBSD was capable to handling all the needed features and support all of them with an issue-free manner.
Once, I will address the remaining ptrace(2) issues on my TODO list - the NetBSD kernel will be capable to host more software
in a similar fashion, and most importantly a fully featured debugger such as GDB and LLDB, however without the remaining hiccups.
<p>
We are also approaching another milestone with the sanitizers' runtime available in the compiler toolchain: sanitizing rumpkernels.
It is already possible to execute the rump code against a homegrown uUBSan runtime, but we are heading now to execute the code under the default
runtime for the remaining sanitizers (ASan, MSan, TSan).
<p>
For the record, it has been reported that kUBSan has been ported from NetBSD to at least two kernels: FreeBSD and XNU.
<p>
<h1>Plan for the next milestone</h1>
<p>
I'm in preparation for my visit to EuroBSDCon (Bucharest, Romania) in September and GSoC Mentor Summit & MeetBSDCa in October (California, the U.S.).
I intend to rest during this month and still provide added value to the project, porting and researching missing software dedicated for developers.
Among others, I'm planning to research the HP libunwind library and if possible, port it to NetBSD.
<p>
<h2>This work was sponsored by The NetBSD Foundation.</h2>
<p>
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:
<p>
<a href="http://netbsd.org/donations/#how-to-donate">http://netbsd.org/donations/#how-to-donate</a>