Struggling to fix a bohrbug in X86 Debug Registers
<p>
I've spent a month on fixes and debugging issues around the debugging facilities in the kernel.
<p>
<h1>Distribution cleanup</h1>
<p>
As planned in the previous month, I've performed cleanup in the distribution:
<p>
<ul>
<li>Removal of unneeded PT_SET_SIGMASK and PT_GET_SIGMASK, followed by pullup to NetBSD-8.</li>
<li>regnsub(3) and regasub(3) API change has been abandoned as it causes backward compatibility bootstrap from older distributions.
Changing it could require additional changes in libtre, but it has been abandoned too.
At the end I've finally removed just the USE_LIBTRE build option, replacing the libc's regex(3) implementation with an alternative
library from libtre/agrep, because it no longer builds as a regex-replacement and the upstream development stalled few years ago.</li>
<li>Backport of _UC_MACHINE_FP() to NetBSD-8 has been finished, this means that we can use the same code in NetBSD-8 and NetBSD-current in the sources of sanitizers</li>
</ul>
<p>
<h1>Improvements in the ATF ptrace(2) tests</h1>
<p>
I've performed the following operations in the ATF ptrace(2) and related tests,
in the following commit order:
<ul>
<li>Correct all ATF failures in t_ptrace_x86_wait.h (debug registers)</li>
<li>ATF: Correct a race bug in attach2 (t_ptrace_wait*)</li>
<li>ATF: Reenable attach2 in t_ptrace_wait*</li>
<li>Add a new function in ATF t_ptrace_wait*: await_zombie_raw()</li>
<li>ATF t_ptrace_wait*: Disable debug messages in msg.h</li>
<li>ATF: Add new test race1 in t_ptrace_wait*</li>
<li>Add new ATF tests: kernel/t_zombie</li>
</ul>
<p>
<h1>Zombie detection race</h1>
<p>
I've detected that the operation polling for a single process status - using sysctl(3) - can report the same process twice,
for the first time as an alive process and for the second time as a zombie.
This used to break the ATF ptrace(2) tests where we have a polling function, detecting the state change of
a dying process.
<p>
I've prepared an applied a fix for this case and the bug detected in ATF ptrace(2) tests is now gone.
I've included additional regression tests as noted below, to catch the race in a dedicated t_zombie test-suite.
<p>
There is also a controversy here, as it's not specified by POSIX what happens when we are polling an
unrelated process, that is not a child of its parent.
This operation can fail and we the previous approach we could report the same process twice, while
with the newer one reporting twice is much more unlikely on the cost of missing the process at times.
<p>
My preferrence is that a process between the state transition of alive->dying->dead and dead->zombie,
can disappear for a while.
I find this metaphor more natural rather than observing two entities that one is dying and the other is a zombie.
There is no perfect solution to this process watch and just following the narrow cases requested with POSIX seems
to be the proper solution and it de facto assumes such races.
<p>
<h1>Bohrbug in X86 Debug Registers</h1>
<p>
The release engineering machines were observing occasional failures with the X86 Debug Registers in the ptrace(2) ATF test-suite.
This was appearing once a while, approximately quarterly, on both amd64 and i386 machines.
<p>
I've missed them previously in my original work a year ago, as this happened to be caused by the fact that this bug
is not reproducible on newer (quicker? more cores?) Intel CPUs.
This failure was reproduced only in software-emulation (slow!) qemu and on Intel Core 2 Duo.
<p>
An attempt to execute the same X86 Debug Register test on Intel i7 for 100M times
does not make this crash to pop op even a single time.
<p>
I've spent the majority of the past month on researching this bug and it happened to be a bohrbug,
this means that adding almost any debug code anywhere makes it disappear completely or make reproducible
siginificantly less often.
<p>
It's not worth the space to describe the process of understanding this bug more closely,
step by step - it's worth to mention the current state of the understanding of mine.
We can observe a repeated syscall for the _lwp_kill(2) operation (executed with raise(SIGSTOP)),
with the same trap frame (or very similar).
I still don't how is it possible.
<p>
I've finally added support  to the NetBSD kernel - in my local copy of the sources - for Bochs-style debug
console/protocol (ISA port 0xe9).
This allows me to log internal qemu debug and NetBSD kernel messages into a single buffer and output to a single file.
I need this property as matching two logging buffers, when the logs of just the the interrupt frames in qemu can
quickly grow to hundreds of megabytes.
Assuming that I've reproduced the bohrbug after 20k executions of the same test and there is a lot of noise from unrelated
kernel/hardware traps, it's a useful property.
I'm planning to cleanup this code and submit to the NetBSD sources.
<p>
For completeness I had to patch the qemu isa-debugcon (Bochs-style debug console) source code to log into the same buffer
as the internal qemu debug messages...
sometimes bugs require non-conventional approaches to research them.
Ideally I would like to have a rewind/record feature in qemu together with the gdb-server emulation,
but we are still not there.
<p>
<h1>ptrace(2) status and plans</h1>
<p>
Now there is the X86 Debug Register race. Also, the following tests still marked as expected failures:
<p>
<ul>
<li>eventmask3</li>
<li>resume1</li>
<li>signal3, signal5, signal6, signal7, signal8, signal9</li>
<li>suspend1, suspend2</li>
<li>vfork1</li>
</ul>
<p>
This shows that the remaining major problems are in:
<ul>
<li>vfork(2)</li>
<li>signals</li>
<li>threads</li>
</ul>
<p>
Once I will squash the X86 Debug Register race,
I plan to work on these bugs in this oder: vfork handling bugs, signals and threads.
<p>
I have in mind addition of a lot of a lot of tests verifying correct signal processing in traced programs.
<p>
In this iteration of ptrace(2) hardening, I plan to skip Machine-Dependent extensions (like AVXv2 registers)
and extending core(5) files with additional features (for example we might want to know exact ABI or FPU hardware layout).
<p>
<h1>Plan for the next milestone</h1>
<p>
Keep working on the bug reproducible with X86 Debug Registers and switch to the remaining ptrace(2) failures.
<p>
<h2>This work was sponsored by The NetBSD Foundation.</h2>
<p>
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:
<p>
<a href="http://netbsd.org/donations/#how-to-donate">http://netbsd.org/donations/#how-to-donate</a>