Over two years ago, I made a pledge to use NetBSD as my sole OS and only operating system,
and to resist booting into <i>any</i> other OS until I had implemented hardware-accelerated
virtualization in the NetBSD kernel (the equivalent of Linux' KVM, or Hyper-V).
<p>
Today, I am here to report: <strong>Mission Accomplished!</strong>*
<p>
It's been a long road, but we now have hardware-accelerated virtualization in the kernel!
And while I had only initially planned to get Oracle VirtualBox working,
I have with the help of the Intel HAXM engine
(the same backend used for virtualization in Android Studio) and a qemu frontend,
successfully managed to boot a range of mainstream operating systems.
<p>
With the advent of Intel's open-sourcing of their HAXM engine,
we now have access to an important set of features:
<p>
<ul>
<li>A BSD-style license.
<li>Support for multiple platforms: Windows, Darwin, Linux, and now NetBSD .
<li>HAXM is an Intel hardware assisted virtualization for their CPUs (VTx and EPT needed).
<li>Support for an arbitrary number of concurrent VMs. For simplicity's sake, NetBSD only supports 8, whereas Windows/Darwin/Linux support 64.
<li>An arbitary number of supported VCPUS per VM. All OSes support up to 64 VCPUs.
<li>ioctl(2) based API (/dev/HAX, /dev/haxmvm/vmXX, /dev/haxmvm/haxmvmXXvcpuYY).
<li>Implement non-intrusively as part of the kernel, rather than as an out-of-tree, standalone executable kernel module.
<li>Default compatibility with qemu as a frontend.
<li>Active upstream support from Intel, which is driven by commercial needs.
<li>Optimized for desktop scenarios.
<li>Probably the only open-source cross-OS virtualization engine.
<li>An active and passionate community that's dedicated to keep improving it. 
</ul>
<p>
As well as a few of HAXM's downsides:
<ul>
<li>No AMD (SVM) support (althought there are community plans to implement it).
<li>No support for non-x86 architectures.
<li>Need for a relatively recent Intel CPU (EPT required).
<li>Not as flexible as KVM-like solutions for embedded use-cases or servers.
<li>Not as quick as KVM (probably 80% as fast as KVM).
</ul>
<p>
If you'd like more details on HAXM, check out the following sites:
<ul>
<li><a href="https://www.qemu.org/2017/11/22/haxm-usage-windows/">https://www.qemu.org/2017/11/22/haxm-usage-windows/</a>
<li><a href="https://software.intel.com/en-us/articles/intel-hardware-accelerated-execution-manager-intel-haxm">https://software.intel.com/en-us/articles/intel-hardware-accelerated-execution-manager-intel-haxm</a>
<li><a href="https://github.com/intel/haxm">https://github.com/intel/haxm</a>
</ul>
<p>
<h1>Showcase</h1>
<p>
I've managed to boot and run several operating systems as guests running on HAXM/NetBSD, all of which are multi-core (2-4 VCPUs):
<p>
<ul>
<li><strong>NetBSD</strong>.
And as our motto goes, <i>of course it runs NetBSD</i>.
And, of course, my main priority as a NetBSD developer is achieving excellent support for NetBSD guest operating systems.
With the massive performance gains of hardware-accelerated virtualization,
it'll finally be possible to run kernel fuzzers and many other practical, real-world workloads on virtualized NetBSD systems.
<p>
<img src="http://netbsd.org/~kamil/screenfetch/netbsd/screen_00021.png">
<br>
[NetBSD at the bootloader]
<p>
<img src="http://netbsd.org/~kamil/screenfetch/netbsd/screen_00031.png">
<br>
[NetBSD kernel booting]
<p>
<img src="http://netbsd.org/~kamil/screenfetch/netbsd/screen_00059.png">
<br>
[NetBSD at a shell prompt]
<p>
<img src="http://netbsd.org/~kamil/screenfetch/netbsd/screen_00169.png">
<br>
[NetBSD with X Window session]
<p>
<img src="http://netbsd.org/~kamil/screenfetch/netbsd/screen_00175.png">
<br>
[NetBSD with X Window session]
<p>
<img src="http://netbsd.org/~kamil/screenfetch/netbsd/screen_00251.png">
<br>
[NetBSD guest and qemu's ACPI integration - emitted poweroff button press]
<p>
<img src="http://netbsd.org/~kamil/screenfetch/netbsd/screen_00352.png">
<br>
[NetBSD in the qemu's curses mode display, which I find convenient to use, especially on a headless remote computer]
<p>
<li><strong>Linux</strong>. When I pledged myself to not boot any other OS before accomplishing my goal, I was mostly thinking about
resigning from benefits of the Linux ecosystem (driver and software base).
There is still a selection of programs that I miss such as valgrind.. but with each week we are getting closer to fill the missing gaps.
Linux guests seem to work, however there is need to tune or disable IOAPIC in order to get it running
(I had to pass "noapic" as a Linux kernel option).
<p>
<img src="http://netbsd.org/~kamil/screenfetch/archlinux-2018.11.01-x86_64/screen_00132.png">
<br>
[ArchLinux at a bootloader]
<p>
<img src="http://netbsd.org/~kamil/screenfetch/archlinux-2018.11.01-x86_64/screen_00900.png">
<br>
[ArchLinux at a shell (ZSH) prompt]
<p>
<img src="http://netbsd.org/~kamil/screenfetch/ubuntu-18.04.1-live-server-amd64/screen_00045.png">
<br>
[Ubuntu's installer]
<p>
<img src="http://netbsd.org/~kamil/screenfetch/ubuntu-18.04.1-live-server-amd64/screen_00104.png">
<br>
[Ubuntu at a bootloader]
<li><strong>Windows</strong>. While I have no personal need nor use-case to run Windows, it's a must-have prestigious target for virtualization.
I've obtained a Windows 7 86 trial image from the official Microsoft webpage for testing purposes.
Windows 8.1 or newer and 64-bit version is still in development in HAXM.
<p>
<img src="http://netbsd.org/~kamil/screenfetch/windows7_x86/crop_screen_00165.png">
<br>
[Windows 7 booting]
<p>
<img src="http://netbsd.org/~kamil/screenfetch/windows7_x86/crop_screen_00199.png">
<br>
[Windows 7 welcome message]
<p>
<img src="http://netbsd.org/~kamil/screenfetch/windows7_x86/crop_screen_00225.png">
<br>
[Windows 7 running]
<p>
<img src="http://netbsd.org/~kamil/screenfetch/windows7_x86/crop_screen_00241.png">
<br>
[Windows 7 Control Panel]
<p>
<img src="http://netbsd.org/~kamil/screenfetch/windows7_x86/crop_screen_00302.png">
<br>
[Windows 7 multitasking]
<p>
<img src="http://netbsd.org/~kamil/screenfetch/windows7_x86/crop_screen_00361.png">
<br>
[Windows 7 MS Paint]
<li><strong>DragonflyBSD</strong>. I was prompted to test this FreeBSD derivation by a developer of this OS and it just worked flawlessly.
<p>
<img src="http://netbsd.org/~kamil/screenfetch/DragonFly-x86_64-LATEST-IMG/screen_00057.png">
<br>
[DragonflyBSD at a bootloader]
<p>
<img src="http://netbsd.org/~kamil/screenfetch/DragonFly-x86_64-LATEST-IMG/screen_00276.png">
<br>
[DragonflyBSD at a shell prompt]
<li><strong>FreeDOS</strong>. It seems to just work, but I have no idea what I can use it for.
<p>
<img src="http://netbsd.org/~kamil/screenfetch/FD12LGCY/screen_00004.png">
<br>
[FREEDOS at an installer]
<p>
<img src="http://netbsd.org/~kamil/screenfetch/FD12LGCY/screen_00020.png">
<br>
[FREEDOS at a command line]
</ul>
Unfortunatelly not all Operating Systems are already supported. I've found issues with the following ones:
<ul>
<li><strong>Android</strong>. The kernel booting seems fine (with "noapic"), but later during distribution load it freezes the host computer.
<p>
<img src="http://netbsd.org/~kamil/screenfetch/android-x86_64-8.1-rc2/screen_00000.png">
<br>
[Android bootloader]
<p>
<img src="http://netbsd.org/~kamil/screenfetch/android-x86_64-8.1-rc2/screen_00110.png">
<br>
[Android just before the host crash]
<li><strong>FreeBSD</strong>. Hangs during the booting process.
<p>
<img src="http://netbsd.org/~kamil/screenfetch/FreeBSD-11.2-RELEASE-amd64/screen_00118.png">
<br>
[FreeBSD hanging at boot]
</ul>
<p>
<h1>Summary and future plans</h1>
<p>
One thing I must clarify, since I'm frequently asked about it,
is that HAXM/NetBSD <b>does not</b> attempt to compete with the NVMM
(NetBSD Virtual Machine Monitor) work being done by Maxime Villard.
I'm primarily doing this for my own educational purposes,
and because I find reaching feature-parity with other open-source projects is important
work. Additionally, NVMM only has AMD CPU support, whereas I'm primarily a user of the
Intel x86 platform, and thus, so is HAXM/NetBSD. The Intel port of NVMM and NVMM in general
is still in development, and this means that HAXM is probably the first solution that has ever
successfuly managed to run Windows on NetBSD (has anyone done it with Xen before?)
<p>
I will keep working on this project in my spare time and try to correct IOAPIC issues on Linux, hangs during FreeBSD's boot process, and Android crashes.
<p>
Most of the NetBSD-specific patches for qemu and Intel HAXM have already been merged upstream. And after this process has been completed, there are plans to make it available in pkgsrc. There's also at least one kernel-level workaround for HAXM behavior related to FPU state, which triggers an assert due to an abnormal condition. For this to be amended, fixes would have to land upstream into HAXM's code.
<p>
<h3>Subnote</h3>
<p>
<small>[*] Althought I confess that I've been playing with OpenVMS/VAX in SIMH as I have got a hobbyist license, but on the other hand it's hardly to
be treated as a competition to NetBSD. Another exception was during a DTrace tutorial during EuroBSDCon 2017 in Paris, where I had to boot
and use a FreeBSD image shared by the lecturer.</small>