LLVM asan and ubsan on NetBSD

Over the last 30 days I was focusing on getting the environment to enable LLVM sanitizers and the Clang compiler on NetBSD. Meanwhile I pushed forward generic parts that were needing enhancements around pkgsrc and LLVM in general to ease the future LLDB work.


When I have realized that in order to work on the LLVM sanitizers I need to use Clang as the compiler. A part of the compiler-rt (lowlevel LLVM compiler runtime library) has code specifically incompatible with GCC. It was mainly related to intrinsic instructions for atomic functions. I tried to research how much work is needed to port it to GCC. It happened to be non-trivial and I filed a bug on the LLVM bugzilla.

These circumstances made me to switch to Clang as the pkgsrc toolchain. I was using it to test the compilation of small bulks of packages and record build and compiler problems. To save time, I used ccache as my cache for builds.

My options in mk.conf(5):

PKGSRC_COMPILER=        ccache clang
CCACHE_BASE=            /usr/local
CCACHE_DIR=             /public/ccache_tmp
CCACHE_LOGFILE=         /tmp/ccache.txt
PKG_CC=                 clang
PKG_CXX=                clang++
CLANGBASE=              /usr/local
HAVE_LLVM=              yes

It's worth noting that ccache with pkgsrc won't check $HOME/.ccache for configuration, it must be placed in $CCACHE_DIR/ccache.conf.

The documented problems can be summarized as:

LLVM and Clang testsuites

I have prepared Clang and LLVM testsuites to execute on NetBSD. Correctness of both projects is crucial for LLDB and the LLVM sanitizers to work because their issues resound problems inside programs that depend on them. Originally I have corrected the tests with local patches to build with GCC, and switched later to Clang. I have restructured the packages in pkgsrc-wip in order to execute the test-suite. I have fixed 20 test failures in LLVM implementing AllocateRWX and ReleaseRWX for the NetBSD flavor of PaX MPROTECT. There are still over 200 failures to solve!

It's worth noting that the googletest library (used in a modified version in LLVM and in a regular one in Clang) finally accepted the NetBSD patches.

LLVM asan and ubsan

I expect to get four LLVM sanitizers working in order to move on to LLDB: asan (address sanitizer), ubsan (undefined behavior sanitizer), tsan (thread sanitizer), msan (memory sanitizer). The other ones like dfsan (data-flow sanitizer) or lsan (leak sanitizer) are currently to be skipped. In general, sanitizers are part of the LLDB functionality that I want to get aboard on NetBSD, as there are plugins to integrate them within the debugger. In the current state I require them to debug bugs inside LLDB/NetBSD.

The original work on sanitizers in GCC (with libsanitizer) has been done by Christos Zoulas. GCC libsanitizer is a close sibling of compiler-rt/lib from the LLVM project. I picked up his work and integrated it into compiler-rt and developed the rest (code differences, fixing bugs, Clang/LLVM specific parts in llvm/ and clang/) and I managed to get asan and ubsan to work.

Users should pickup pkgsrc-wip in revision 3e7c52b97b4d6cb8ea69a081409ac818c812c34a and install wip/{llvm,clang,compiler-rt}-netbsd. Clang will be ready for usage:

/usr/pkg/bin/clang -fsanitize=undefined test.c
/usr/pkg/bin/clang -fsanitize=address test.c

Additional compiler commands that may improve the experience:

-g -O0 -fno-omit-frame-pointer -pie -fPIE

Plan for the next milestone

Roadmap for the next month:

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can: