GSoC 2016 Reports: Split debug symbols for pkgsrc builds, part 2

Google Summer of Code 2016 is now over. We have polished the code and documentation and submitted the final term evaluation on 23rd of August 2016. The mentors evaluated us in the following week.

If you are impatient (and this time impatience is definitely a virtue!) please take a look to all The NetBSD Foundation GSoC 2016 projects' code submissions!

Introduction

In part 1 we have learned what happens under the hood when we split debugging symbols: how debugging information is stored/stripped off, the relevant ELF sections involved, various tools to dump that information, etc..

In particular, we have studied what happens on NetBSD when we set the MKDEBUG* flags.

With this background we can finally try to implement this functionality in the pkgsrc infrastructure. In this blog post we will first give a practical look at PKG_DEBUGDATA and then we will analyze the code that I have written in the first mid-term of the GSoC to accomplish that.

PKG_DEBUGDATA in action

Before digging in the implementation of debugdata functionality let's see what it produces!

In part 1 we took as example a very simple program that just prints out the lyrics of Ten Green Bottles song. After packaging it as local/green-bottles let's just build it without any special pkgsrc system variables set: 

$ cd pkgsrc/local/green-bottles
$ make install
[...]
$ green-bottles
ten green bottles hanging on the wall
[...]
$ pkg_info -L green-bottles
Information for green-bottles-0:

Files:
/usr/pkg/bin/green-bottles
$ gdb `which green-bottles`
GNU gdb (GDB) 7.10.1
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
[...]
Reading symbols from /usr/pkg/bin/green-bottles...(no debugging symbols found)...done.
(gdb) quit

That's expected because we did not set any special CFLAGS nor INSTALL_UNSTRIPPED.

Before setting PKG_DEBUGDATA to "yes" let's inspect its documentation via the help target: 

$ make help topic=PKG_DEBUGDATA
===> mk/bsd.debugdata.mk (keywords: INSTALL_UNSTRIPPED PKG_DEBUGLEVEL PKG_DEBUGDATA debug):
# This Makefile fragment is included by bsd.pkg.mk and implements the
# logic needed to strip debug symbols off the program/libraries.
#
# User-settable variables:
#
# PKG_DEBUGDATA
#       If "yes", install stripped debug symbols for all programs and shared
#       libraries. Please notice that if it is defined INSTALL_UNSTRIPPED will
#       be also defined internally.
#
# PKG_DEBUGLEVEL
#       Used to control the granularity of the debug information. Can be
#       "small", "default", "detailed".
#       TODO: the name of this variable should be changed because it can be
#       TODO: easily confused with PKG_DEBUG_LEVEL!
#
# See also:
#       INSTALL_UNSTRIPPED
#

Now that we have finally an idea of what PKG_DEBUGDATA does, let's set it and reinstall the green-bottles package: 

$ make clean deinstall
[...]
$ PKG_DEBUGDATA=yes make install
[...]
===> Installing for green-bottles-0
[...]
=> Stripping debug symbols
=> Installing debug data
[...]
=> Checking for missing debug data in green-bottles-0
[...]

Apart from the various phases we can see three extra messages that are now printed out: stripping of debug symbols, installing of debug data and a check for missing debug data.

If we inspect the files installed we can now also see green-bottles.debug: 

$ pkg_info -L green-bottles
Information for green-bottles-0:

Files:
/usr/pkg/bin/green-bottles.debug
/usr/pkg/bin/green-bottles

Indeed if we now try to run it through gdb(1): 

$ gdb `which green-bottles`
GNU gdb (GDB) 7.10.1
Copyright (C) 2015 Free Software Foundation, Inc.
[...]
Reading symbols from /usr/pkg/bin/green-bottles...Reading symbols from /usr/pkg/bin/green-bottles.debug...done.
done.
(gdb) b main
Breakpoint 1 at 0xad0: file green-bottles.c, line 29.
(gdb) run
Starting program: /usr/pkg/bin/green-bottles

Breakpoint 1, main () at green-bottles.c:29
29      {
(gdb)
[...]

green-bottles was installed as usual without requiring any changes from MAINTAINERs, in fact the various .debug files are generated, installed and added to the PLIST dynamically.

A look at mk/bsd.debugdata.mk and mk/check/check-debugdata.mk implementation

All the functionalities that implement stripping of the debug data from installed programs and libraries are implemented entirely in bsd.debugdata.mk.

check/check-debugdata.mk checks that all programs and libraries have the debug data correctly stripped into their corresponding *.debug files.

mk/bsd.debugdata.mk

The first 23 lines of bsd.debugdata.mk contain a comment that is also accessible via the help target. In particular it describes all the user-settable variables:

Then _VARGROUPS and _USER_VARS.<vargroup> are set accordingly. These are used by the show-all target (if you are more curious regarding that please just try make show-all and/or give a look to mk/misc/show.mk).

After various definitions of some used variables (should be self-explainable) _FIND_DEBUGGABLEDATA_ERE is defined as an Extended Regular Expression that is used to match potential files that can be stripped. _FIND_DEBUGGABLEDATA_ERE is then used to limit the file list generated via _FIND_DEBUGGABLEDATA_FILELIST_CMD

Lines 63-86 pass the appropriate debug flags and debug level to the compiler.

Then _PLIST_DEBUGDATA is added to the PLIST_SRC_DFLT variable. PLIST_SRC_DFLT contains all the default PLISTs that usually resides in pkgsrc/category/PLIST* (e.g. PLIST, PLIST.common, PLIST.NetBSD, etc.). If you are more interested in how it works, please give a look to mk/plist/plist.mk. In this way all the *.debug files listed in _PLIST_DEBUGDATA are dynamically appended to the package PLIST.

In lines 90-111 generate-strip-debugdata target is defined. It basically just does the operations we have explored in part 1 with little adjustments for pkgsrc:

To accomplish that RUN is used to run all shell commands without printing them. Please note that when RUN is used all shell commands should be terminated with a semicolon. For more information regarding RUN please take a look at make help topic=RUN.

In lines 113-122 install-strip-debugdata is defined. It just installs all *.debug files via INSTALL_DATA. install-strip-debugdata target will then be invoked after the post-install phase.

mk/check/check-debugdata.mk

First 26 lines of check-debugdata.mk contain a comment for the pkgsrc online documentation. It describes the following user-settable variables:

...and the following package-settable variables:

Like what it was done for bsd.debugdata.mk, _VARGROUPS and _{USER,PKG}_VARS.check-debugdata are defined accordingly.

Similarly to what was done for _FIND_DEBUGGABLEDATA_ERE and _FIND_DEBUGGABLEDATA_FILELIST_CMD in bsd.debugdata.mk, _CHECK_DEBUGDATA_ERE and _CHECK_DEBUGDATA_FILELIST_CMD are defined.

In lines 54-103 _check-debugdata target is defined. _check-debugdata performs the following checks, respectively:

Other minor changes in mk/*

In order to instruct pkgsrc to pick up bsd.debugdata.mk and checks/check-debugdata.mk, nothing intrusive though:

Conclusion

In this blog post we have learned what PKG_DEBUGDATA does via a practical example.

Then we have examinated how bsd.debugdata.mk and checks/check-debugdata.mk are implemented.

PKG_DEBUGDATA is completely agnostic if a package uses GNU configure, cmake, etc., and it is expected to work without any changes. I have only tested it with simple packages also containing libraries and the ones that uses libtool but I have still not tested it in the wild and some minor adjustements can be probably needed.

Thanks again to Google for organizing Google Summer of Code, The NetBSD Foundation and in particular my mentors David Maxwell, Jöerg Sonnenberger, Taylor R. Campbell, Thomas Klausner and William J. Coldwell.

References