Chapter 7. Creating binary packages for everything in pkgsrc (bulk builds)

Table of Contents

7.1. Preparations
7.2. Running a pbulk-style bulk build
7.2.1. Configuration
7.3. Requirements of a full bulk build
7.4. Creating a multiple CD-ROM packages collection
7.4.1. Example of cdpack

For a number of reasons you may want to build binary packages for a large selected set of packages in pkgsrc or even for all pkgsrc packages. For instance, when you have multiple machines that should run the same software, it is wasted time if they all build their packages themselves from source. Or you may want to build a list of packages you want and check them before deploying onto production system. There is a way of getting a set of binary packages: The bulk build system, or pbulk ("p" stands for "parallel). This chapter describes how to set it up so that the packages are most likely to be usable later.

7.1. Preparations

First of all, you have to decide whether you build all packages or a limited set of them. Full bulk builds usually consume a lot more resources, both space and time, than builds for some practical sets of packages. There exists a number of particularly heavy packages that are not actually interesting to a wide audience. For a limited bulk builds you need to make a list of packages you want to build. Note, that all their dependencies will be built, so you don't need to track them manually.

During bulk builds various packages are installed and deinstalled in /usr/pkg (or whatever LOCALBASE is), so make sure that you don't need any package during the builds. Essentially, you should provide fresh system, either a chroot environment or something even more restrictive, depending on what the operating system provides, or dedicate the whole physical machine. As a useful side effect this makes sure that bulk builds cannot break anything in your system. There have been numerous cases where certain packages tried to install files outside the LOCALBASE or wanted to edit some files in /etc.

7.2. Running a pbulk-style bulk build

Running a pbulk-style bulk build works roughly as follows:

  • First, build the pbulk infrastructure in a fresh pkgsrc location.

  • Then, build each of the packages from a clean installation directory using the infrastructure.

7.2.1. Configuration

To simplify configuration we provide helper script mk/pbulk/pbulk.sh.

In order to use it, prepare a clear system (real one, chroot environment, jail, zone, virtual machine). Configure network access to fetch distribution files. Create user with name "pbulk".

Fetch and extract pkgsrc. Use a command like one of these:

# (cd /usr && ftp -o - http://ftp.NetBSD.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-)
# (cd /usr && fetch -o - http://ftp.NetBSD.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-)
# (cd /usr && cvs -Q -z3 -d anoncvs@anoncvs.NetBSD.org:/cvsroot get -P pkgsrc)

Or any other way that fits (e.g., curl, wget).

Deploy and configure pbulk tools, e.g.:

# sh pbulk.sh -n # native (NetBSD)
# sh pbulk.sh -n -c mk.conf.frag # native, apply settings from given mk.conf fragment
# sh pbulk.sh -nlc mk.conf.frag # native, apply settings, configure for limited build

Note

mk.conf.frag is a fragment of mk.conf that contains settings you want to apply to packages you build. For instance,

PKG_DEVELOPER=		yes		# perform more checks
X11_TYPE=		modular		# use pkgsrc X11
SKIP_LICENSE_CHECK=	yes		# accept all licences (useful when building all packages)

If configured for limited list, replace the list in /usr/pbulk/etc/pbulk.list with your list of packages one per line without empty lines or comments. E.g.:

www/firefox
mail/thunderbird
misc/libreoffice4

At this point you can also review configuration in /usr/pbulk/etc and make final amendments, if wanted.

Start it:

# /usr/pbulk/bin/bulkbuild

After it finishes, you'll have /mnt filled with distribution files, binary packages, and reports, plain text summary in /mnt/bulklog/meta/report.txt

Note

The pbulk.sh script does not cover all possible use cases. While being ready to run, it serves as a good starting point to understand and build more complex setups. The script is kept small enough for better understanding.

Note

The pbulk.sh script supports running unprivileged bulk build and helps configuring distributed bulk builds.

7.3. Requirements of a full bulk build

A complete bulk build requires lots of disk space. Some of the disk space can be read-only, some other must be writable. Some can be on remote filesystems (such as NFS) and some should be local. Some can be temporary filesystems, others must survive a sudden reboot.

  • 40 GB for the distfiles (read-write, remote, temporary)

  • 30 GB for the binary packages (read-write, remote, permanent)

  • 1 GB for the pkgsrc tree (read-only, remote, permanent)

  • 5 GB for LOCALBASE (read-write, local, temporary)

  • 10 GB for the log files (read-write, remote, permanent)

  • 5 GB for temporary files (read-write, local, temporary)

7.4. Creating a multiple CD-ROM packages collection

After your pkgsrc bulk-build has completed, you may wish to create a CD-ROM set of the resulting binary packages to assist in installing packages on other machines. The pkgtools/cdpack package provides a simple tool for creating the ISO 9660 images. cdpack arranges the packages on the CD-ROMs in a way that keeps all the dependencies for a given package on the same CD as that package.

7.4.1. Example of cdpack

Complete documentation for cdpack is found in the cdpack(1) man page. The following short example assumes that the binary packages are left in /usr/pkgsrc/packages/All and that sufficient disk space exists in /u2 to hold the ISO 9660 images.

# mkdir /u2/images
# pkg_add /usr/pkgsrc/packages/All/cdpack
# cdpack /usr/pkgsrc/packages/All /u2/images
      

If you wish to include a common set of files (COPYRIGHT, README, etc.) on each CD in the collection, then you need to create a directory which contains these files. e.g.

# mkdir /tmp/common
# echo "This is a README" > /tmp/common/README
# echo "Another file" > /tmp/common/COPYING
# mkdir /tmp/common/bin
# echo "#!/bin/sh" > /tmp/common/bin/myscript
# echo "echo Hello world" >> /tmp/common/bin/myscript
# chmod 755 /tmp/common/bin/myscript
      

Now create the images:

# cdpack -x /tmp/common /usr/pkgsrc/packages/All /u2/images

Each image will contain README, COPYING, and bin/myscript in their root directories.