Skip to main content.

Making a binary package repository with poudriere

2013-09-11

Live demo in BSD Now Episode 002 | Originally written by TJ for bsdnow.tv | Last updated: 2015/01/24

NOTE: the author/maintainer of the tutorial(s) is no longer with the show, so the information below may be outdated or incorrect.

While using the extremely powerful and flexible ports collection is the traditional BSD way of installing software, fetching binary packages is also an option. Binary packages are just precompiled ports. Most people switching from other operating systems aren't used to using ports, and binary packages can make the upgrade to BSD a bit more comfortable for them at first. They're a quick way to get a new system up and running, and can be a very appealing option when deploying a lot of systems at once. This tutorial will teach you how to configure your own binary package building server and how to use it to distribute packages to your client machines. It’s recommended to do this on a system with a fast CPU and decent amount of RAM. We’ll be using the pkgng system and a mass port building tool called poudriere.

So, what exactly is poudriere? To quote the documentation: “poudriere is a BSD-2 licensed tool primarily designed to test package production on FreeBSD. However, most people will find it useful to bulk build ports for FreeBSD. Its goals are to use modern facilities present in FreeBSD, to be easy to use, to depend only on base, and to be parallel.”

We’re going to install it on the building machine, tell it which ports we want to build and install a webserver to distribute the files. This assumes you already have a FreeBSD (8.3 or newer) system installed with a current ports tree. Let’s start by installing the tool and getting some configuration files in place.

# cd /usr/ports/ports-mgmt/poudriere
# make install clean
# cp /usr/local/etc/poudriere.conf.sample /usr/local/etc/poudriere.conf
# vi /usr/local/etc/poudriere.conf

We’ll use the following:

## If you have a ZFS pool named tank, uncomment this
#ZPOOL=tank
## If you are only using UFS, uncomment this
#NO_ZFS=yes
FREEBSD_HOST=ftp://ftp.freebsd.org
RESOLV_CONF=/etc/resolv.conf
BASEFS=/usr/local/poudriere
USE_TMPFS=yes
DISTFILES_CACHE=/usr/ports/distfiles
USE_COLORS=no
POUDRIERE_DATA=${BASEFS}/data
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes
PKG_REPO_SIGNING_KEY=/usr/local/etc/ssl/keys/pkg.key
WRKDIR_ARCHIVE_FORMAT=txz
NOLINUX=yes

Now we check out a fresh copy of the ports tree for poudriere to use.

# poudriere ports -c

Create a jail with the version of FreeBSD for which you want to build the packages. In this example, I’ll be compiling them for 9.1-RELEASE systems. You can also do -STABLE or -CURRENT jails by grabbing them from SVN and running buildworld. In my case, I’m on the x86_64 architecture and will name my jail “91x64.”

# poudriere jail -c -j 91x64 -v 9.1-RELEASE -a amd64

The jail can be updated with freebsd-update by using the following command. I’ll go ahead and update mine.

# poudriere jail -u -j 91x64

Next we’ll create a make.conf file for the jail that tells it any specific options we want built for our packages. This will vary HIGHLY depending on your needs, so don’t blindly copy and paste this. Use it as a foundation and make changes to fit your specific situation.

# vi /usr/local/etc/poudriere.d/91x64-make.conf

Mine consists of:

WITH_PKGNG=yes       # Only required for versions before 10.0
CPUTYPE?=atom        # Example, for an Atom CPU
CC=clang             # Highly recommended over GCC,
CXX=clang++          # but only needed for 8.X and 9.X
CPP=clang-cpp        # since it's the default in 10.0
FETCH_BEFORE_ARGS=-p4 -T 10
MASTER_SITE_BACKUP?= \
http://ftp2.us.freebsd.org/pub/FreeBSD/ports/distfiles/${DIST_SUBDIR/}
OPTIONS_UNSET= DEBUG HELP STATIC GNUTLS DOCS EXAMPLES IPV6 \
           MANPAGES PTH IDN LIBIDN NLS DBUS SOUND ALSA PULSEAUDIO \
           DOCBOOK CUPS TESTS HTMLDOCS BONJOUR GSSAPI APIDOCS

Next we’ll create a list of ports that we want this box to compile for us.

# vi /usr/local/etc/poudriere-list

The syntax is very simple:

www/firefox
net-p2p/rtorrent
games/cowsay
irc/irssi
sysutils/tmux

Dependencies will be pulled in automatically, so don’t worry about them. Now we tell poudriere to build the ports we listed. If you want to export a list of already-installed ports on a system, you can generate the file like so by using portmaster:

# portmaster --list-origins | sort -d > /usr/local/etc/poudriere-list

If you want to overwrite the options a specific port is built with, including its dependencies, you can use something like:

# poudriere options -c www/firefox

Or if you want to configure all the options all the ports will be built with:

# poudriere options -cf /usr/local/etc/poudriere-list

Build time! If you intend to sign your repo with an RSA key, skip to the next section before beginning the bulk build and come back.

# poudriere bulk -f /usr/local/etc/poudriere-list -j 91x64

Or, if you want to build the entire ports tree (which is over 24,000 applications as of the time of this writing),

# poudriere bulk -a -j 91x64

Your binaries should end up in ${POUDRIERE_DATA}/packages/91x64-default/. If you want to easily distribute them to other systems, you can setup a webserver (www/nginx or www/lighttpd) or an FTP server (ftp/vsftpd) to point to this directory. There’s even a very useful JSON-based web frontend to poudriere that’s included. Point your webserver or FTP server to show /usr/local/poudriere/data/logs/bulk/91x64/latest and take a look. This is an easy way to monitor the status of bulk port builds without looking at the terminal.

You’ll want to keep the package repo up to date. To do so, run the following commands:

# poudriere ports -u
# poudriere bulk -f /usr/local/etc/poudriere-list -j 91x64

It is possible (and advised) to add an RSA key for package authentication before building.

# mkdir -p /usr/local/etc/ssl/keys /usr/local/etc/ssl/certs
# chmod 600 /usr/local/etc/ssl/keys
# openssl genrsa -out /usr/local/etc/ssl/keys/pkg.key 4096
# openssl rsa -in /usr/local/etc/ssl/keys/pkg.key -pubout > /usr/local/etc/ssl/certs/pkg.cert

Be sure to copy the pkg.cert file to your client systems via a secure method like SCP or sneakernet. Now we move over to those client systems and set the appropriate pkg config options to download from the server you (hopefully) setup.

# mkdir -p /usr/local/etc/pkg/repos
# vi /usr/local/etc/pkg/repos/poudriere.conf

Add some settings:

poudriere: {
  url: "http://your-web-server/path/to/repo",
  mirror_type: "http",
  signature_type: "pubkey",
  pubkey: "/usr/local/etc/ssl/certs/pkg.cert",
  enabled: yes
}

Update the repo info:

# pkg update

From there, you should be able to install the authenticated binary packages. For more information and options, see our pkgng tutorial.

Latest News

Two Year Anniversary

2015-08-08

We're quickly approaching our two-year anniversary, which will be on episode 105. To celebrate, we've created a unique t-shirt design, available for purchase until the end of August. Shirts will be shipped out around September 1st. Most of the proceeds will support the show, and specifically allow us to buy...

New discussion segment

2015-01-17

We're thinking about adding a new segment to the show where we discuss a topic that the listeners suggest. It's meant to be informative like a tutorial, but more of a "free discussion" format. If you have any subjects you want us to explore, or even just a good name...

How did you get into BSD?

2014-11-26

We've got a fun idea for the holidays this year: just like we ask during the interviews, we want to hear how all the viewers and listeners first got into BSD. Email us your story, either written or a video version, and we'll read and play some of them for...

EuroBSDCon 2014

2014-09-18

As you might expect, both Allan and Kris will be at EuroBSDCon this year. They'll be busy hunting down various BSD developers and forcing them to do interviews, but don't hesitate to say hi if you're a listener!...


Episode 170: Sandboxing Cohabitation

2016-11-30

Direct Download:VideoHD VideoMP3 AudioOGG AudioTorrent This episode was brought to you by Headlines EuroBSDcon 2016 Presentation Slides Due to circumstances beyond the control of the organizers of EuroBSDCon, there were not recordings of the talks given at the event. However, they have collected the slide decks from each of the speakers and assembled them on...

Episode 169: Scheduling your NetBSD

2016-11-23

Direct Download:VideoHD VideoMP3 AudioOGG AudioTorrent This episode was brought to you by Headlines Production ready Ted Unangst brings us a piece on what it means to be Production Ready He tells the story of a project he worked on that picked a framework that was “production ready” They tested time zones, and it all seemed to...

Episode 168: The Post Show Show

2016-11-16

Direct Download:VideoHD VideoMP3 AudioOGG AudioTorrent This episode was brought to you by Headlines Build a FreeBSD 11.0-release Openstack Image with bsd-cloudinit We are going to prepare a FreeBSD image for Openstack deployment. We do this by creating a FreeBSD 11.0-RELEASE instance, installing it and converting it using bsd-cloudinit. We'll use the CloudVPS...

Episode 167: Playing the Long Game

2016-11-09

Direct Download:VideoHD VideoMP3 AudioOGG AudioTorrent This episode was brought to you by Interview - Scott Long - scottl@freebsd.org FreeBSD & Netflix Feedback/Questions Zack - USB Config Jens - VMs, Jails and Containers Ranko - Tarsnap Keys Alex - OpenBSD in Hyper-V Curt - Discussion Segment Send questions, comments, show ideas/topics, or stories you want mentioned...