Skip to main content.

The ultimate OpenBSD router

2013-11-13

Live demo in BSD Now Episode 011 | Originally written by TJ for bsdnow.tv | Last updated: 2014/11/30

"Friends don't let friends use consumer networking equipment."

This is a saying that many sysadmins and BSD fans have probably heard at least once before. Consumer grade routers - yes, you know the ones - are particularly bad. They're proprietary, have security issues and offer very little flexibility. Why would you let something like that sit at the core of your network? Often times, these routers are severely limited in functionality because they're aimed at consumers who aren't very tech-savvy. We're going to show you how to build your own router, based on OpenBSD and PF, and take back control of your network!


Hardware

This is a list of hardware I'll be using (although nothing in this tutorial is specific to it).

Buy whatever hardware you want, just make sure the network cards are supported beforehand. The Soekris board I chose uses Intel NICs that are known to have good BSD support.


Background

First, let's define what a router is, since everyone has different requirements. I've got three computers that need to share my internet connection. One of them is a server that I'd like to be able to SSH into remotely, but otherwise I don't want any of the systems exposed to the internet. The router will be doing the following things:

  • Performing Network Address Translation
  • Giving my server "meimei" and my laptop "suigintou" static IPs, based on their MAC address
  • Handing out IP addresses via DHCP to everyone else
  • Doing local DNS caching for the LAN and encrypting all outgoing DNS lookups
  • Allowing incoming SSH connections to my server and to the router itself
  • Using only IPv4 since I don't want any headaches
  • Automatically emailing me when there's a security patch I need to apply

This ultimate router will be running nothing more than OpenBSD. Almost everything I'm using is included in the base system, pretty cool! I'm going to assume you're capable of installing the OS on your machine and have a working internet connection. If you want a fully-encrypted installation, see our tutorial for that. If you have a serial cable, install it that way. Combining FDE and serial requires some additional steps though. If you don't feel like bothering with a serial console, install the OS on the USB drive from another computer, then put it in the Soekris box. You can also install over PXE if that's your thing. I used the whole device as one partition and installed to that. I let DHCP on the modem configure my first network interface and left the rest of them untouched. The hardware I chose has four NICs, which show up in the OS as em0, em1, em2 and em3. I'm going to be using the first one as the external interface and the other three as the internal interfaces for the LAN. If you need more, I'd recommend getting a system with more NICs or picking up a cheap gigabit switch. The Soekris I'm using also has a PCIe x1 slot that can be used for expansion.


Filesystem and Network

So, let's get a shell and start working. You can use the serial console, SSH into the router from another system or just do this all on another computer booted from the USB drive (swapping it into the Soekris when you're done). First of all, since I'm using a flash drive for the OS, I want to minimize the number of writes to it. I'll append the "noatime" flag to the mount point and enable soft updates. You may also want to consider using mfs or tmpfs for /var and /tmp.

# vi /etc/fstab

Assuming my root device is "sd0" (yours might not be) my fstab will look like this:

/dev/sd0a / ffs rw,noatime,softdep 1 1

You'll need to reboot for this to take effect, but let's get everything in place before we do that. Next, let's add a few network settings so we can bridge your internal interfaces with the virtual vether interface.

# echo 'dhcp -inet6' > /etc/hostname.em0
# echo 'up -inet6' > /etc/hostname.em1
# echo 'up -inet6' > /etc/hostname.em2
# echo 'up -inet6' > /etc/hostname.em3
# echo 'inet 192.168.1.1 255.255.255.0 192.168.1.255 -inet6' > /etc/hostname.vether0
# vi /etc/hostname.bridge0

Add the following:

add vether0
add em1
add em2
add em3
blocknonip vether0
blocknonip em1
blocknonip em2
blocknonip em3
up -inet6

We need to enable IP forwarding and adjust a couple other values to increase performance.

# cp /etc/examples/sysctl.conf /etc
# vi /etc/sysctl.conf

Add the following:

net.inet.ip.forwarding=1
net.inet.ip.redirect=0
kern.bufcachepercent=50
net.inet.ip.ifq.maxlen=1024
net.inet.tcp.mssdflt=1440
kern.securelevel=2
  • net.inet.ip.forwarding lets traffic pass through the interfaces when needed (required for NAT).
  • net.inet.ip.redirect disables sending IP redirects
  • kern.bufcachepercent tells the kernel how much memory it can use for cache.
  • net.inet.ip.ifq.maxlen should be 256 times the number of NICs you have - four in this case.
  • net.inet.tcp.mssdflt should match the "max-mss" value in our firewall config. A value of 1440 is a good general rule for most networks, but you can adjust it to be higher or lower depending on your needs or disable it entirely.
  • kern.securelevel locks the securelevel to the highest setting and prevents changes to the firewall rules. This is optional, but recommended.

DHCP and NTP

Next we tell the DHCP server to start on boot, but we'll wait until we have a valid config file in place before we actually start it. Let's also configure ntpd so that our router's time will always be correct. If you plan on running an NTP server on your router, even only for your LAN, we have a tutorial for that.

# echo 'dhcpd_flags="vether0"' >> /etc/rc.conf.local
# echo 'ntpd_flags=""' >> /etc/rc.conf.local
# cp /etc/examples/ntpd.conf /etc
# ntpd -s

Now we configure the DHCP server:

# vi /etc/dhcpd.conf

Take this example and modify it for your needs:

option domain-name-servers 192.168.1.1;
subnet 192.168.1.0 netmask 255.255.255.0 {
    option routers 192.168.1.1;
    range 192.168.1.4 192.168.1.254;
    host meimei {
        fixed-address 192.168.1.2;
        hardware ethernet 00:00:00:00:00:00;
        }
    host suigintou {
        fixed-address 192.168.1.3;
        hardware ethernet 11:11:11:11:11:11;
        }
}

You can specify any IP address range you want to use and any DNS servers you want to use. By default, I want all clients to query the local DNS resolver that we'll set up in just a minute. This will speed up repeated lookups and is handy to have. Use the MAC addresses of your computers if you want static IPs.


The Firewall

The centerpiece of this entire guide is the file /etc/pf.conf. If you'd like to learn about pf, I recommend our tutorial on different rulesets.

# mv /etc/pf.conf /etc/pf.conf.orig
# vi /etc/pf.conf

After studying the documentation, I ended up with something like this:

int_if="{ vether0 em1 em2 em3 }"
broken="224.0.0.22 127.0.0.0/8 192.168.0.0/16 172.16.0.0/12 \
        10.0.0.0/8 169.254.0.0/16 192.0.2.0/24 \
        192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24, \
        169.254.0.0/16 0.0.0.0/8 240.0.0.0/4 255.255.255.255/32"
set block-policy drop
set loginterface egress
set skip on lo0
match in all scrub (no-df random-id max-mss 1440)
match out on egress inet from !(egress:network) to any nat-to (egress:0)
antispoof quick for (egress)
block in quick on egress from { $broken no-route urpf-failed } to any
block in quick inet6 all
block out quick inet6 all
block return out quick log on egress proto { tcp udp } from any to any port 53
block return out quick log on egress from any to { no-route $broken }
block in all
pass out quick inet keep state
pass in on $int_if inet
pass in on $int_if inet proto { tcp udp } from any to ! 192.168.1.1 port 53 rdr-to 192.168.1.1
pass in on egress inet proto tcp to (egress) port 222 rdr-to 192.168.1.2
pass in on egress inet proto tcp from any to (egress) port 2222

My router will be running SSH on port 2222 and the server will be running SSH on port 222, both open to the internet. Check /etc/ssh/sshd_config for more options. Running sshd on an alternative port, while not really providing any additional security, will cut down the rogue login attempts. You definitely don't want your USB flash drive being worn out from /var/log filling up with Chinese IPs. Modify the "ListenAddress" line of sshd_config if you only want the router's sshd listening for connections on the LAN.


DNS

Setting up a local DNS caching server is pretty easy. We'll be using unbound, which is part of the base system, along with DNSCrypt.

# echo 'unbound_flags=""' >> /etc/rc.conf.local
# mv /var/unbound/etc/unbound.conf /var/unbound/etc/unbound.conf.orig
# vi /var/unbound/etc/unbound.conf

Add the following:

server:
    interface: 192.168.1.1
    interface: 127.0.0.1
    do-ip6: no
    access-control: 192.168.1.0/24 allow
    do-not-query-localhost: no
    hide-identity: yes
    hide-version: yes

forward-zone:
        name: "."
        forward-addr: 127.0.0.1@40

Now we'll set up dnscrypt-proxy. It's not part of the base system, so we'll need to install it from ports or packages.

# export PKG_PATH=http://ftp.openbsd.org/pub/OpenBSD/`uname -r`/packages/`uname -m`/
# pkg_add dnscrypt-proxy
# echo '/usr/local/sbin/dnscrypt-proxy -l /dev/null -R opendns \
       -a 127.0.0.1:40 -u _dnscrypt-proxy -d' >> /etc/rc.local
# echo 'nameserver 127.0.0.1' > /etc/resolv.conf

You can edit /etc/dhclient.conf's "supersede domain-name-servers" section so it doesn't overwrite your local nameserver, or use a more "forceful" approach:

# chflags schg /etc/resolv.conf

The dnscrypt-proxy port won't use any server by default; you need to specify one with the "-R" flag. Check the included documentation for a list of supported hostnames. Lastly, we probably don't need a sound server running on a router, so let's disable it.

# echo 'sndiod_flags=NO' >> /etc/rc.conf.local
# reboot

At this point you should be able to plug in some computers to the other ethernet ports and everything will work. They'll be assigned IP addresses and granted access to the internet, while being protected by the firewall. If that's all you want, you can stop reading here. However, there are some other cool things you can do...


Email

It's possible to configure the router to send you nightly emails using nothing but smtpd in the OpenBSD base system and an email account. I'm using a throwaway gmail account for this example, but you can obviously use any mail server. Make sure your system's hostname is in present in /etc/hosts:

127.0.0.1       localhost bsdnow.tv

We'll add the email account you'll be sending the mail from.

# newaliases
# echo 'gmail youruser@gmail.com:yourpassword' > /etc/mail/secrets
# chmod 640 /etc/mail/secrets
# chown root:_smtpd /etc/mail/secrets
# makemap /etc/mail/secrets

Move the default smtpd configuration to a backup file and create a new one.

# mv /etc/mail/smtpd.conf /etc/mail/smtpd.conf.orig
# vi /etc/mail/smtpd.conf

Add the following, changing the server to whatever you used:

listen on lo0
table aliases db:/etc/mail/aliases.db
table secrets db:/etc/mail/secrets.db
accept for local alias <aliases> deliver to mbox
accept for any relay via tls+auth://gmail@smtp.gmail.com:587 auth <secrets>

Finally, enable it on startup.

# echo 'smtpd_flags=""' >> /etc/rc.conf.local
# /etc/rc.d/smtpd start

Now you should be able to take the output of any command and send it with your email account. We can test it by doing something like this:

# echo 'Woah, my router can send emails! Nice tutorials as always dude!' | mail -v feedback@bsdnow.tv

While this is very basic, the ability to send email from the router is really cool. You can pipe any command or script's output to an email, send it off and then check what's going on in the morning. It can be used for firewall logs, automatically checking for updates and patches, or really anything you can think of.


Automatic Security Notifications

This is a script to check for errata (security or reliability updates to the OS) every night. Before we make it, I'm going to get a checksum of the current errata page for my version of the OS. If you want to create a new unprivileged user to run the script, that might be a good idea. The automatic source code patching will only work as root, unless you change the permissions of /usr/src. If you don't want the automatic patching, everything else can be adjusted to run as a normal user. Make sure you actually have the system source code in /usr/src or that part won't work at all. If you want something less intrusive that simply alerts you that there are patches, comment out the relevant lines. You'll have to manually check the errata and patch it yourself in that case. The project also emails errata patches via the announce list, so it's recommended to subscribe to it. OpenBSD once included the lynx web browser in the base system, but now you'll have to install it yourself.

# pkg_add lynx
# ftp -V4o - http://www.openbsd.org/errata`uname -r | cut -c 1,3`.html | sha256 > /root/checksum
# ftp -V4o /usr/local/bin/erratacheck http://www.bsdnow.tv/patches/erratachecker.txt

The script contains the following:

#!/bin/sh
# OpenBSD errata alert script from www.bsdnow.tv
# For src only, but xenocara can be easily added.

sleep `jot -r 1 0 3600` 

UNM=`uname -r`
VER=`uname -r | cut -c 1,3`
CVS=`uname -r | sed 's/\./_/'`
ERR=/root/errata
CHK=/root/checksum
URL=http://www.openbsd.org/errata$VER.html
SHA=sha256
WATASHI='me@myemail.com'
LASTKNOWN=`cat $CHK` 
FETCH=`ftp -V4o - $URL | $SHA`

if [ "$LASTKNOWN" == "$FETCH" ]; then
    exit
else
    echo "Check $URL for rebuilding instructions." > $ERR
    echo "Files affected:" >> $ERR
    cd /usr/src
        if sysctl kern.version | grep -q stable; then
            cvs -q up -rOPENBSD_$CVS -Pd 2>&1 | tee -a $ERR
        else
        if sysctl kern.version | grep -q current; then
            echo "this script is not for -current."
            exit 1
        else
        if sysctl kern.version | grep -q beta; then
            echo "this script is not for -beta."
            exit 1
        else
        ftp -V4 http://ftp.openbsd.org/pub/OpenBSD/patches/$UNM.tar.gz
        tar xzf $UNM.tar.gz
        rm $UNM.tar.gz
        mv $UNM/*/*sig .
        rm -r $UNM
        for f in *sig
            do
            signify -Vep /etc/signify/openbsd-`uname -r | sed 's/\.//'`-base.pub \
            -x $f -m - | (cd /usr/src && patch -p0 -sN)
            rm $f
            done
        fi
        fi
    fi
    lynx -dump -nolist $URL | sed -e '1,23d' >> $ERR
    cat $ERR | mail -s "New OpenBSD patches for `hostname`" $WATASHI
    rm $ERR
    echo $FETCH > $CHK
fi

Be sure to replace the email variable with your address. Then just put /usr/local/bin/erratacheck in the crontab and make it executable. The script will run at 7:30AM every day. You can change it to any time you want.

# chmod +x /usr/local/bin/erratacheck
# crontab -e

Add a line similar to this one:

30    7    *       *       *       /bin/sh /usr/local/bin/erratacheck

And that's that.


Bandwidth Throttling

If you have to share your internet connection with other users, it's quite possible that they will hog all your bandwidth if you let them. Fortunately, pf provides a way to assign packets to different queues, giving them specific bandwidth limitations. There are two approaches to throttling: setting a maximum limit a connection can use, or reserving a minimum amount that it will always have access to. Since we're nice, we'll just reserve a minimum amount for certain types of traffic that are particularly annoying to use while all the bandwidth is being hoarded. You probably don't want your interactive SSH session lagging because of your friend torrenting. Being forced to wait ten seconds for a website to load because someone is uploading a video of their cat is also unacceptable. For this example, we'll just focus on SSH, FTP(S) and HTTP(S) traffic. The pf.conf manpage has additional details for all the different things you can do with queueing. We'll assume you have a twenty megabit connection, but you can adjust the numbers up or down to suite your needs.

queue limits on em0 bandwidth 20M
queue shell parent limits bandwidth 1M min 1M
queue ftp parent limits bandwidth 8M max 8M
queue web parent limits bandwidth 5M min 5M max 10M default

Our example protocols each get assigned to their own queue. One megabit will always be reserved for SSH traffic, as to prevent any frustrating delays between typing in an interactive session and getting a response. FTP(S) traffic will be choked to a max of eight megabits. Web traffic will always have five megabits reserved for it, but will also be able to use up to ten megabits, depending on how much is available. Add the pass lines for them:

pass out quick inet proto tcp from any to any port 22 set queue shell
pass out quick inet proto tcp from any to any port { 20 21 989 990 } set queue ftp
pass out quick inet proto tcp from any to any port { 80 443 } set queue web

With throttling enabled, the full pf.conf would look something like this:

int_if="{ vether0 em1 em2 em3 }"
broken="224.0.0.22 127.0.0.0/8 192.168.0.0/16 172.16.0.0/12 \
         10.0.0.0/8 169.254.0.0/16 192.0.2.0/24 \
         192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24, \
         169.254.0.0/16 0.0.0.0/8 240.0.0.0/4 255.255.255.255/32"
set block-policy drop
set loginterface egress
set skip on lo0
match in all scrub (no-df random-id max-mss 1440)
match out on egress inet from !(egress:network) to any nat-to (egress:0)
queue limits on em0 bandwidth 20M
queue shell parent limits bandwidth 1M min 1M
queue ftp parent limits bandwidth 8M max 8M
queue web parent limits bandwidth 5M min 5M max 10M default
antispoof quick for (egress)
block in quick on egress from { $broken no-route urpf-failed } to any
block in quick inet6 all
block out quick inet6 all
block return out quick log on egress proto { tcp udp } from any to any port 53
block return out quick log on egress from any to { no-route $broken }
block in all
pass out quick inet proto tcp from any to any port 22 set queue shell
pass out quick inet proto tcp from any to any port { 20 21 989 990 } set queue ftp
pass out quick inet proto tcp from any to any port { 80 443 } set queue web
pass out quick inet keep state
pass in on egress inet proto tcp to (egress) port 222 rdr-to 192.168.1.2
pass in on egress inet proto tcp from any to (egress) port 2222 flags S/SA synproxy state
pass in on $int_if inet
pass in on $int_if proto { tcp udp } from any to ! 192.168.1.1 port 53 rdr-to 192.168.1.1

If you want to throttle both download and upload, you could do it on a per-interface basis. For example, limit the outbound interface (em0) for download and the internal interfaces (em1-3) for upload. With pf's powerful syntax, you can get creative and combine this with certain IPs or hostnames. It's possible to limit a specific user's connection to a video streaming site based on their UID and destination address, add groups that have no limits whatsoever or really anything else you can think of.


Power Saving

You may also want to enable apmd to save power if your hardware supports it. It will scale the CPU down during idle times and turn it up when the load reaches a certain point. Check the man page for a few different options. I'll be using the "cool running performance adjustment" mode.

# echo 'apmd_flags="-C"' >> /etc/rc.conf.local
# /etc/rc.d/apmd start

You can check what level (with 0 being the lowest, 100 being the highest) the CPU is running at with:

# sysctl hw.setperf

Or force it to use the lowest power setting:

# sysctl hw.setperf=0

Try different levels and apmd settings to find the balance you're most comfortable with. Always running it on the lowest setting might limit the data throughput too much, but it will really depend on what hardware you're using. The apmd stuff has been reworked for 5.7, so this section may have to be rewritten when it comes out.


Other Ideas

You may also be interested in using these tools on a router:

Latest News

How did you get into BSD?

2014-11-26

We've got a fun idea for the holidays this year: just like we ask during the interviews, we want to hear how all the viewers and listeners first got into BSD. Email us your story, either written or a video version, and we'll read and play some of them for...

EuroBSDCon 2014

2014-09-18

As you might expect, both Allan and Kris will be at EuroBSDCon this year. They'll be busy hunting down various BSD developers and forcing them to do interviews, but don't hesitate to say hi if you're a listener!...

BSDCan 2014

2014-04-30

We just wrapped up episode 35 after having some horrible audio issues. Sorry about the quality being lower than usual, we did the best we could given the circumstances. Next week we've got a normal episode, but the following week Allan and Kris will be at BSDCan. That week will...

AsiaBSDCon 2014

2014-03-05

Both Allan and Kris will be going to AsiaBSDCon this year, so episode 28 will be shorter than usual. We'll be back the following week with a huge episode. Hopefully they can get some interviews there!...


Episode 068: Just the Essentials

2014-12-17

Direct Download: Video | HD Video | MP3 Audio | OGG Audio | Torrent This episode was brought to you by Headlines More BSD conference videos We mentioned it a few times, but the "New Directions in Operating Systems" conference was held in November in the UK The presentations videos are now online, with a few...

Episode 067: Must Be Rigged

2014-12-10

Direct Download: Video | HD Video | MP3 Audio | OGG Audio | Torrent This episode was brought to you by Headlines Bitrig 1.0 released If you haven't heard of it, Bitrig is a fork of OpenBSD that started a couple years ago According to their FAQ, some of their goals include: only supporting modern hardware...

Episode 066: Conference Connoisseur

2014-12-03

Direct Download: Video | HD Video | MP3 Audio | OGG Audio | Torrent This episode was brought to you by Headlines More BSD presentation videos The MeetBSD video uploading spree continues with a few more talks, maybe this'll be the last batch Corey Vixie, Web Apps in Embedded BSD Allan Jude, UCL config Kip Macy, iflib While we're...

Episode 065: 8,000,000 Mogofoo-ops

2014-11-26

Direct Download: Video | HD Video | MP3 Audio | OGG Audio | Torrent This episode was brought to you by Headlines Even more BSD presentation videos More videos from this year's MeetBSD and OpenZFS devsummit were uploaded since last week Robert Ryan, At the Heart of the Digital Economy FreeNAS & ZFS, The Indestructible Duo -...