NOTES

This episode of BSDNow is brought to you by Tarsnap and the BSDNow Patreon

Headlines

Key Considerations for Benchmarking Network Storage Performance

OpenZFS 2.3.0 available

News Roundup

Updates on AsiaBSDCon 2025 - Cancelled -

GhostBSD Desktop Conference

Recovering from external zroot

Create a new issue in a Github repository with Ansible

Stories I refuse to believe

Defer the January 19, 2038 date limit in UFS1 filesystems to February 7, 2106

Tarsnap

This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups.

Feedback/Questions

Feedback - Nelson - Ada/GCC

• Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv

• Join us and other BSD Fans in our BSD Now Telegram channel

Headlines

A Pi-Powered Plan 9 Cluster

Plan 9 from Bell Labs comes from the same stable as the UNIX operating system, which of course Linux was designed after, and Apple’s OS X runs on top of a certified UNIX operating system. Just like UNIX, Plan 9 was developed as a research O/S — a vehicle for trying out new concepts — with it building on key UNIX principles and taking the idea of devices are just files even further. In this post, we take a quick look at the Plan 9 O/S and some of the notable features, before moving on to the construction of a self-contained 4-node Raspberry Pi cluster that will provide a compact platform for experimentation.

---

Endlessh: an SSH Tarpit

I’m a big fan of tarpits: a network service that intentionally inserts delays in its protocol, slowing down clients by forcing them to wait. This arrests the speed at which a bad actor can attack or probe the host system, and it ties up some of the attacker’s resources that might otherwise be spent attacking another host. When done well, a tarpit imposes more cost on the attacker than the defender. The Internet is a very hostile place, and anyone who’s ever stood up an Internet-facing IPv4 host has witnessed the immediate and continuous attacks against their server. I’ve maintained such a server for nearly six years now, and more than 99% of my incoming traffic has ill intent. One part of my defenses has been tarpits in various forms.

---

News Roundup

rdist(1) – when Ansible is too much

The post written about rdist(1) on johan.huldtgren.com sparked us to write one as well. It's a great, underappreciated, tool. And we wanted to show how we wrapped doas(1) around it. There are two services in our infrastructure for which we were looking to keep the configuration in sync and to reload the process when the configuration had indeed changed. There is a pair of nsd(8)/unbound(8) hosts and a pair of hosts running relayd(8)/httpd(8) with carp(4) between them. We didn't have a requirement to go full configuration management with tools like Ansible or Salt Stack. And there wasn't any interest in building additional logic on top of rsync or repositories. > Enter rdist(1), rdist is a program to maintain identical copies of files over multiple hosts. It preserves the owner, group, mode, and mtime of files if possible and can update programs that are executing.

---

Falling in love with OpenBSD again

I was checking the other day and was appalled at how long it has been since I posted here. I had been working a job during 2018 that had me traveling 3,600 miles by air every week so that is at least a viable excuse. So what is my latest project? I wanted to get something better than the clunky old T500 “freedom laptop” that I could use as my daily driver. Some background here. My first paid gig as a programmer was on SunOS 4 (predecessor to Solaris) and Ultrix (on a DEC MicroVAX). I went from there to a Commodore Amiga (preemptive multitasking in 1985!). I went from there to OS/2 (I know, patron saint of lost causes) and then finally decided to “sell out” and move to Windows as the path of least resistance in the mid 90’s. My wife bought me an iPod literally just as they started working with computers other than Macs and I watched with fascination as Apple made the big gamble and moved away from PowerPC chips to Intel. That was the beginning of the Apple Fan Boi years for me. My gateway drug was a G4 MacMini and I managed somehow to get in on the pre-production, developer build of an Intel-based Mac. I was quite happy on the platform until about three years ago.

---

How I Created My First FreeBSD Port

I created my first FreeBSD port recently. I found that FreeBSD didn't have a port for GoCD, which is a continuous integration and continuous deployment (CI/CD) system. This was a great opportunity to learn how to build a FreeBSD port while also contributing back to the community

---

The Tilde Institute of OpenBSD Education

Welcome to tilde.institute! This is an OpenBSD machine whose purpose is to provide a space in the tildeverse for experimentation with and education of the OpenBSD operating system. A variety of editors, shells, and compilers are installed to allow for development in a native OpenBSD environment. OpenBSD's httpd(8) is configured with slowcgi(8) as the fastcgi provider and sqlite3 available. This allows users to experiment with web development using compiled CGI in C, aka the BCHS Stack. In addition to php7.0 and mysql (mariadb) by request, this provides an environment where the development of complex web apps is possible.

---

Beastie Bits

• SoloBSD 19.03-STABLE
• WireGuard for NetBSD
• [NetBSD - Removing PF](https://mail-index.netbsd.org/tech-kern/2019/03/29/msg024883.html )
• What does the N in nmake stand for?
• A Map of the Internet from May 1973
• NSA-B-Gone : A sketchy hardware security device for your x220

Feedback/Questions

• Jake - A single jail as a VPN client
• Matt - Surprising BSD Features
• cia - Routing and ZFS

• Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv

##Headlines
###MidnightBSD 1.0 now available

I’m happy to announce the availability of MidnightBSD 1.0 for amd64 and i386. Over the years, many ambitious goals were set for our 1.0 release. As it approached, it was clear we wouldn’t be able to accomplish all of them. This release is more of a natural progression rather than a groundbreaking event. It includes many updates to the base system, improvements to the package manager, an updated compiler, and tools.
Of particular note, you can now boot off of ZFS and use NVME SSDs and some AMD Radeon graphics cards support acceleration. AMD Ryzen support has greatly improved in this release. We also have added bhyve from FreeBSD.
The 1.0 release is finally available. Still building packages for i386 and plan to do an amd64 package build later in the week. The single largest issue with the release process has been the web server performance. The CPU is overloaded and has been at solid 100% for several days. The server has a core i7 7700 in it. I’m trying to figure out what to buy as an upgrade so that we don’t continue to have this issue going forward. As it’s actually blocked in multiple processes, a 6 or 8 core chip might be an improvement for the workload…

• Download links: https://www.midnightbsd.org/download/
• https://www.youtube.com/watch?time_continue=33&v=-rlk2wFsjJ4

###MeetBSD Review

MeetBSD 2018 took place at the sprawling Intel Santa Clara campus. The venue itself felt more like an olive branch than a simple friendly gesture by Intel. In truth it felt like a bit of an apology. You get the subtle sense they feel bad about how the BSD’s were treated with the Meltdown and Specter flaws. In fact, you may be right to think they felt a bit sorry towards the entire open source community.

• MeetBSD 2018

At most massive venues the parking is the first concern, not so here - in fact that was rather straightforward. No, the real challenge is navigating the buildings. Luckily I had help from navigator extraordinaire, Hadea, who located the correct building, SC12 quickly. Finding the entrance took a moment or two though. The lobby itself was converted by iXsystems efficiently into the MeetBSD expo hall, clean, efficient and roomy with registration, some seating, and an extra conference room for on-on-one sessions. On day two sponsor booths were also setup. All who showed up on day one were warmly greeted with badges, lanyards and goodies by Denise and her friendly team.
Like every great BSD event, plenty of food was made available. And as always they make it look effortless. These events showcase iXsystem’s inherent generosity toward its community; with breakfast items in the back of the main auditorium room in the morning, boxed lunches, fruit and cookies at lunch time, and snacks for the rest of the day. But just in case your still hungry, there is a pizza meetup in another Intel room after day one and two.
MeetBSD leverages it’s realistically small crowd size on day one. The morning starts off with introductions of the entire group, the mic is passed around the room.
The group is a good mix of pros in the industry (such as Juniper, Intel, Ebay, Groupon, Cisco, etc), iX staff, and a few enthusiast. Lots of people with a focus or passion for networking. And, of course, some friendly Linux bashing went down for good measure, always followed by a good natured chuckle.

• MeetBSD Gives me The Feels

I find that I am subtly unnerved at this venue, and at lunch I saw it clearly. I have always had a strong geek radar, allowing me to navigate a new area (like Berkeley for MeetBSD of 2016, or even SCALE earlier this year in Pasadena), and in a glance I can see who is from my conference and who isn’t. This means it is easy, nearly effortless to know who to greet with a smile and a wave. These are MY people. Here at the Intel campus though it is different. The drive in alone reveals behemoth complexes all with well known tech names prominently displayed. This is Silicon Valley, and all of these people look like MY people. So much for knowing who’s from my conference. Thank goodness for those infamous BSD horns. None-the-less I am struck by how massive these tech giants are. And Intel is one of the largest of those giants, and see the physical reminders of this fact brought home the significance that they had opened their doors, wifi, and bathrooms to the BSD community.

###[EuroBSDcon 2018 Trip Reports]
https://www.freebsdfoundation.org/blog/eurobsd-2018-trip-report-joseph-mingrone/
https://www.freebsdfoundation.org/blog/eurobsd-2018-trip-report-vinicius-zavam/
https://www.freebsdfoundation.org/blog/eurobsd-2018-trip-report-emmanuel-vadot/

##News Roundup
###DNS over TLS in FreeBSD 12

With the arrival of OpenSSL 1.1.1, an upgraded Unbound, and some changes to the setup and init scripts, FreeBSD 12.0, currently in beta, now supports DNS over TLS out of the box.
DNS over TLS is just what it sounds like: DNS over TCP, but wrapped in a TLS session. It encrypts your requests and the server’s replies, and optionally allows you to verify the identity of the server. The advantages are protection against eavesdropping and manipulation of your DNS traffic; the drawbacks are a slight performance degradation and potential firewall traversal issues, as it runs over a non-standard port (TCP port 853) which may be blocked on some networks. Let’s take a look at how to set it up.

• Conclusion

We’ve seen how to set up Unbound—specifically, the local_unbound service in FreeBSD 12.0—to use DNS over TLS instead of plain UDP or TCP, using Cloudflare’s public DNS service as an example. We’ve looked at the performance impact, and at how to ensure (and verify) that Unbound validates the server certificate to prevent man-in-the-middle attacks.
The question that remains is whether it is all worth it. There is undeniably a performance hit, though this may improve with TLS 1.3. More importantly, there are currently very few DNS-over-TLS providers—only one, really, since Quad9 filter their responses—and you have to weigh the advantage of encrypting your DNS traffic against the disadvantage of sending it all to a single organization. I can’t answer that question for you, but I can tell you that the parameters are evolving quickly, and if your answer is negative today, it may not remain so for long. More providers will appear. Performance will improve with TLS 1.3 and QUIC. Within a year or two, running DNS over TLS may very well become the rule rather than the experimental exception.

###Upgrading OpenBSD with Ansible

• My router runs OpenBSD -current

A few months ago, I needed software that had just hit the ports tree. I didn’t want to wait for the next release, so I upgraded my router to use -current. Since then, I’ve continued running -current, which means upgrading to a newer snapshot every so often. Running -current is great, but the process of updating to a newer snapshot was cumbersome. Initially, I had to plug in a serial cable and then reboot into bsd.rd, hit enter ten times, then reboot, run sysmerge and update packages.
I eventually switched to upobsd to be able to upgrade without the need for a serial connection. The process was better, but still tiresome. Usually, I would prepare the special version of bsd.rd, boot on bsd.rd, and do something like wash the dishes in the meantime. After about ten minutes, I would dry my hands and then go back to my workstation to see whether the bsd.rd part had finished so I could run sysmerge and pkg_add, and then return to the dishes while it upgraded packages.
Out of laziness, I thought: “I should automate this,” but what happened instead is that I simply didn’t upgrade that machine very often. (Yes, laziness). With my router out of commission, life is very dull, because it is my gateway to the Internet. Even services hosted at my place (like my Mastodon instance) are not reachable when the router is down because I use multiple VLANs (so I need the router to jump across VLANs).

• Ansible Reboot Module

I recently got a new job, and one of my first tasks was auditing the Ansible roles written by my predecessors. In one role, the machine rebooted and they used the wait_for_connection module to wait for it to come back up. That sounded quite hackish to me, so out of curiosity, I tried to determine whether there was a better way. I also thought I might be able to use something similar to further automate my OpenBSD upgrades, and wanted to assess the cleanliness of this method. ;-)
I learned that with the then-upcoming 2.7 Ansible release, a proper reboot module would be included. I went to the docs, which stated that for a certain parameter:
I took this to mean that there was no support for OpenBSD. I looked at the code and, indeed, there was not. However, I believed that it wouldn’t be too hard to add it. I added the missing pieces for OpenBSD, tested it on my poor Pine64 and then submitted it upstream. After a quick back and forth, the module’s author merged it into devel (having a friend working at Red Hat helped the process, merci Cyril !) A couple days later, the release engineer merged it into stable-2.7.
I proceeded to actually write the playbook, and then I hit a bug. The parameter reboot_timeout was not recognized by Ansible. This feature would definitely be useful on a slow machine (such as the Pine64 and its dying SD card). Again, my fix was merged into master by the module’s author and then merged into stable-2.7. 2.7.1 will be the first release to feature these fixes, but if you use OpenBSD -current, you already have access to them. I backported the patches when I updated ansible.
Fun fact about Ansible and reboots: “The win_reboot module was […] included with Ansible 2.1,” while for unix systems it wasn’t added until 2.7. :D For more details, you can read the module’s author blog article.

• The explanations

Ansible runs my script on the remote host to fetch the sets. It creates an answer file from the template and then gives it to upobsd. Once upobsd has created the kernel, Ansible copies it in place of /bsd on the host. The router reboots and boots on /bsd, which is upobsd’s bsd.rd. The installer runs in auto_update mode. Once it comes back from bsd.rd land, it archives the kernel and finishes by upgrading all the packages.
It also supports upgrading without fetching the sets ahead of time. For instance, I upgrade this way on my Pine64 because if I cared about speed, I wouldn’t use this weak computer with its dying SD card. For this case, I just comment out the path_sets variable and Ansible instead creates an answer file that will instruct the installer to fetch the sets from the designated mirror.
I’ve been archiving my kernels for a few years. It’s a nice way to fill up / keep a history of my upgrades. If I spot a regression, I can try a previous kernel … which may not work with the then-desynchronized userland, but that’s another story.
sysmerge already runs with rc.sysmerge in batch mode and sends the result by email. I don’t think there’s merit to running it again in the playbook. The only perk would be discovering in the terminal whether any files need to be manually merged, rather than reading exactly the same output in the email.
Initially, I used the openbsd_pkg module, but it doesn’t work on -current just before a release because pkg_add automatically looks for pub/OpenBSD/${release}/packages/${arch} (which is empty). I wrote and tested this playbook while 6.4 was around the corner, so I switched to command to be able to pass the -Dsnap parameter.

• The result

I’m very happy with the playbook! It performs the upgrade with as little intervention as possible and minimal downtime. \o/

###Using smartd to automatically run tests on your drives

Those programs can “control and monitor storage systems using the Self-Monitoring, Analysis and Reporting Technology System (SMART) built into most modern ATA/SATA, SCSI/SAS and NVMe disks. In many cases, these utilities will provide advanced warning of disk degradation and failure.” See the smartmontools website for more information.

NOTE: “Due to OS-specific issues and also depending on the different state of smartmontools development on the platforms, device support is not the same for all OS platforms.” – use the documentation for your OS.

I first started using smartd in March 2010 (according to that blog post, that’s when I still writing on both The FreeBSD Diary and this blog). Back then, and until recently, all I did was start smartd. As far as I can tell, all it did was send daily status messages via the FreeBSD periodic tools. I would set my drive devices via daily_status_smart_devices in /etc/periodic.conf and the daily status reports would include drive health information.

• Two types of tests
• My original abandoned attempt
• How do you prove it works?
• Looking at the test results
• Failed drive to the rescue
• smartd.conf I am using
• supernews

##Beastie Bits

• Decent Pics of “Relayd & Httpd Mastery” signature
• A Unix Shell poster from 1983
• Cambridge UNIX historians (Cambridge, United Kingdom)
• Goals for FreeBSD 13
• September/October 2018 Issue of the FreeBSD Journal Now Available
• Using acme.sh for Let’s Encrypt certificates on pkgsrc.org servers
• Deploying Anycast DNS Using OpenBSD and BGP
• How to check your data integrity?

##Feedback/Questions

• Raymond - MeetBSD California
  • Dev Summit Videos: https://www.youtube.com/playlist?list=PLb87fdKUIo8TNG6f94xo9_W-XXrEbqgWI
  • Conference Videos: https://www.youtube.com/playlist?list=PLb87fdKUIo8Q41aoPE6vssP-uF4dxk86b
  • Conference videos are still being processed, the rest should appear over the next few weeks.



• Greg - Stable vs Release


• Mjrodriguez - Open/FreeBSD support for Single Board computers

• Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv

This episode was brought to you by

Headlines

FreeBSD 10.0-RELEASE is out

• The long awaited, giant release of FreeBSD is now official and ready to be downloaded
• One of the biggest releases in FreeBSD history, with tons of new updates
• Some features include: LDNS/Unbound replacing BIND, Clang by default (no GCC anymore), native Raspberry Pi support and other ARM improvements, bhyve, hyper-v support, AMD KMS, VirtIO, Xen PVHVM in GENERIC, lots of driver updates, ZFS on root in the installer, SMP patches to pf that drastically improve performance, Netmap support, pkgng by default, wireless stack improvements, a new iSCSI stack, FUSE in the base system... the list goes on and on
• Start up your freebsd-update or do a source-based upgrade ***

OpenSSH 6.5 CFT

• Our buddy Damien Miller announced a Call For Testing for OpenSSH 6.5
• Huge, huge release, focused on new features rather than bugfixes (but it includes those too)
• New ciphers, new key formats, new config options, see the mailing list for all the details
• Should be in OpenBSD 5.5 in May, look forward to it - but also help test on other platforms! ***

DIY NAS story, FreeNAS 9.2.1-BETA

• Another new blog post about FreeNAS!
• Instead of updating the older tutorials, the author started fresh and wrote a new one for 2014
• "I did briefly consider suggesting nas4free for the EconoNAS blog, since it’s essentially a fork off the FreeNAS tree but may run better on slower hardware, but ultimately I couldn’t recommend anything other than FreeNAS"
• Really long article with lots of nice details about his setup, why you might want a NAS, etc.
• Speaking of FreeNAS, they released 9.2.1-BETA with lots of bugfixes ***

OpenBSD needed funding for electricity.. and they got it

• Briefly mentioned at the end of last week's show, but has blown up over the internet since
• OpenBSD in the headlines of major tech news sites: slashdot, zdnet, the register, hacker news, reddit, twitter.. thousands of comments
• They needed about $20,000 to cover electric costs for the server rack in Theo's basement
• Lots of positive reaction from the community helping out so far, and it appears they have reached their goal and got $100,000 in donations
• From Bob Beck: "we have in one week gone from being in a dire situation to having a commitment of approximately $100,000 in donations to the foundation"
• This is a shining example of the BSD community coming together, and even the Linux people realizing how critical BSD is to the world at large ***

Interview - Colin Percival - cperciva@freebsd.org / @cperciva

FreeBSD on Amazon EC2, backups with Tarsnap, 10.0-RELEASE, various topics

Tutorial

Bandwidth monitoring and testing

News Roundup

pfSense talk at Tokyo FreeBSD Benkyoukai

• Isaac Levy will be presenting "pfSense Practical Experiences: from home routers, to High-Availability Datacenter Deployments"
• He's also going to be looking for help to translate the pfSense documentation into Japanese
• The event is on February 17, 2014 if you're in the Tokyo area ***

m0n0wall 1.8.1 released

• For those who don't know, m0n0wall is an older BSD-based firewall OS that's mostly focused on embedded applications
• pfSense was forked from it in 2004, and has a lot more active development now
• They switched to FreeBSD 8.4 for this new version
• Full list of updates in the changelog
• This version requires at least 128MB RAM and a disk/CF size of 32MB or more, oh no! ***

Ansible and PF, plus NTP

• Another blog post from our buddy Michael Lucas
• There've been some NTP amplification attacks recently in the news
• The post describes how he configured ntpd on a lot of servers without a lot of work
• He leverages pf and ansible for the configuration
• OpenNTPD is, not surprisingly, unaffected - use it ***

ruBSD videos online

• Just a quick followup from a few weeks ago
• Theo and Henning's talks from ruBSD are now available for download
• There's also a nice interview with Theo ***

PCBSD weekly digest

• 10.0-RC4 images are available
• Wine PBI is now available for 10
• 9.2 systems will now be able to upgrade to version 10 and keep their PBI library ***

Feedback/Questions

• Sha'ul writes in
• Kjell-Aleksander writes in
• Mike writes in
• Charlie writes in (and gets a reply)
• Kevin writes in ***