NOTES

This episode of BSDNow is brought to you by Tarsnap and the BSDNow Patreon

Headlines

People have no doubt heard of this by now, but are not aware of the BSD side of
things since its mostly been Linux getting all the news. It'd be nice if we
could give a summary of the issue and then address how it does/doesn't affect
the BSDs.
The XZ Backdoor

• NetBSD's statement
• FreeBSD's statement
• OpenBSD?

NetBSD 10.0

News Roundup

iX announces that they will put out a release of TrueNAS 13.3

• A community fork has been announced

State of the Terminal

LibreSSL 3.8.4 and 3.9.1 released

Tarsnap

This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups.

Feedback/Questions

• Derek via feedback has asked for some discussion around this NetBSD security advisory
  -- Advisory Link

• Ben - Nextcloud Installation

• Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv

• Join us and other BSD Fans in our BSD Now Telegram channel

NOTES

This episode of BSDNow is brought to you by Tarsnap and the BSDNow Patreon

Headlines

Using Git offline

Make your own E-Mail server - FreeBSD, OpenSMTPD, Rspamd and Dovecot included - Part 1

News Roundup

quiz: a tool for rapid OpenZFS development

Configuring openzfs for nvme databases

Mirroring OmniOS: The Complete Guide; Part One

Installing OpenBSD 7.4 on a VisionFive 2 rev 1.2a

Terminal Latency

Tarsnap

This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups.

Feedback/Questions

• Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv

• Join us and other BSD Fans in our BSD Now Telegram channel

Headlines

Text processing in the shell

This article is part of a self-published book project by Balthazar Rouberol and Etienne Brodu, ex-roommates, friends and colleagues, aiming at empowering the up and coming generation of developers. We currently are hard at work on it!

One of the things that makes the shell an invaluable tool is the amount of available text processing commands, and the ability to easily pipe them into each other to build complex text processing workflows. These commands can make it trivial to perform text and data analysis, convert data between different formats, filter lines, etc.

When working with text data, the philosophy is to break any complex problem you have into a set of smaller ones, and to solve each of them with a specialized tool.

Rebalancing data on ZFS mirrors

One of the questions that comes up time and time again about ZFS is “how can I migrate my data to a pool on a few of my disks, then add the rest of the disks afterward?”

If you just want to get the data moved and don’t care about balance, you can just copy the data over, then add the new disks and be done with it. But, it won’t be distributed evenly over the vdevs in your pool.

Don’t fret, though, it’s actually pretty easy to rebalance mirrors. In the following example, we’ll assume you’ve got four disks in a RAID array on an old machine, and two disks available to copy the data to in the short term.

News Roundup

Using OpenBSD relayd to Add Security Headers

I am a huge fan of OpenBSD’s built-in httpd server as it is simple, secure, and quite performant. With the modern push of the large search providers pushing secure websites, it is now important to add security headers to your website or risk having the search results for your website downgraded. Fortunately, it is very easy to do this when you combine httpd with relayd. While relayd is principally designed for layer 3 redirections and layer 7 relays, it just so happens that it makes a handy tool for adding the recommended security headers. My website automatically redirects users from http to https and this gets achieved using a simple redirection in /etc/httpd.conf So if you have a configuration similar to mine, then you will still want to have httpd listen on the egress interface on port 80. The key thing to change here is to have httpd listen on 127.0.0.1 on port 443.

How we set up our ZFS filesystem hierarchy in our ZFS pools

Our long standing practice here, predating even the first generation of our ZFS fileservers, is that we have two main sorts of filesystems, home directories (homedir filesystems) and what we call 'work directory' (workdir) filesystems. Homedir filesystems are called /h/NNN (for some NNN) and workdir filesystems are called /w/NNN; the NNN is unique across all of the different sorts of filesystems. Users are encouraged to put as much stuff as possible in workdirs and can have as many of them as they want, which mattered a lot more in the days when we used Solaris DiskSuite and had fixed-sized filesystems.

Speeding up ZSH

https://web.archive.org/web/20200315184849/https://blog.jonlu.ca/posts/speeding-up-zsh

I was opening multiple shells for an unrelated project today and noticed how abysmal my shell load speed was. After the initial load it was relatively fast, but the actual shell start up was noticeably slow. I timed it with time and these were the results.

In the future I hope to actually recompile zsh with additional profiling techniques and debug information - keeping an internal timer and having a flag output current time for each command in a tree fashion would make building heat maps really easy.

How do Unix Pipes work

Pipes are cool! We saw how handy they are in a previous blog post. Let’s look at a typical way to use the pipe operator. We have some output, and we want to look at the first lines of the output. Let’s download The Brothers Karamazov by Fyodor Dostoevsky, a fairly long novel.

What we do to enable us to grow our ZFS pools over time

In my entry on why ZFS isn't good at growing and reshaping pools, I mentioned that we go to quite some lengths in our ZFS environment to be able to incrementally expand our pools. Today I want to put together all of the pieces of that in one place to discuss what those lengths are.
Our big constraint is that not only do we need to add space to pools over time, but we have a fairly large number of pools and which pools will have space added to them is unpredictable. We need a solution to pool expansion that leaves us with as much flexibility as possible for as long as possible. This pretty much requires being able to expand pools in relatively small increments of space.

Linux maintains bugs: The real reason ifconfig on Linux is deprecated

In my third installment of FreeBSD vs Linux, I will discuss underlying reasons for why Linux moved away from ifconfig(8) to ip(8).

In the past, when people said, “Linux is a kernel, not an operating system”, I knew that was true but I always thought it was a rather pedantic criticism. Of course no one runs just the Linux kernel, you run a distribution of Linux. But after reviewing userland code, I understand the significant drawbacks to developing “just a kernel” in isolation from the rest of the system.

Clear Your Terminal in Style

if you’re someone like me who habitually clears their terminal, sometimes you want a little excitement in your life. Here is a way to do just that.

This post revolves around the idea of giving a command a percent chance of running. While the topic at hand is not serious, this simple technique has potential in your scripts.

Feedback/Questions

• Guy - AMD GPU Help
• MLShroyer13 - VLANs and Jails
• Master One - ZFS Suspend/resume

• Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv

Headlines

DragonFlyBSD Project Update - colo upgrade, future trends

For the last week I've been testing out a replacement for Monster, our 48-core opteron server. The project will be removing Monster from the colo in a week or two and replacing it with three machines which together will use half the power that Monster did alone.

The goal is to clear out a little power budget in the colo and to really beef-up our package-building capabilities to reduce the turn-around time needed to test ports syncs and updates to the binary package system.

Currently we use two blades to do most of the building, plus monster sometimes. The blades take almost a week (120 hours+) to do a full synth run and monster takes around 27.5 hours. But we need to do three bulk builds more or less at the same time... one for the release branch, one for the development branch, and one for staging updates. It just takes too long and its been gnawing at me for a little while.

Well, Zen 2 to the rescue! These new CPUs can take ECC, there's actually an IPMI mobo available, and they are fast as hell and cheap for what we get.

The new machines will be two 3900X based servers, plus a dual-xeon system that I already had at home. The 3900X's can each do a full synth run in 24.5 hours and the Xeon can do it in around 31 hours. Monster will be retired. And the crazy thing about this? Monster burns 1000W going full bore. Each of the 3900X servers burns 160W and the Xeon burns 200W. In otherwords, we are replacing 1000W with only 520W and getting roughly 6x the performance efficiency in the upgrade. This tell you just how much more power-efficient machines have become in the last 9 years or so. > This upgrade will allow us to do full builds for both release and dev in roughly one day instead of seven days, and do it without interfering with staging work that might be happening at the same time.

Future trends - DragonFlyBSD has reached a bit of a cross-roads. With most of the SMP work now essentially complete across the entire system the main project focus is now on supplying reliable binary ports for release and developer branches, DRM (GPU) support and other UI elements to keep DragonFlyBSD relevant on workstations, and continuing Filesystem work on HAMMER2 to get multi-device and clustering going.

Resuming ZFS send

One of the amazing functionalities of ZFS is the possibility of sending a whole dataset from one place to another. This mechanism is amazing to create backups of your ZFS based machines. Although, there were some issues with this functionality for a long time when a user sent a big chunk of data. What if you would do that over the network and your connection has disappeared? What if your machine was rebooted as you are sending a snapshot?

For a very long time, you didn't have any options - you had to send a snapshot from the beginning. Now, this limitation was already bad enough. However, another downside of this approach was that all the data which you already send was thrown away. Therefore, ZFS had to go over all this data and remove them from the dataset. Imagine the terabytes of data which you sent via the network was thrown away because as you were sending the last few bytes, the network went off.

In this short post, I don't want to go over the whole ZFS snapshot infrastructure (if you think that such a post would be useful, please leave a comment). Now, to get back to the point, this infrastructure is used to clone the datasets. Some time ago a new feature called “Resuming ZFS send” was introduced. That means that if there was some problem with transmitting the dataset from one point to another you could resume it or throw them away. But the point is, that yes, you finally have a choice.

News Roundup

Realtime bandwidth terminal graph visualization

If for some reasons you want to visualize your bandwidth traffic on an interface (in or out) in a terminal with a nice graph, here is a small script to do so, involving ttyplot, a nice software making graphics in a terminal.

The following will works on OpenBSD. You can install ttyplot by pkg_add ttyplot as root, ttyplot package appeared since OpenBSD 6.5.

fixing telnet fixes

There’s a FreeBSD commit to telnet. fix a couple of snprintf() buffer overflows. It’s received a bit of attention for various reasons, telnet in 2019?, etc. I thought I’d take a look. Here’s a few random observations.

1. The first line is indented with spaces while the others use tabs.

2. The correct type for string length is size_t not unsigned int.

3. sizeof(char) is always one. There’s no need to multiply by it.

4. If you do need to multiply by a size, this is an unsafe pattern. Use calloc or something similar. (OpenBSD provides reallocarray to avoid zeroing cost of calloc.)

5. Return value of malloc doesn’t need to be cast. In fact, should not be, lest you disguise a warning.

6. Return value of malloc is not checked for NULL.

7. No reason to cast cp to char * when passing to snprintf. It already is that type. And if it weren’t, what are you doing?

8. The whole operation could be simplified by using asprintf.

9. Although unlikely (probably impossible here, but more generally), adding the two source lengths together can overflow, resulting in truncation with an unchecked snprintf call. asprintf avoids this failure case.

A Chapter from the FBI’s History with OpenBSD and an OpenSSH Vuln

Earlier this year I FOIAed the FBI for details on allegations of backdoor installed in the IPSEC stack in 2010, originally discussed by OpenBSD devs (https://marc.info/?l=openbsd-tech&m=129236621626462 …) Today, I got an interesting but unexpected responsive record:

• Freedom of Information Act: FBI: OpenBSD
• GitHub Repo

Beastie Bits

• “Sudo Mastery, 2nd Edition” open for tech review
• FreeBSD Journal: FreeBSD for Makers
• OpenBSD and NetBSD machines at Open Source Conference 2019 Nagoya
• FreeBSD 12.0: WINE Gaming
• Introduction to the Structure and Interpretation of TNF (The NetBSD Foundation)
• vBSDcon speakers announced

Feedback/Questions

• Pat - NYCBug Aug 7th
• Tyler - SSH keys vs password
• Lars - Tor-Talk

• Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv