Headlines

The Earliest Unix Code: An Anniversary Source Code Release

What is it that runs the servers that hold our online world, be it the web or the cloud? What enables the mobile apps that are at the center of increasingly on-demand lives in the developed world and of mobile banking and messaging in the developing world? The answer is the operating system Unix and its many descendants: Linux, Android, BSD Unix, MacOS, iOS—the list goes on and on. Want to glimpse the Unix in your Mac? Open a Terminal window and enter “man roff” to view the Unix manual entry for an early text formatting program that lives within your operating system.

2019 marks the 50th anniversary of the start of Unix. In the summer of 1969, that same summer that saw humankind’s first steps on the surface of the Moon, computer scientists at the Bell Telephone Laboratories—most centrally Ken Thompson and Dennis Ritchie—began the construction of a new operating system, using a then-aging DEC PDP-7 computer at the labs.

This man sent the first online message 50 years ago

• As many of you have heard in the past, the first online message ever sent between two computers was "lo", just over 50 years ago, on Oct. 29, 1969.

It was supposed to say "log," but the computer sending the message — based at UCLA — crashed before the letter "g" was typed. A computer at Stanford 560 kilometres away was supposed to fill in the remaining characters "in," as in "log in."

• The CBC Radio show, “The Current” has a half-hour interview with the man who sent that message, Leonard Kleinrock, distinguished professor of computer science at UCLA

"The idea of the network was you could sit at one computer, log on through the network to a remote computer and use its services there,"

50 years later, the internet has become so ubiquitous that it has almost been rendered invisible. There's hardly an aspect in our daily lives that hasn't been touched and transformed by it.

Q: Take us back to that day 50 years ago. Did you have the sense that this was going to be something you'd be talking about a half a century later?

A: Well, yes and no. Four months before that message was sent, there was a press release that came out of UCLA in which it quotes me as describing what my vision for this network would become. Basically what it said is that this network would be always on, always available. Anybody with any device could get on at anytime from any location, and it would be invisible.

Well, what I missed ... was that this is going to become a social network. People talking to people. Not computers talking to computers, but [the] human element.

Q: Can you briefly explain what you were working on in that lab? Why were you trying to get computers to actually talk to one another?

A: As an MIT graduate student, years before, I recognized I was surrounded by computers and I realized there was no effective [or efficient] way for them to communicate. I did my dissertation, my research, on establishing a mathematical theory of how these networks would work. But there was no such network existing. AT&T said it won't work and, even if it does, we want nothing to do with it.

So I had to wait around for years until the Advanced Research Projects Agency within the Department of Defence decided they needed a network to connect together the computer scientists they were supervising and supporting.

Q: For all the promise of the internet, it has also developed some dark sides that I'm guessing pioneers like yourselves never anticipated.

A: We did not. I knew everybody on the internet at that time, and they were all well-behaved and they all believed in an open, shared free network. So we did not put in any security controls.

When the first spam email occurred, we began to see the dark side emerge as this network reached nefarious people sitting in basements with a high-speed connection, reaching out to millions of people instantaneously, at no cost in time or money, anonymously until all sorts of unpleasant events occurred, which we called the dark side.

But in those early days, I considered the network to be going through its teenage years. Hacking to spam, annoying kinds of effects. I thought that one day this network would mature and grow up. Well, in fact, it took a turn for the worse when nation states, organized crime and extremists came in and began to abuse the network in severe ways.

Q: Is there any part of you that regrets giving birth to this?

A: Absolutely not. The greater good is much more important.

News Roundup

How to use blacklistd(8) with NPF as a fail2ban replacement

blacklistd(8) provides an API that can be used by network daemons to communicate with a packet filter via a daemon to enforce opening and closing ports dynamically based on policy.

The interface to the packet filter is in /libexec/blacklistd-helper (this is currently designed for npf) and the configuration file (inspired from inetd.conf) is in etc/blacklistd.conf

Now, blacklistd(8) will require bpfjit(4) (Just-In-Time compiler for Berkeley Packet Filter) in order to properly work, in addition to, naturally, npf(7) as frontend and syslogd(8), as a backend to print diagnostic messages. Also remember npf shall rely on the npflog* virtual network interface to provide logging for tcpdump() to use.

Unfortunately (dont' ask me why ??) in 8.1 all the required kernel components are still not compiled by default in the GENERIC kernel (though they are in HEAD), and are rather provided as modules. Enabling NPF and blacklistd services would normally result in them being automatically loaded as root, but predictably on securelevel=1 this is not going to happen.

• FreeBSD’s handbook chapter on blacklistd

OpenBSD crossed 400,000 commits

Sometime in the last week OpenBSD crossed 400,000 commits (*) upon all our repositories since starting at 1995/10/18 08:37:01 Canada/Mountain. That's a lot of commits by a lot of amazing people.

(*) by one measure. Since the repository is so large and old, there are a variety of quirks including ChangeLog missing entries and branches not convertible to other repo forms, so measuring is hard. If you think you've got a great way of measuring, don't be so sure of yourself -- you may have overcounted or undercounted.

• Subject to the notes Theo made about under and over counting, FreeBSD should hit 1 million commits (base + ports + docs) some time in 2020
• NetBSD + pkgsrc are approaching 600,000, but of course pkgsrc covers other operating systems too

How to Install Bolt CMS with Nginx and Let's Encrypt on FreeBSD 12

Bolt is a sophisticated, lightweight and simple CMS built with PHP. It is released under the open-source MIT-license and source code is hosted as a public repository on Github. A bolt is a tool for Content Management, which strives to be as simple and straightforward as possible. It is quick to set up, easy to configure, uses elegant templates. Bolt is created using modern open-source libraries and is best suited to build sites in HTML5 with modern markup. In this tutorial, we will go through the Bolt CMS installation on FreeBSD 12 system by using Nginx as a web server, MySQL as a database server, and optionally you can secure the transport layer by using acme.sh client and Let's Encrypt certificate authority to add SSL support.

• Requirements
• The system requirements for Bolt are modest, and it should run on any fairly modern web server:
  • PHP version 5.5.9 or higher with the following common PHP extensions: pdo, mysqlnd, pgsql, openssl, curl, gd, intl, json, mbstring, opcache, posix, xml, fileinfo, exif, zip.
  • Access to SQLite (which comes bundled with PHP), or MySQL or PostgreSQL.
  • Apache with mod_rewrite enabled (.htaccess files) or Nginx (virtual host configuration covered below).
  • A minimum of 32MB of memory allocated to PHP.

hammer2 - Optimize hammer2 support threads and dispatch

Refactor the XOP groups in order to be able to queue strategy calls, whenever possible, to the same CPU as the issuer. This optimizes several cases and reduces unnecessary IPI traffic between cores. The next best thing to do would be to not queue certain XOPs to an H2 support thread at all, but I would like to keep the threads intact for later clustering work.

The best scaling case for this is when one has a large number of user threads doing I/O. One instance of a single-threaded program on an otherwise idle machine might see a slightly reduction in performance but at the same time we completely avoid unnecessarily spamming all cores in the system on the behalf of a single program, so overhead is also significantly lower.

This will tend to increase the number of H2 support threads since we need a certain degree of multiplication for domain separation.

This should significantly increase I/O performance for multi-threaded workloads.

You know, we might as well just run every network service over HTTPS/2 and build another six layers on top of that to appease the OSI 7-layer burrito guys

I've seen the writing on the wall, and while for now you can configure Firefox not to use DoH, I'm not confident enough to think it will remain that way. To that end, I've finally set up my own DoH server for use at Chez Boca. It only involved setting up my own CA to generate the appropriate certificates, install my CA certificate into Firefox, configure Apache to run over HTTP/2 (THANK YOU SO VERY XXXXX­XX MUCH GOOGLE FOR SHOVING THIS HTTP/2 XXXXX­XXX DOWN OUR THROATS!—no, I'm not bitter) and write a 150 line script that just queries my own local DNS, because, you know, it's more XXXXX­XX secure or some XXXXX­XXX reason like that.

Sigh.

Beastie Bits

• An Oral History of Unix
• NUMA Siloing in the FreeBSD Network Stack [pdf]
• EuroBSDCon 2019 videos available
• Barbie knows best
• For the #OpenBSD #e2k19 attendees. I did a pre visit today.
• Drawer Find
• Slides - Removing ROP Gadgets from OpenBSD - AsiaBSDCon 2019

Feedback/Questions

• Bostjan - Open source doesn't mean secure
• Malcolm - Allan is Correct.

• Michael - FreeNAS inside a Jail

  ---

• Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv

Headlines

ZFSonFreeBSD ports renamed OpenZFS

• The ZFS on FreeBSD project has renamed the userland and kernel ports from zol and zol-kmod to openzfs and openzfs-kmod
• The new versions from this week are IOCTL compatible with the command line tools in FreeBSD 12.0, so you can use the old userland with the new kernel module (although obviously not the new features)
• With the renaming it is easier to specify which kernel module you want to load in /boot/loader.conf: > zfs_load=”YES”
• or > openzfs_load=”YES”
• To load traditional or the newer version of ZFS
• The kmod still requires FreeBSD 12-stable or 13-current because it depends on the newer crypto support in the kernel for the ZFS native encryption feature. Allan is looking at ways to work around this, but it may not be practical.
• We would like to do an unofficial poll on how people would the userland to co-exist. Add a suffix to the new commands in /usr/local (zfs.new zpool.new or whatever). One idea i’ve had is to move the zfs and zpool commands to /libexec and make /sbin/zfs and /sbin/zpool a switcher script, that will call the base or ports version based on a config file (or just based on if the port is installed)
• For testing purposes, generally you should be fine as long as you don’t run ‘zpool upgrade’, which will make your pool only importable using the newer ZFS.
• For extra safety, you can create a ‘zpool checkpoint’, which will allow you to undo any changes that are made to the pool during your testing with the new openzfs tools. Note: the checkpoint will undo EVERYTHING. So don’t save new data you want to keep.
• Note: Checkpoints disable all freeing operations, to prevent any data from being overwritten so that you can re-import at the checkpoint and undo any operation (including zfs destroy-ing a dataset), so also be careful you don’t run out of space during testing.
• Please test and provide feedback.

How to use blacklistd(8) with NPF as a fail2ban replacement

• About blacklistd(8)

blacklistd(8) provides an API that can be used by network daemons to communicate with a packet filter via a daemon to enforce opening and closing ports dynamically based on policy.
The interface to the packet filter is in /libexec/blacklistd-helper (this is currently designed for npf) and the configuration file (inspired from inetd.conf) is in etc/blacklistd.conf
Now, blacklistd(8) will require bpfjit(4) (Just-In-Time compiler for Berkeley Packet Filter) in order to properly work, in addition to, naturally, npf(7) as frontend and syslogd(8), as a backend to print diagnostic messages. Also remember npf shall rely on the npflog* virtual network interface to provide logging for tcpdump() to use.
Unfortunately (dont' ask me why :P) in 8.1 all the required kernel components are still not compiled by default in the GENERIC kernel (though they are in HEAD), and are rather provided as modules. Enabling NPF and blacklistd services would normally result in them being automatically loaded as root, but predictably on securelevel=1 this is not going to happen

News Roundup

[WIP] raidz expansion, alpha preview 1

• Motivation and Context > This is a alpha-quality preview of RAID-Z expansion. This feature allows disks to be added one at a time to a RAID-Z group, expanding its capacity incrementally. This feature is especially useful for small pools (typically with only one RAID-Z group), where there isn't sufficient hardware to add capacity by adding a whole new RAID-Z group (typically doubling the number of disks). > For additional context as well as a design overview, see my short talk from the 2017 OpenZFS Developer Summit: slides video

Rant: running audio VU-meter increases my CO2 footprint

A couple months ago I noticed that the monitor on my workstation never power off anymore. Screensaver would go on, but DPMs (to do the poweroff) never kicked in.
I grovels the output of various tools that display DPMS settings, which as usual in Xorg were useless. Everybody said DPMS is on with a timeout. I even wrote my own C program to use every available Xlib API call and even the xscreensaver library calls. (should make it available) No go, everybody says that DPMs is on, enabled and set on a timeout. Didn’t matter whether I let xscreeensaver do the job or just the X11 server.
After a while I noticed that DPMS actually worked between starting my X11 server and starting all my clients. I have a minimal .xinitrc and start the actual session from a script, that is how I could notice. If I used a regular desktop login I wouldn’t have noticed. A server state bug was much more likely than a client bug.

• See the article for the rest...

XSAVE and compat32 kernel work for LLDB

Upstream describes LLDB as a next generation, high-performance debugger. It is built on top of LLVM/Clang toolchain, and features great integration with it. At the moment, it primarily supports debugging C, C++ and ObjC code, and there is interest in extending it to more languages.
In February, I have started working on LLDB, as contracted by the NetBSD Foundation. So far I've been working on reenabling continuous integration, squashing bugs, improving NetBSD core file support and lately extending NetBSD's ptrace interface to cover more register types. You can read more about that in my Apr 2019 report.
In May, I was primarily continuing the work on new ptrace interface. Besides that, I've found and fixed a bug in ptrace() compat32 code, pushed LLVM buildbot to ‘green’ status and found some upstream LLVM regressions. More below.

Some things about where icons for modern X applications come from

If you have a traditional window manager like fvwm, one of the things it can do is iconify X windows so that they turn into icons on the root window (which would often be called the 'desktop'). Even modern desktop environments that don't iconify programs to the root window (or their desktop) may have per-program icons for running programs in their dock or taskbar. If your window manager or desktop environment can do this, you might reasonably wonder where those icons come from by default.
Although I don't know how it was done in the early days of X, the modern standard for this is part of the Extended Window Manager Hints. In EWMH, applications give the window manager a number of possible icons, generally in different sizes, as ARGB bitmaps (instead of, say, SVG format). The window manager or desktop environment can then pick whichever icon size it likes best, taking into account things like the display resolution and so on, and display it however it wants to (in its original size or scaled up or down).
How this is communicated in specific is through the only good interprocess communication method that X supplies, namely X properties. In the specific case of icons, the _NET_WM_ICON property is what is used, and xprop can display the size information and an ASCII art summary of what each icon looks like. It's also possible to use some additional magic to read out the raw data from _NET_WM_ICON in a useful format; see, for example, this Stackoverflow question and its answers.

Beastie Bits

• Recent Security Innovations
• Old Unix books + Solaris
• Pro-Desktop - A Tiling Desktop Environment
• The Tar Pipe
• At least one vim trick you might not know

Feedback/Questions

• Johnny - listener feedback
• Brian - Questions
• Mark - ZFS Question

• Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv

This episode was brought to you by

Headlines

ALTQ removed from PF

• Kicking off our big PF episode...
• The classic packet queueing system, ALTQ, was recently removed from OpenBSD -current
• There will be a transitional phase between 5.5 and 5.6 where you can still use it by replacing the "queue" keyword with "oldqueue" in your pf.conf
• As of 5.6, due about six months from now, you'll have to change your ruleset to the new syntax if you're using it for bandwidth shaping
• After more than ten years, bandwidth queueing has matured quite a bit and we can finally put ALTQ to rest, in favor of the new queueing subsystem
• This doesn't affect FreeBSD, PCBSD, NetBSD or DragonflyBSD since all of their PFs are older and maintained separately. ***

FreeBSD Quarterly Status Report

• The quarterly status report from FreeBSD is out, detailing some of the project's ongoing tasks
• Some highlights include the first "stable" branch of ports, ARM improvements (including SMP), bhyve improvements, more work on the test suite, desktop improvements including the new vt console driver and UEFI booting support finally being added
• We've got some specific updates from the cluster admin team, core team, documentation team, portmgr team, email team and release engineering team
• LOTS of details and LOTS of topics to cover, give it a read ***

OpenBSD's OpenSSL rewrite continues with m2k14

• A mini OpenBSD hackathon begins in Morocco, Africa
• You can follow the changes in the -current CVS log, but a lot of work is mainly going towards the OpenSSL cleaning
• We've got two trip reports so far, hopefully we'll have some more to show you in a future episode
• You can see some of the more interesting quotes from the tear-down or see everything
• Apparently they are going to call the fork "LibreSSL" ....
• What were the OpenSSL developers thinking? The RSA private key was used to seed the entropy!
• We also got some mainstream news coverage and another post from Ted about the history of the fork
• Definitely consider donating to the OpenBSD foundation, this fork will benefit all the other BSDs too ***

NetBSD 6.1.4 and 6.0.5 released

• New updates for the 6.1 and 6.0 branches of NetBSD, focusing on bugfixes
• The main update is - of course - the heartbleed vulnerability
• Also includes fixes for other security issues and even a kernel panic... on Atari
• Patch your Ataris right now, this is serious business ***

Interview - Peter Hansteen - peter@bsdly.net / @pitrh

The Book of PF: 3rd edition

Tutorial

BSD Firewalls: PF

News Roundup

New Xorg now the default in FreeBSD

• For quite a while now, FreeBSD has had two versions of X11 in ports
• The older, stable version was the default, but you could install a newer one by having "WITH_NEW_XORG" in /etc/make.conf
• They've finally made the switch for 10-STABLE and 9-STABLE
• Check this wiki page for more info ***

GSoC-accepted BSD projects

• The Google Summer of Code team has got the list of accepted project proposals uploaded so we can see what's planned
• OpenBSD's list includes DHCP configuration parsing improvements, systemd replacements, porting capsicum, GPT and UEFI support, and modernizing the DHCP daemon
• The FreeBSD list was also posted
• Theirs includes porting FreeBSD to the Android emulator, CTF in the kernel debugger, improved unicode support, converting firewall rules to a C module, pkgng improvements, MicroBlaze support, PXE fixes, bhyve caching, bootsplash and lots more
• Good luck to all the students participating, hopefully they become full time BSD users ***

Complexity of FreeBSD VFS using ZFS as an example

• HybridCluster posted the second part of their VFS and ZFS series
• This new post has lots of technical details once again, definitely worth reading if you're a ZFS guy
• Of course, also watch episode 24 for our interview with HybridCluster - they do really interesting stuff ***

PCBSD weekly digest

• Preload has been ported over, it's a daemon that prefetches applications
• PCBSD is developing their own desktop environment, Lumina (there's also an FAQ)
• It's still in active development, but you can try it out by installing from ports
• We'll be showing a live demo of it in a few weeks (when development settles down a bit)
• Some kid in Australia subjects his poor mother to being on camera while she tries out PCBSD and gives her impressions of it ***