Headlines

The Earliest Unix Code: An Anniversary Source Code Release

What is it that runs the servers that hold our online world, be it the web or the cloud? What enables the mobile apps that are at the center of increasingly on-demand lives in the developed world and of mobile banking and messaging in the developing world? The answer is the operating system Unix and its many descendants: Linux, Android, BSD Unix, MacOS, iOS—the list goes on and on. Want to glimpse the Unix in your Mac? Open a Terminal window and enter “man roff” to view the Unix manual entry for an early text formatting program that lives within your operating system.

2019 marks the 50th anniversary of the start of Unix. In the summer of 1969, that same summer that saw humankind’s first steps on the surface of the Moon, computer scientists at the Bell Telephone Laboratories—most centrally Ken Thompson and Dennis Ritchie—began the construction of a new operating system, using a then-aging DEC PDP-7 computer at the labs.

This man sent the first online message 50 years ago

• As many of you have heard in the past, the first online message ever sent between two computers was "lo", just over 50 years ago, on Oct. 29, 1969.

It was supposed to say "log," but the computer sending the message — based at UCLA — crashed before the letter "g" was typed. A computer at Stanford 560 kilometres away was supposed to fill in the remaining characters "in," as in "log in."

• The CBC Radio show, “The Current” has a half-hour interview with the man who sent that message, Leonard Kleinrock, distinguished professor of computer science at UCLA

"The idea of the network was you could sit at one computer, log on through the network to a remote computer and use its services there,"

50 years later, the internet has become so ubiquitous that it has almost been rendered invisible. There's hardly an aspect in our daily lives that hasn't been touched and transformed by it.

Q: Take us back to that day 50 years ago. Did you have the sense that this was going to be something you'd be talking about a half a century later?

A: Well, yes and no. Four months before that message was sent, there was a press release that came out of UCLA in which it quotes me as describing what my vision for this network would become. Basically what it said is that this network would be always on, always available. Anybody with any device could get on at anytime from any location, and it would be invisible.

Well, what I missed ... was that this is going to become a social network. People talking to people. Not computers talking to computers, but [the] human element.

Q: Can you briefly explain what you were working on in that lab? Why were you trying to get computers to actually talk to one another?

A: As an MIT graduate student, years before, I recognized I was surrounded by computers and I realized there was no effective [or efficient] way for them to communicate. I did my dissertation, my research, on establishing a mathematical theory of how these networks would work. But there was no such network existing. AT&T said it won't work and, even if it does, we want nothing to do with it.

So I had to wait around for years until the Advanced Research Projects Agency within the Department of Defence decided they needed a network to connect together the computer scientists they were supervising and supporting.

Q: For all the promise of the internet, it has also developed some dark sides that I'm guessing pioneers like yourselves never anticipated.

A: We did not. I knew everybody on the internet at that time, and they were all well-behaved and they all believed in an open, shared free network. So we did not put in any security controls.

When the first spam email occurred, we began to see the dark side emerge as this network reached nefarious people sitting in basements with a high-speed connection, reaching out to millions of people instantaneously, at no cost in time or money, anonymously until all sorts of unpleasant events occurred, which we called the dark side.

But in those early days, I considered the network to be going through its teenage years. Hacking to spam, annoying kinds of effects. I thought that one day this network would mature and grow up. Well, in fact, it took a turn for the worse when nation states, organized crime and extremists came in and began to abuse the network in severe ways.

Q: Is there any part of you that regrets giving birth to this?

A: Absolutely not. The greater good is much more important.

News Roundup

How to use blacklistd(8) with NPF as a fail2ban replacement

blacklistd(8) provides an API that can be used by network daemons to communicate with a packet filter via a daemon to enforce opening and closing ports dynamically based on policy.

The interface to the packet filter is in /libexec/blacklistd-helper (this is currently designed for npf) and the configuration file (inspired from inetd.conf) is in etc/blacklistd.conf

Now, blacklistd(8) will require bpfjit(4) (Just-In-Time compiler for Berkeley Packet Filter) in order to properly work, in addition to, naturally, npf(7) as frontend and syslogd(8), as a backend to print diagnostic messages. Also remember npf shall rely on the npflog* virtual network interface to provide logging for tcpdump() to use.

Unfortunately (dont' ask me why ??) in 8.1 all the required kernel components are still not compiled by default in the GENERIC kernel (though they are in HEAD), and are rather provided as modules. Enabling NPF and blacklistd services would normally result in them being automatically loaded as root, but predictably on securelevel=1 this is not going to happen.

• FreeBSD’s handbook chapter on blacklistd

OpenBSD crossed 400,000 commits

Sometime in the last week OpenBSD crossed 400,000 commits (*) upon all our repositories since starting at 1995/10/18 08:37:01 Canada/Mountain. That's a lot of commits by a lot of amazing people.

(*) by one measure. Since the repository is so large and old, there are a variety of quirks including ChangeLog missing entries and branches not convertible to other repo forms, so measuring is hard. If you think you've got a great way of measuring, don't be so sure of yourself -- you may have overcounted or undercounted.

• Subject to the notes Theo made about under and over counting, FreeBSD should hit 1 million commits (base + ports + docs) some time in 2020
• NetBSD + pkgsrc are approaching 600,000, but of course pkgsrc covers other operating systems too

How to Install Bolt CMS with Nginx and Let's Encrypt on FreeBSD 12

Bolt is a sophisticated, lightweight and simple CMS built with PHP. It is released under the open-source MIT-license and source code is hosted as a public repository on Github. A bolt is a tool for Content Management, which strives to be as simple and straightforward as possible. It is quick to set up, easy to configure, uses elegant templates. Bolt is created using modern open-source libraries and is best suited to build sites in HTML5 with modern markup. In this tutorial, we will go through the Bolt CMS installation on FreeBSD 12 system by using Nginx as a web server, MySQL as a database server, and optionally you can secure the transport layer by using acme.sh client and Let's Encrypt certificate authority to add SSL support.

• Requirements
• The system requirements for Bolt are modest, and it should run on any fairly modern web server:
  • PHP version 5.5.9 or higher with the following common PHP extensions: pdo, mysqlnd, pgsql, openssl, curl, gd, intl, json, mbstring, opcache, posix, xml, fileinfo, exif, zip.
  • Access to SQLite (which comes bundled with PHP), or MySQL or PostgreSQL.
  • Apache with mod_rewrite enabled (.htaccess files) or Nginx (virtual host configuration covered below).
  • A minimum of 32MB of memory allocated to PHP.

hammer2 - Optimize hammer2 support threads and dispatch

Refactor the XOP groups in order to be able to queue strategy calls, whenever possible, to the same CPU as the issuer. This optimizes several cases and reduces unnecessary IPI traffic between cores. The next best thing to do would be to not queue certain XOPs to an H2 support thread at all, but I would like to keep the threads intact for later clustering work.

The best scaling case for this is when one has a large number of user threads doing I/O. One instance of a single-threaded program on an otherwise idle machine might see a slightly reduction in performance but at the same time we completely avoid unnecessarily spamming all cores in the system on the behalf of a single program, so overhead is also significantly lower.

This will tend to increase the number of H2 support threads since we need a certain degree of multiplication for domain separation.

This should significantly increase I/O performance for multi-threaded workloads.

You know, we might as well just run every network service over HTTPS/2 and build another six layers on top of that to appease the OSI 7-layer burrito guys

I've seen the writing on the wall, and while for now you can configure Firefox not to use DoH, I'm not confident enough to think it will remain that way. To that end, I've finally set up my own DoH server for use at Chez Boca. It only involved setting up my own CA to generate the appropriate certificates, install my CA certificate into Firefox, configure Apache to run over HTTP/2 (THANK YOU SO VERY XXXXX­XX MUCH GOOGLE FOR SHOVING THIS HTTP/2 XXXXX­XXX DOWN OUR THROATS!—no, I'm not bitter) and write a 150 line script that just queries my own local DNS, because, you know, it's more XXXXX­XX secure or some XXXXX­XXX reason like that.

Sigh.

Beastie Bits

• An Oral History of Unix
• NUMA Siloing in the FreeBSD Network Stack [pdf]
• EuroBSDCon 2019 videos available
• Barbie knows best
• For the #OpenBSD #e2k19 attendees. I did a pre visit today.
• Drawer Find
• Slides - Removing ROP Gadgets from OpenBSD - AsiaBSDCon 2019

Feedback/Questions

• Bostjan - Open source doesn't mean secure
• Malcolm - Allan is Correct.

• Michael - FreeNAS inside a Jail

  ---

• Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv

Headlines

ZFSonFreeBSD ports renamed OpenZFS

• The ZFS on FreeBSD project has renamed the userland and kernel ports from zol and zol-kmod to openzfs and openzfs-kmod
• The new versions from this week are IOCTL compatible with the command line tools in FreeBSD 12.0, so you can use the old userland with the new kernel module (although obviously not the new features)
• With the renaming it is easier to specify which kernel module you want to load in /boot/loader.conf: > zfs_load=”YES”
• or > openzfs_load=”YES”
• To load traditional or the newer version of ZFS
• The kmod still requires FreeBSD 12-stable or 13-current because it depends on the newer crypto support in the kernel for the ZFS native encryption feature. Allan is looking at ways to work around this, but it may not be practical.
• We would like to do an unofficial poll on how people would the userland to co-exist. Add a suffix to the new commands in /usr/local (zfs.new zpool.new or whatever). One idea i’ve had is to move the zfs and zpool commands to /libexec and make /sbin/zfs and /sbin/zpool a switcher script, that will call the base or ports version based on a config file (or just based on if the port is installed)
• For testing purposes, generally you should be fine as long as you don’t run ‘zpool upgrade’, which will make your pool only importable using the newer ZFS.
• For extra safety, you can create a ‘zpool checkpoint’, which will allow you to undo any changes that are made to the pool during your testing with the new openzfs tools. Note: the checkpoint will undo EVERYTHING. So don’t save new data you want to keep.
• Note: Checkpoints disable all freeing operations, to prevent any data from being overwritten so that you can re-import at the checkpoint and undo any operation (including zfs destroy-ing a dataset), so also be careful you don’t run out of space during testing.
• Please test and provide feedback.

How to use blacklistd(8) with NPF as a fail2ban replacement

• About blacklistd(8)

blacklistd(8) provides an API that can be used by network daemons to communicate with a packet filter via a daemon to enforce opening and closing ports dynamically based on policy.
The interface to the packet filter is in /libexec/blacklistd-helper (this is currently designed for npf) and the configuration file (inspired from inetd.conf) is in etc/blacklistd.conf
Now, blacklistd(8) will require bpfjit(4) (Just-In-Time compiler for Berkeley Packet Filter) in order to properly work, in addition to, naturally, npf(7) as frontend and syslogd(8), as a backend to print diagnostic messages. Also remember npf shall rely on the npflog* virtual network interface to provide logging for tcpdump() to use.
Unfortunately (dont' ask me why :P) in 8.1 all the required kernel components are still not compiled by default in the GENERIC kernel (though they are in HEAD), and are rather provided as modules. Enabling NPF and blacklistd services would normally result in them being automatically loaded as root, but predictably on securelevel=1 this is not going to happen

News Roundup

[WIP] raidz expansion, alpha preview 1

• Motivation and Context > This is a alpha-quality preview of RAID-Z expansion. This feature allows disks to be added one at a time to a RAID-Z group, expanding its capacity incrementally. This feature is especially useful for small pools (typically with only one RAID-Z group), where there isn't sufficient hardware to add capacity by adding a whole new RAID-Z group (typically doubling the number of disks). > For additional context as well as a design overview, see my short talk from the 2017 OpenZFS Developer Summit: slides video

Rant: running audio VU-meter increases my CO2 footprint

A couple months ago I noticed that the monitor on my workstation never power off anymore. Screensaver would go on, but DPMs (to do the poweroff) never kicked in.
I grovels the output of various tools that display DPMS settings, which as usual in Xorg were useless. Everybody said DPMS is on with a timeout. I even wrote my own C program to use every available Xlib API call and even the xscreensaver library calls. (should make it available) No go, everybody says that DPMs is on, enabled and set on a timeout. Didn’t matter whether I let xscreeensaver do the job or just the X11 server.
After a while I noticed that DPMS actually worked between starting my X11 server and starting all my clients. I have a minimal .xinitrc and start the actual session from a script, that is how I could notice. If I used a regular desktop login I wouldn’t have noticed. A server state bug was much more likely than a client bug.

• See the article for the rest...

XSAVE and compat32 kernel work for LLDB

Upstream describes LLDB as a next generation, high-performance debugger. It is built on top of LLVM/Clang toolchain, and features great integration with it. At the moment, it primarily supports debugging C, C++ and ObjC code, and there is interest in extending it to more languages.
In February, I have started working on LLDB, as contracted by the NetBSD Foundation. So far I've been working on reenabling continuous integration, squashing bugs, improving NetBSD core file support and lately extending NetBSD's ptrace interface to cover more register types. You can read more about that in my Apr 2019 report.
In May, I was primarily continuing the work on new ptrace interface. Besides that, I've found and fixed a bug in ptrace() compat32 code, pushed LLVM buildbot to ‘green’ status and found some upstream LLVM regressions. More below.

Some things about where icons for modern X applications come from

If you have a traditional window manager like fvwm, one of the things it can do is iconify X windows so that they turn into icons on the root window (which would often be called the 'desktop'). Even modern desktop environments that don't iconify programs to the root window (or their desktop) may have per-program icons for running programs in their dock or taskbar. If your window manager or desktop environment can do this, you might reasonably wonder where those icons come from by default.
Although I don't know how it was done in the early days of X, the modern standard for this is part of the Extended Window Manager Hints. In EWMH, applications give the window manager a number of possible icons, generally in different sizes, as ARGB bitmaps (instead of, say, SVG format). The window manager or desktop environment can then pick whichever icon size it likes best, taking into account things like the display resolution and so on, and display it however it wants to (in its original size or scaled up or down).
How this is communicated in specific is through the only good interprocess communication method that X supplies, namely X properties. In the specific case of icons, the _NET_WM_ICON property is what is used, and xprop can display the size information and an ASCII art summary of what each icon looks like. It's also possible to use some additional magic to read out the raw data from _NET_WM_ICON in a useful format; see, for example, this Stackoverflow question and its answers.

Beastie Bits

• Recent Security Innovations
• Old Unix books + Solaris
• Pro-Desktop - A Tiling Desktop Environment
• The Tar Pipe
• At least one vim trick you might not know

Feedback/Questions

• Johnny - listener feedback
• Brian - Questions
• Mark - ZFS Question

• Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv

This episode was brought to you by

Headlines

New PAE support in OpenBSD

• OpenBSD has just added Physical Address Extention support to the i386 architecture, but it's probably not what you'd think of when you hear the term
• In most operating systems, PAE's main advantage is to partially circumvent the 4GB memory limit on 32 bit platforms - this version isn't for that
• Instead, this change specifically allows the system to use the No-eXecute Bit of the processor for the userland, further hardening the in-place memory protections
• Other operating systems enable the CPU feature without doing anything to the page table entries, so they do get the available memory expansion, but don't get the potential security benefit
• As we discussed in a previous episode, the AMD64 platform already saw some major W^X kernel and userland improvements - the i386 kernel reworking will begin shortly
• Not all CPUs support this feature, but, if yours supports NX, this will improve upon the previous version of W^X that was already there
• The AMD64 improvements will be in 5.7, due out in just a couple days as of when we're recording this, but the i386 improvements will likely be in 5.8 ***

Booting Windows in bhyve

• Work on FreeBSD's bhyve continues, and a big addition is on the way
• Thus far, bhyve has only been able to boot operating systems with a serial console - no VGA, no graphics, no Windows
• This is finally changing, and a teasing screenshot of Windows Server was recently posted on Twitter
• Graphics emulation is still in the works; this image was taken by booting headless and using RDP
• A lot of the needed code is being committed to -CURRENT now, but the UEFI portion of it requires a bit more development (and the aim for that is around the time of BSDCan)
• Not a lot of details on the matter currently, but we'll be sure to bring you more info as it comes out
• Are you more interested in bhyve or Xen on FreeBSD? Email us your thoughts ***

MidnightBSD 0.6 released

• MidnightBSD is a smaller project we've not covered a lot on the show before
• It's an operating system that was forked from FreeBSD back in the 6.1 days, and their focus seems to be on ease-of-use
• They also have their own, smaller version of FreeBSD ports, called "mports"
• If you're already using it, this new version is mainly a security and bugfix release
• It syncs up with the most recent FreeBSD security patches and gets a lot of their ports closer to the latest versions
• You can check their site for more information about the project
• We're trying to get the lead developer to come on for an interview, but haven't heard anything back yet ***

OpenBSD rewrites the file utility

• We're all probably familiar with the traditional file command - it's been around since the 1970s
• For anyone who doesn't know, it's used to determine what type of file something actually is
• This tool doesn't see a lot of development these days, and it's had its share of security issues as well
• Some of those security issues remain unfixed in various BSDs even today, despite being publicly known for a while
• It's not uncommon for people to run file on random things they download from the internet, maybe even as root, and some of the previous bugs have allowed file to overwrite other files or execute code as the user running it
• When you think about it, file was technically designed to be used on untrusted files
• OpenBSD developer Nicholas Marriott, who also happens to be the author of tmux, decided it was time to do a complete rewrite - this time with modern coding practices and the usual OpenBSD scrutiny
• This new version will, by default, run as an unprivileged user with no shell, and in a systrace sandbox, strictly limiting what system calls can be made
• With these two things combined, it should drastically reduce the damage a malicious file could potentially do
• Ian Darwin, the original author of the utility, saw the commit and replied, in what may be a moment in BSD history to remember
• It'll be interesting to see if the other BSDs, OS X, Linux or other UNIXes consider adopting this implementation in the future - someone's already thrown together an unofficial portable version
• Coincidentally, the lead developer and current maintainer of file just happens to be our guest today… ***

Interview - Christos Zoulas - christos@netbsd.org

blacklistd and NetBSD advocacy

News Roundup

GSoC-accepted BSD projects

• The Google Summer of Code people have published a list of all the projects that got accepted this year, and both FreeBSD and OpenBSD are on that list
• FreeBSD's list includes: NE2000 device model in userspace for bhyve, updating Ficl in the bootloader, type-aware kernel virtual memory access for utilities, JIT compilation for firewalls, test cluster automation, Linux packages for pkgng, an mtree parsing and manipulation library, porting bhyve to ARM-based platforms, CD-ROM emulation in CTL, libc security extensions, gptzfsboot support for dynamically discovering BEs during startup, CubieBoard support, a bhyve version of the netmap virtual passthrough for VMs, PXE support for FreeBSD guests in bhyve and finally.. memory compression and deduplication
• OpenBSD's list includes: asynchronous USB transfer submission from userland, ARM SD/MMC & controller driver in libsa, improving USB userland tools and ioctl, automating module porting, implementing a KMS driver to the kernel and, wait for it... porting HAMMER FS to OpenBSD
• We'll be sure to keep you up to date on developments from both projects
• Hopefully the other BSDs will make the cut too next year ***

FreeBSD on the Gumstix Duovero

• If you're not familiar with the Gumstix Duovero, it's an dual core ARM-based computer-on-module
• They actually look more like a stick of RAM than a mini-computer
• This article shows you how to build a FreeBSD -CURRENT image to run on them, using crochet-freebsd
• If anyone has any interesting devices like this that they use BSD on, write up something about it and send it to us ***

EU study recommends OpenBSD

• A recent study by the European Parliament was published, explaining that more funding should go into critical open source projects and tools
• This is especially important, in all countries, after the mass surveillance documents came out
• "[...] the use of open source computer operating systems and applications reduces the risk of privacy intrusion by mass surveillance. Open source software is not error free, or less prone to errors than proprietary software, the experts write. But proprietary software does not allow constant inspection and scrutiny by a large community of experts."
• The report goes on to mention users becoming more and more security and privacy-aware, installing additional software to help protect themselves and their traffic from being spied on
• Alongside Qubes, a Linux distro focused on containment and isolation, OpenBSD got a special mention: "Proactive security and cryptography are two of the features highlighted in the product together with portability, standardisation and correctness. Its built-in cryptography and packet filter make OpenBSD suitable for use in the security industry, for example on firewalls, intrusion-detection systems and VPN gateways"
• Reddit, Undeadly and Hacker News also had some discussion, particularly about corporations giving back to the BSDs that they make use of in their infrastructure - something we've discussed with Voxer and M:Tier before ***

FreeBSD workflow with Git

• If you're interested in contributing to FreeBSD, but aren't a big fan of SVN, they have a Github mirror too
• This mailing list post talks about interacting between the official source repository and the Git mirror
• This makes it easy to get pull requests merged into the official tree, and encourages more developers to get involved ***

Feedback/Questions

• Sean writes in
• Bryan writes in
• Sean writes in
• Charles writes in ***