BSD systems are the technological neighbors many of us meet daily, but few know much about. Martin Husemann of NetBSD explains, analyses, compares and refers to everything there is to know about BSD and NetBSD. Meet this fantastic operating system through another Monday’s interview on Unixmen.
Tell us a few things about you, how you got involved with NetBSD and
what exactly your position and your main activities in the project are.
About me: I am one of the “old farts” in the NetBSD community, born 1965, married, two kids. I received a master degree in computer science from the University of Paderborn (Diplominformatiker). After doing some
consultant jobs, I ended up doing 3D graphics (mostly on Windows) and CAD software. I ran my own company for 10 years, which recently got “swallowed” by a big customer for a new project – so now I am head of developement at ELECO Software GmbH, still doing 3D CAD stuff.
I started using NetBSD, mostly by coincidence, right after the project started in 1993. I was using a commercial unix variant at that time, plus a DOS/KA9Q based router connecting via ISDN to a nearby university. Two machines, where one could do – I wanted to work on a new ISDN stack, which would have been way too difficult without access to full kernel sources.
Later, when DSL became available at my home, I wrote the NetBSD pppoe(4) driver to make use of it. I reused my old ISDN router, which only had 4 MB ram and a 50 MHz cpu, and the existing userland solutions could not deliver (DSL) line speed on this machine – so I had to write something more efficient. And it turned out to be quite good.
I became portmaster for NetBSD/sparc64, which basically means: I am the default victim if something machine dependant needs to be added due to general machine independant enhancements, and if other sparc64 developers fail to agree on something technical, I would get the final vote. However, I can not recall this ever happening.
Additionally I served on the board of directors for four years. The board handles all the boring stuff, from taxes over legal issues to arranging annual meetings. Nothing fancy, for all the interesting things we have another group, called “the core team” – they are the technical leadership of the project.
Right now I am part of the release engineering team (which takes care to sometimes, but not too often, get a real release ready), and the security team.
I am also heavily involved in the automatic testing.
Finally, I am one of the mentors for a Google Summer of Code project again this year.
Many people in the Linux world often confuse open source BSD with GNU/Linux distributions, or don’t clearly know the exact difference between these two types of operating systems. Can you mention the main differencies that highlight the diversity of these two OSes?
I have full sympathy for this confusion. If you see a fully setup desktop computer running Linux and NetBSD, it might be hard to tell the differences withouth at least invoking a command line and closer checking. But then my mother runs Windows 7 on her notebook and uses Thunderbird, Firefox and LibreOffice – from a quick look the difference to my NetBSD notebook is not that big either.
This similarity of course is the result of sharing. NetBSD uses Xorg, just as Linux distributions do, and of course Gnome, KDE, Firefox are the same as well. In Linux land, you will probably divide your system into the distribution and the linux kernel. In NetBSD, we divide the system into three parts: the NetBSD kernel (which is not based on Linux, but the much older BSD unix kernel), the NetBSD userland (libc, all the
standard binaries like /bin/ls and daemons like initd), and “third party” packages (like Firefox). The separation between the “NetBSD userland” and the “third party” software is not fixed nor exactly clear: programs can move from one category to the other and back, or even exist in both. An example of this is postfix, the mail system. It is included in NetBSD “base” userland, because we think a mailer is an important part of a unix like system. However, it can also be installed as package, optionally including features not present in the base version.
To add to the confusion, parts of the NetBSD userland are optional – for example there is no need to install compiler, linker, binutils and similar tools on a server or a firewall (actually recommended practise is not to do so).
At a more technical layer, the main difference between “NetBSD userland” and “third party packages” is how we manage them. All programs in “NetBSD userland” are managed inside the actual NetBSD source tree and are (about once daily) (cross-)build for all supported machines/architectures.
The other applications are managed in a system called “pkgsrc”, which by the way is not NetBSD specific and is nowadays also used on other operating systems, like DragonFly BSD and Minix. In the best case a pkgsrc entry is just a small (like 10 lines) makefile, pointing to a URL where to download the original source code, and a few lines of meta information (like: what license it has, what other pkgsrc entries it depends on, whom to contact in case the package does not work, …). In other, slighly unlucky, cases (example: Firefox) it is a slightly larger collection of makefiles (totalling to, say, 200 lines) plus an extensive set of patches (firefox has more than 100!).
Now for someone used to linux distributions, the “src” part of “pkgsrc” is probably a strange thing. The pkgsrc goal is to provide an easy way to create the application “from source”. This, in many cases, has to be
done on a system running the targeted operating system on the right machine. Some packages can be cross build (like gcc), for others (like perl) this is completely impossible.
But for a user this is all irrelevant. When installing NetBSD they will want to select available applications and just get them – we call this “binary packages”, basically the result of pkgsrc for one package, tared and compressed.
We try to provide these binary packages ASAP for i386 and amd64. Other architectures take their time to follow (macppc, sparc64), and for others (vax, several of the embeded platforms) there is no official binary package build available. This does not mean they are not fully supported, but a user needing, say, gnome on a vax would have to build (i.e. compile) it locally from pkgsrc.
Now the puzzle resolves, and you probably see why we separate the daily build “NetBSD userland” from additional applications. We try to keep the “base system” small, but “good enough”. It should have what a typical unix install always had (compiler, text editor, mailer) and it includes some useful things (web server) that especially small, embedded systems can use but that would be a pain to compile locally from pkgsrc.
First thing a Linux user will miss is bash – the NetBSD /bin/sh is smaller, but still includes important usability features like command history and interactive command line editing. It is Posix compliant, however, it is not bash, and more complex scripts will notice the incompatibility.
What would you say is the special need in the operating systems market, that BSD systems in general fill with success? The percentage of people using BSD systems is low, but obviously there must be a good reason for them to make this choice.
The obvious answer here is “the license”, at least if you talk about commercial companies as users.
The BSD license is one of the first open source licenses, back from times when this term was not even coined, and it actually meant something slightly different: in the academic unix world, a lot of code sharing was always going on, and the Berkeley Software Distribution was collecting the most important things, and shipping tapes around (this was before even universities were properly connecteded to “the Net”).
The license was meant to encourage code sharing, contributions back to the authors, and code reuse.
Unfortunatley, since these tapes had the full unix source code on them, you needed a AT&T/Unix Labs source code license as well. Only after a long jusristic battle this state finally was cleared with the 4.4BSD
Lite 2 Release 2 in 1995 (with first free versions appearing already in 1993 in various variants, of which NetBSD and FreeBSD are the well known ones still around today).
Later, with more (but less drastic) juristical work the license was simplified and shrunken to one or two clause variants. Basically it only says (besides some BIG PRINT avoiding not applicable warranty claims): if you distribute this source, leave the copyright intact. If you distribute only binaries, provide the license in material you provide to your customers. That is all. Realy simple, really free. Now go and look at GPLv3 and find a lawyer who understands it…
While this is all extremely interesting for corporate users, it usually will not matter a lot to private users. There are, of course, a lot of technical differences, and differences in the release cycle. However, these differ quite a bit between various BSDs, and I can best answer for NetBSD.
My own personal motivation to run NetBSD on most of my computers is simple: one OS fits them all. This makes managing the different machines a lot easier. In case you wonder about the “most” above: I do not run NetBSD on my iphone (though lots of NetBSD code made it into iOS, and I am mentioned in the copyright list, same for Android), and doing windows software for a living I run windows on some machines, of course. I always planned to port one of the 3D browser plugins we produce at work to MacOS X, so I have an installation of that around as well (but never got around to do the porting).
So, someone want to have the simplicity of the BSD license, and needs “one OS to fit them all”, but still has to choose between various BSD systems like FreeBSD, OpenBSD and NetBSD. Why would one choose NetBSD instead of other BSD systems? What makes NetBSD technically unique?
I am not sure I would count OpenBSD in that list, but I would certainly include DragonFly. I have limited experience with OpenBSD, as the few times I tried to use it, I failed in the installer already. Must be a
mental problem on my side. The NetBSD installer, btw., is the most efficient one I know, even if I include various commercial OSes (MacOS X, Solaris, Windows).
My personal impression of OpenBSD is that it is mostly about hype about security. The system is slow, does not scale well to big SMP machines, and the community seems to have a strange attitude. Whenever I see
gutter language used on professional security related lists, it always is their great leader – this seem to bleed through. On the other hand, I know a few OpenBSD developers, and they are both great guys and technical
experts – I wish they would convert 😉
FreeBSD (and even more its cousin DragonFly) are very good choices, if you run i386/amd64 hardware. FreeBSD has, after being x86 only for quite some time, added a few other architectures again. They do not seem to be first class citizens though.
I run NetBSD on a VAX. There is no real other choice of a modern operating system for that machine. I run it (for testing purposes) on a m68k based old Macintosh. I run the same OS on my desktop, my notebook, my servers. This makes my life a lot easier.
NetBSD is the BSD with the slowest release cycle, the most conservative technical planning and long time support for old releases, as well as a strong goal on binary compatibility. This means you can update a server that has been running some of your proprietary code for years to a newer NetBSD version, getting all the security fixes, and it will still run your old binaries – of course, you may be able to recompile them, but there is no urgent need.
NetBSD is, at least starting with the upcoming NetBSD 6 release, most likely the best tested free operating system. We made a huge effort to create an automatic testing environment and are running it on different hardware and emulators automatically. We have been doing automatic builds for all supported architectures since a long time (crosscompiling, we already talked about this above). This helps to find machine dependant code that does not compile – and identifies changes that break only on some architectures (typical examples are signed vs. unsigned characters or printf formats). We build most of the source tree with quite high warning levels and -Werorr, so this all is a good first test.
The automatic test setup goes one step further: it creates a full build, and then uses qemu to simulate an installation on a fresh machine. It then reboots the emulated machine and does a full test run. Besides on qemu, we do this fully automated run (including install) on Xen (both with i386 and amd64 domU). This triggers slightly different bugs and also helps to separate bugs in the tests or in NetBSD from bugs in qemu.
On a few other architectures, we run the full test suite (without installation from scratch) manually on real hardware.
This all has helped us several times to quickly identify regressions and verify fixes. The full test suite is constantly evolving. You can see this comparing netbsd-6 test results (this is the not yet released NetBSD-6 branch) with -current results – there are already about 100 new test cases in -current.
Security and stability are the main virtues of NetBSD, making it a very valuable tool for corporate environment use. Can you give us some examples of use of NetBSD in real life, that highlight the power of it?
There are various systems out there that use NetBSD or parts of its source code. Some of them are public (like Force10 routers), some are not so well known (NetBSD is used by some printer manufacturers for their
laser printers). If you read the legalese fine print in some Sony or Nintendo products, you will notice that they use parts of the network or usb code from NetBSD. This is exactly what the BSD license tries to encourage: just use the best available code, verbatim if you like, in your commercial product. If you enhance something while porting/using it – please feed it back (but there is no force behind this part, as company legal departments get nervous and tend to misunderstand things).
But you were probably looking for NetBSD in its pure form – I can only provide examples from my limited personal experience here.
A co-worker runs a pretty heavy traffic amateur astronomy photo site on a NetBSD XEN domU with very minimalistic resources at a xen virtualization provider. I just checked, it has been up 557 days now – last reboot was for a kernel update, and next reboot will likely be to upgrade to NetBSD 6. It runs the typical services – apache for the web site, nsd for the primary nameserver, webalizer for the traffic charts.
At work we use a NetBSD server as file server and for version control. It backs itself up automatically over a VPN to other offices, storing the data on an encrypted partition on an allnet NAS system. It also acts as a firewall.
How do you manage to meet the raised security expectations that users of NetBSD have? Is there a security team working constantly on delivering security patch updates? Do you have any kind of collaboration with the security teams of other “enterprise environment”, or security specialized open source projects?
We have a security team with an on-call rotation, and a small team of security-officers, whose main task is to cryptographically sign security publications and release binaries. Our security team is cooperating with CERT, and similar groups in other open source operating systems.
The security team coordinates issues and relies on specialists in the affected areas to create and test fixes, then coordinates those commits with the release-engineering team to get pullups to all needed release
branches as soon as possible. Especially with issues under embargo until public disclosure, this requires bookkeeping and organization and usually works in a timely manner.
Since we auto-build all supported release branches for all archs, we have fixed binaries soon after pullup of the commits to the relevant branch.
We have a separate team working on security issues for pkgsrc, which usualy does not have to develop patches themself, but relies on upstream fixes, which then result either in an updated package (new version released upstream), or additional patches applied in pkgsrc to an unchanged upstream version.
How does NetBSD compare in key features, stability and performance of virtualization, storage managment and interoperability?
That is a lot of buzz words, which we all have on our roadmaps as main goals.
Stability and performance are major targets, and benchmarks regularly show us to play on par with the big ones (Linux, FreeBSD and Solaris). Of course, as benchmarks are just like statistics, you can always find
some where one system is better than the other and vice versa. We take such results serious, and it has often happened that a new benchmark published showing deficiencies in NetBSD caused fixes to appear within a few days and correct this. We also have long term projects ongoing that are aiming to resolve performance bottlenecks already identified in parts of our kernel.
NetBSD aims to be standards compliant (as long as the standards make sense), which covers interoperability as well. One sour point in standards compliance is the collation support of locales – currently we just do not support that, because we have not yet settled a dispute on the optimal implementation. Other operating systems have picked something “good enough” for now. We should have done as well, but it seems we are now (finally!) pretty close to fixing it for real.
We have had XEN support in the source tree since 2004, and it still is regularly enhanced. We use it as part of our test setup, as well as on some NetBSD.org servers.
How difficult is it for you to support 57 hardware platforms and 15 different processor architectures?
The count is tricky. We do build 60 binary platforms for NetBSD-6 now (see here), some of which are the same, but build for dual endianess (evbmips el and eb), so that we subtract 3 and come up with 57.
There are a few difficulties involved, at different layers.
The most complex one is the compiler/toolchain layer. By default we use gcc/binutils. Other compilers like llvm or ppc are also supported, but do not cover all architectures (yet), or only partly work. Sometimes,
there is no working modern compiler available – this is how we lost the playstation2 port. We just had to downgrade the VAX port to gcc-4.1.3, since there are too many codegen/optimizer bugs for this architecture in gcc 4.5.3 (which we use on other architectures) – but we hope to fix gcc in -current.
At the userland level, it is not very difficult any more, since all abstractions needed are in place already. Errors still happen (for example if somebody uses printf with %d for a size_t argument, instead of %zu), but compiler warnings usually will catch them and break the automatic build. Also our developers are educated with portability in mind, and easily find someone to ask if they need help.
Inside the kernel, the challenge is ever ongoing. We have early on adopted abstractions, e.g. for DMA, and see that most other operating systems nowadays do something similar. Interestingly, the main problem nowadays is with i386/amd64 cpus and modern graphics drivers. We are playing catchup with the Xorg developers, who mostly develop on Linux and do not consider portability a main goal. Thinks like KMS and GEM are coded very Linux centric, and also a quickly moving target. We are struggling hard to provide an alternative usable for all relevant NetBSD platforms, and are currently trying to join forces in this area with
the DragonFly project.
What can you tell us about NetBSD in portable devices?
As long as the core has a MMU, everything should be ready. We have NAND flash support, a flash file system, power management and the kernel is configurable for tiny environments. We have something similar to “busybox” that we use in standard builds for a set of /rescue binaries, all statically linked into a single blob – so for example if you manage to break your dynamic libc.so and neither /bin/ls nor /bin/rm or /sbin/mount (to mount your backup from the NFS server) work, you just use /rescue/ls, /rescue/rm or /rescue/mount.
We do not support cpu cores without memory management unit.
Would you say that NetBSD is suitable for desktop users? Are things like the Kernel authorization framework and other security oriented tools standing in the way of doing simple daily tasks on a computer?
This reminds me of a Dilbert comic and no, NetBSD is not at all like this.
The security tools are just there to protect your data, you usually will not even notice them, if you do not actively use them. Some of them will run nightly in the background and mail you about unusual events, but they will not interfere with daily work.
The kauth framework is an internal subsystem in the kernel and only leaks to userland if you configure special security models (the default being traditional Unix, “root may do anything”). The authorization is separated (inside the kernel) from the subsystem needing to allow/deny access (for example the filesystem) to an abstract “security model”, which is replacable for special uses. Of course, the administrator can lock
down the systems, so no future modifications can be made. As a normal user, in a standard install, you should not even notice the availability of this abstraction/indirection.
On your desktop use, there may be problems – I bet your readers all hate flash websites (we can run the Linux binary from Adobe on NetBSD). Another issue is 3D graphics (very sour point for me personally, given
the personal background I described in the introduction), and on some graphics cards video display – see the Xorg vs. KMS/GEM statements above, I hope we will make good progress on this front together with the DragonFly developers pretty soon.
Can you say a few things about the community around the project, and how exactly a serious OS like NetBSD is developed in collaboration with many people around the world?
NetBSD is developed by The NetBSD Foundation, Inc., a 501(c)(3) classified non-profit organization (for the non-US readers: this basically means that donations you make to the NetBSD foundation from money earned inside the US are tax-deductible).
The Foundation owns the servers and some copyrights. The developers are “members” in this foundation, and have a voting right for the board of directors. We just had our annual meeting, and according to the numbers presented there are 181 developers with active commit rights right now, plus 112 members which (for security reasons) have disabled their commit access right now but expect to become active again in the future.
Collaboration is done mostly on mailing lists. We try to avoid cabals in the dark – everything should be proposed and discussed on mailing lists until consensus is reached. Only if that is impossible, the core team
is invoked and either defers the proposal for further discussion, or selects one of the discussed options. We try to keep the number of such core decisions minimal. Sometimes they are necessary because they are like choosing names – it does not realy matter, somebody just has to flip a coin.
The mailing lists are quite specialiced (i.e. we have way too many of them). According to my experience, the community on the mailing lists is very friendly and helpful. Even obvious newbie questions get friendly answers (though sometimes only a short pointer to the proper documentation). You have good chances to find someone who knows about your problem on the matching list, and the people will try to avoid guesses if they are not sure. This all makes for a friendly environment and a good signal/noise ratio.
What are the new features and additions we should expect on the upcoming version 6.0?
The release date is not yet fixed. We just anounced the second public beta, which hopefully is the last – so in a few weeks a release candidate should follow, and the release happen soon after that.
There is a new features list in this blog post
The highlights are:
– we switched time_t to 64bit, which is not as easy as it sounds, since we also kept bianry compatibility, i.e. you can run old binaries with newer kernel and shared libraries, and they will still work. A lot of interfaces had to be versioned for this to properly work. This solved the year-2038 problem for us.
– we replaced the old LKM kernel modules by a new framework and modularized a lot more parts of the kernel.
– we now have netpgp, a bsd licensed implementation of PGP (of course, if you prefer, you can still use gnupg from pkgsrc).
– we gained logical volume manager functionality
– lots of improvements for XEN, including SMP and suspend/resume support
– lots of improvements for Linux compatibility, so for example we can now run the current version of Flash for Linux.
– support for 64 bit MIPS processors
– in kernel iSCSI initiator
– Thread Local Storage support for most platforms
– lots of new machines supported for sparc64
– flash and nand subsystem, plus a file system for it (CHFS)
– sqlite3 is now in base, and used for apropos(1)
We also updated many of the xorg components and of the third party software like gdb, gcc and binutils
Thanks Martin, it’s been great getting to know some details about the various aspects of the BSD world. See you everyone next Monday with a “gaming” interview…
