As of this year, I'm no longer a FreeBSD user. I've been a hold-out... finally converting over to Linux this year after years of denial. It was a very hard move for me.
The problem is that FreeBSD developers are still fighting wars of the 90's. By and large, with the welcome exception of Colin Percival (of http://www.tarsnap.com/ fame), they have not embraced virtualization. The only saving grace of FreeBSD is that it has adopted Sun's ZFS. This won't hold the fort for long though.
Have you ever used FreeBSD jails? I wish there was something as easy to use and flexible as that on Linux. The fact that each jail has minimal overhead and with dedupe on ZFS I can easily give every user their own "machine" while spending a lot less on hardware then a linux solution would require.
Having said that I also wish I could run FreeBSD on EC2 however I haven't put my money where my mouth is. I'm sure if you and many others wanted it badly enough you could sponsor development to fix any current issues. Also it would probably help if someone made a large business case to Amazon to ensure that things were targeted towards FreeBSD that would help too. (i.e. not needing an ext2 filesystem for the user provided kernel on micro instances, etc)
Absolute goodness of jails and the simplicity of making a small bootable readonly CDROM image made FreeBSD a clear winner for me in the late 1990's & early 2000's. In the 2010's... the world moves forward, now it's about having a standard virtual image.
The idea is very similar: you create a set of jails that behave as independent systems. Then every process started from a jail is tagged with a jail ID corresponding to the jail that the process was started from. All processes run in the same kernel and the jail ID gives the kernel just enough information to tell whether or not a process should have access to a given resource when sys calls are made.
I've never used OpenVZ so I don't know how they compare in practice but I can say that FreeBSD jails worked beautifully when I was using them heavily (about 2 years ago). They are super easy to manage and were as stable as you'd expect in FreeBSD (you'd have to be trying really hard to knock them down).
I'm actually thinking of moving to FreeBSD for the very reasons you're stating.
FreeBSD has:
* ZFS - which is miles ahead of btrfs for production use
* libcapsicum - something that isn't completely braindead like posix 'capabilities'
* racct and jails - cgroups is a mess, containerization isn't wanted from posts on lkml ( as it's 'not needed' - considering vps hosting is a multi-billion dollar industry ) and it needs quiet a bit of work.
> * ZFS - which is miles ahead of btrfs for production use
Can you explain in what way? btrfs is less mature, sure, but "enterprise Linux" vendors are about to start fully supporting in their next minor version update releases, and that is a good indicator of production use readiness, since they wouldn't do that if they weren't pretty sure it's not going to blow up in their face via an endless stream of bug reports.
> * racct and jails - cgroups is a mess
Thank you for sharing your opinion, an explanation of why it is a mess would be nice.
ZFS has l2arc, raid, scrubbing, dedup and it's stable, no enterprise will use btrfs in production for quite awhile due to bugs still being ironed out (read the mailing list).
Regarding cgroups? Accounting is bolted on effectively tracking tracked pages.
There is support for ZFS on Linux at the kernel level [1]. I'm not sure how stable it is (something on my TODO list), but I believe it's actually more current than what's in FreeBSD.
For jails on Linux there's jailkit [2].
I'm not sure if you've seen these, but I figured I'd offer them up as solutions that don't involve changing your platform!
For those who, like me, wonder why FreeBSD should embrace visualization (what's wrong with a 80x24 tty?): from the answers, I think clarkevans meant to write "virtualization". Maybe his spelling checker 'improved' that word?
Everytime I wanted to install FBSD on an old box I would always run into awful issues wioth acpi/acpm. All of the boxes I tried to install it on recently ran linux without a quirk.
I really want to get a fbsd box up and running so I can run phk's ntpns
The problem is that FreeBSD developers are still fighting wars of the 90's. By and large, with the welcome exception of Colin Percival (of http://www.tarsnap.com/ fame), they have not embraced virtualization. The only saving grace of FreeBSD is that it has adopted Sun's ZFS. This won't hold the fort for long though.
For BSD-on-EC2 see http://www.daemonology.net/freebsd-on-ec2/