Jails on FreeBSD(ogris.de)
ogris.de
Jails on FreeBSD
https://ogris.de/howtos/freebsd-jails.html
116 comments
> Can jails be used to run subprocesses in the normal filesystem, but with a different network environment (for example making a given command run its net traffic through TAP)?
Jails are basically like a VM-like/light: it appears like jail is its own system. Not quite as heavy as an actual VM under (e.g.) Qemu with virtualized hardware. A jail can have its own network stack:
* https://klarasystems.com/articles/virtualize-your-network-on...
You can then create a 'virtual patch cable' between the host and the jail and send one side of the 'cable' to the jail and do routing and stuff on the host:
* https://man.freebsd.org/cgi/man.cgi?epair
But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup.
> Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process?
Yes:
* https://wiki.freebsd.org/JailResourceLimits
> Can live processes be moved into or out of a jail?
No: jails are VM-like in functionality.
> Can jails be used to make a process think it's running as a different user?
The jail, being VM-like, would have its own passwd.
Jails are basically like a VM-like/light: it appears like jail is its own system. Not quite as heavy as an actual VM under (e.g.) Qemu with virtualized hardware. A jail can have its own network stack:
* https://klarasystems.com/articles/virtualize-your-network-on...
You can then create a 'virtual patch cable' between the host and the jail and send one side of the 'cable' to the jail and do routing and stuff on the host:
* https://man.freebsd.org/cgi/man.cgi?epair
But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup.
> Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process?
Yes:
* https://wiki.freebsd.org/JailResourceLimits
> Can live processes be moved into or out of a jail?
No: jails are VM-like in functionality.
> Can jails be used to make a process think it's running as a different user?
The jail, being VM-like, would have its own passwd.
> Jails are basically like a VM-like/light […]
With all due respect, they are not.
The definition of a «virtual machine» is a settled matter, and the «M» in «VM» is important and is the differentiator: it allows one to run a different operating system kernel on the same host under the auspices of a hardware or a software supervisor (somewhat less of a defining feature).
Neither jails nor cgroups possess such a property, and both restrict users to the same host operating system kernel and its version, so none of them are «VM-like/light» irregardless of the semantic interpretation of the «-like» suffix.
VM-like functionality is provided by a different OS subsystem in both, Linux and FreeBSD, kernels.
With all due respect, they are not.
The definition of a «virtual machine» is a settled matter, and the «M» in «VM» is important and is the differentiator: it allows one to run a different operating system kernel on the same host under the auspices of a hardware or a software supervisor (somewhat less of a defining feature).
Neither jails nor cgroups possess such a property, and both restrict users to the same host operating system kernel and its version, so none of them are «VM-like/light» irregardless of the semantic interpretation of the «-like» suffix.
VM-like functionality is provided by a different OS subsystem in both, Linux and FreeBSD, kernels.
>> Can live processes be moved into or out of a jail?
>No: jails are VM-like in functionality.
Could this be implemented though if moving between sufficiently similar operating systems?
Seems like might be useful to imprison a suspicious acting process or to release once seen to be safe.
>No: jails are VM-like in functionality.
Could this be implemented though if moving between sufficiently similar operating systems?
Seems like might be useful to imprison a suspicious acting process or to release once seen to be safe.
Calling "VM-like" is not helpful because containers have also been called that, it also doesn't explain anything. Is it running a separate kernal image? Does it do hypercalls to the host kernel? If not then I wouldn't call it VM-like.
The low-level details and fiddly knobs matter, not how it looks like from userspace when all is done.
Based on the jail(8) manpage I get the impression that jails are more like all the namespace and cgroup things rolled into a single entity. Which I guess makes it more difficult to use them incorrectly. But it also prevents other uses such as the one that jmillikin has in mind.
The low-level details and fiddly knobs matter, not how it looks like from userspace when all is done.
Based on the jail(8) manpage I get the impression that jails are more like all the namespace and cgroup things rolled into a single entity. Which I guess makes it more difficult to use them incorrectly. But it also prevents other uses such as the one that jmillikin has in mind.
It's called VM-like because:
- Often boots user land like a VM (i.e. PID 1 is not the process you want to run, but whatever metal or VM freebsd runs as PID 1)
- Normally has its own network stack (VIMAGE)
- Normally runs a bunch of background services like a regular VM or on-metal would.
- Normally has the entire copy of user-land
- I've never seen short-lived jails either - you make a base dataset, clone it, but after that, you just run `freebsd-update` like you would in VM.
The majority of jail users treat them like lightweight VMs, only difference between VM and Jail from consumer standpoint: shared kernel, access to a subset of host's FS (sharing host FS to VMs in freebsd is not as easy as it is on linux).
To be clear, it's possible to use Jails like Docker, there is just no good tooling to do it. People would yell that whatever we have for jails is all we need with foam from their mouths every time you mention it.
- Often boots user land like a VM (i.e. PID 1 is not the process you want to run, but whatever metal or VM freebsd runs as PID 1)
- Normally has its own network stack (VIMAGE)
- Normally runs a bunch of background services like a regular VM or on-metal would.
- Normally has the entire copy of user-land
- I've never seen short-lived jails either - you make a base dataset, clone it, but after that, you just run `freebsd-update` like you would in VM.
The majority of jail users treat them like lightweight VMs, only difference between VM and Jail from consumer standpoint: shared kernel, access to a subset of host's FS (sharing host FS to VMs in freebsd is not as easy as it is on linux).
To be clear, it's possible to use Jails like Docker, there is just no good tooling to do it. People would yell that whatever we have for jails is all we need with foam from their mouths every time you mention it.
That's just saying how people tend to use it, not what the technology does. Namespaces/Cgroups can be used both ways, to isolate a single process (or even partially-isolate individual threads) or entire userland stacks.
The question that was asked what the technology can do, not how it's being used. What are the primitives, the atoms its made of.
The question that was asked what the technology can do, not how it's being used. What are the primitives, the atoms its made of.
I think you need to look into usage of "-like" suffix.
> That's just saying how people tend to use it
People tend to use words that describe how they use things to describe what they are.
People tend to use words that describe how they use things to describe what they are.
Well, take a car as an example.
The question is not what you use a car for, but rather how the car is built.
The question is not what you use a car for, but rather how the car is built.
Car is a carriage-like mode of transportation.
The question was "Calling "VM-like" is not helpful because containers have also been called that, it also doesn't explain anything."
I've provided examples why "VM-like" is used to describe jails.
The question was "Calling "VM-like" is not helpful because containers have also been called that, it also doesn't explain anything."
I've provided examples why "VM-like" is used to describe jails.
> But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup.
Technically, you don't have to. It's just because jails don't have convenient tooling around them (like docker or podman) it's easier to just boot it up like a normal system.
Technically, you don't have to. It's just because jails don't have convenient tooling around them (like docker or podman) it's easier to just boot it up like a normal system.
> But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup.
It's an option to run jails with a whole init and everything; then it's like a separate host. But you can also just run stuff in the jail, without a whole everything. I currently run two daemons in a vnet jail to get a separate network for those daemons, but chrooted to /, because they don't need a separate filesystem. At my last job, we ran a TLS termination proxy chrooted to a very limited directory, because we didn't trust OpenSSL after Heartbleed; the chroot had just the executable, ld-elf and the libraries it loaded, unix sockets to communicate with the origin server, and logfiles, most of which was chflags schg.
It's an option to run jails with a whole init and everything; then it's like a separate host. But you can also just run stuff in the jail, without a whole everything. I currently run two daemons in a vnet jail to get a separate network for those daemons, but chrooted to /, because they don't need a separate filesystem. At my last job, we ran a TLS termination proxy chrooted to a very limited directory, because we didn't trust OpenSSL after Heartbleed; the chroot had just the executable, ld-elf and the libraries it loaded, unix sockets to communicate with the origin server, and logfiles, most of which was chflags schg.
> I currently run two daemons in a vnet jail to get a separate network for those daemons, but chrooted to /
When you do this, do the jails/chroots act like separate overlays on top of / to the daemons?
For example, if you use systemd-nspawn or systemd-run, you can run daemons in containers that get their own view of /, but writes take place in separate overlay file systems.
I'm curious how that compares with what you're doing with jails.
When you do this, do the jails/chroots act like separate overlays on top of / to the daemons?
For example, if you use systemd-nspawn or systemd-run, you can run daemons in containers that get their own view of /, but writes take place in separate overlay file systems.
I'm curious how that compares with what you're doing with jails.
No, my jailed deamons don't have an overlay, writes manifest to the filesystem the same as if they weren't jailed (which is good for me). Someone elsewhere described a vps they user which did have an overlay system, but I haven't used that.
You can run Linux containers with their own systemd's (obviously, they have their own PID 1). And, of course, that doesn't make them VMs.
If you don't trust a component running inside a Jail or a Linux container, you shouldn't be running it there; the kernel attack surface is big. At that point you've adopted the same security model as a phone jailbreaker has (except you don't get to dedicate hardware to the anti-jailbreaking problem the way the phone vendors do).
If you don't trust a component running inside a Jail or a Linux container, you shouldn't be running it there; the kernel attack surface is big. At that point you've adopted the same security model as a phone jailbreaker has (except you don't get to dedicate hardware to the anti-jailbreaking problem the way the phone vendors do).
Well what are you going to do when you have to run something you don't trust?
Can't run TLS termination on an isolated host, because then the traffic to the origin goes over the network, which you also don't trust.
Don't want to run OpenSSL integrated into the daemon, because OpenSSL is garbage.
Couldn't run anything else in the immediate aftermath of Heartbleed, because GnuTLS was worse, and LibreSSL and BoringSSL hadn't been released yet.
Can't run TLS termination on an isolated host, because then the traffic to the origin goes over the network, which you also don't trust.
Don't want to run OpenSSL integrated into the daemon, because OpenSSL is garbage.
Couldn't run anything else in the immediate aftermath of Heartbleed, because GnuTLS was worse, and LibreSSL and BoringSSL hadn't been released yet.
If you really don't trust it, you run it inside a virtual machine.
The truth is, you really do trust OpenSSL. You just trust it less than other things. That's fine! Layer controls on top of it. In Linux-land, this is the point where you'd start thinking about things like seccomp-bpf.
A truly untrusted workload is, like, a compute job you've accepted from a SAAS customer; it's arms-length multitenancy. You can't share kernels in that situation.
The truth is, you really do trust OpenSSL. You just trust it less than other things. That's fine! Layer controls on top of it. In Linux-land, this is the point where you'd start thinking about things like seccomp-bpf.
A truly untrusted workload is, like, a compute job you've accepted from a SAAS customer; it's arms-length multitenancy. You can't share kernels in that situation.
Worth pointing out that VM escape is a thing too: https://en.wikipedia.org/wiki/Virtual_machine_escape
It is. It's a much smaller attack surface.
I'm unconvinced it is. All of those virtual hardware drivers require their own code, often at kernel privilege. Hypervisor functionality at the processor level helps but is a huge attack surface in itself. X86 "rings" are under tested because it's hard to - many violations just kill your whole VM, but more importantly there's a ton of instructions that access "physical" memory and the state of the page mapper, even at the ring-0 level, matters a lot.
I do think VMs are likely more secure than containers in a cloud environment, because of course at that level you have both problems, but I don't believe that.the number of vulnerabilities found at the VM layer is at all reflective of their actual vulnerability.
I do think VMs are likely more secure than containers in a cloud environment, because of course at that level you have both problems, but I don't believe that.the number of vulnerabilities found at the VM layer is at all reflective of their actual vulnerability.
It simply is. Even in a heavyweight VM with lots of hardware support, those hardware drivers are a tiny fraction of the user/kernel interface, no matter how you choose to count it (lines of code, number of foreign calls, number of exposed modules).
If you don't want to derive this axiomatically, fair enough: count vulnerabilities. The tally you're looking for is every Linux LPE versus every Linux KVM escape.
If you don't want to derive this axiomatically, fair enough: count vulnerabilities. The tally you're looking for is every Linux LPE versus every Linux KVM escape.
Jails are not VM-like. The defining attribute of a VM is that it runs its own kernel. FreeBSD has a virtualization system: it's called bhyve.
> The defining attribute of a VM is that it runs its own kernel.
of a VM.
The difference between "VM" and "VM-like" is the trailing modifier "-like". This means something is similar, but not the same thing. If it looks like a duck, but is made of rubber, we call it a "rubber ducky." If it looks like a duck, is wearing a blue shirt, has a speech impediment, and is not wearing any pants, we call it Donald Duck.
Thus the fact that jails, and docker look like VMs because they have their own PID 1, their own file system, their own slice of memory/cpu/IO/network; that is why people are, accurately, using the descriptor "VM-like" to describe them. People understand that docker is not a true virtual machine because it's not running its own kernel. In cases where it matters, pedantry between true VM vs fake VM, or VM-like (for eg security isolation between VM and docker container) is crucial, but most discussions where "VM-like" is used to help people understand OpenVZ, LXC, jails, docker, cgroups, etc aren't focused on the possibility of an RCE in the container escaping the container, but helping people understand what a container even is in the first place.
Or to put it another way, strcmp("VM", "VM-like") != 0.
of a VM.
The difference between "VM" and "VM-like" is the trailing modifier "-like". This means something is similar, but not the same thing. If it looks like a duck, but is made of rubber, we call it a "rubber ducky." If it looks like a duck, is wearing a blue shirt, has a speech impediment, and is not wearing any pants, we call it Donald Duck.
Thus the fact that jails, and docker look like VMs because they have their own PID 1, their own file system, their own slice of memory/cpu/IO/network; that is why people are, accurately, using the descriptor "VM-like" to describe them. People understand that docker is not a true virtual machine because it's not running its own kernel. In cases where it matters, pedantry between true VM vs fake VM, or VM-like (for eg security isolation between VM and docker container) is crucial, but most discussions where "VM-like" is used to help people understand OpenVZ, LXC, jails, docker, cgroups, etc aren't focused on the possibility of an RCE in the container escaping the container, but helping people understand what a container even is in the first place.
Or to put it another way, strcmp("VM", "VM-like") != 0.
If VM "like" means "doesn't do the one thing that VMs do aka run a separate kernel" it's not a very useful term
I believe I recall there being a period quite a while ago when things like LXC and OpenVZ were described/marketed as being "VM-like" around the same time that jails were popularized.
It feels like jails are still stuck with being described using older terms even after Linux containers and the distinction between them and VMs were mainstreamed.
It feels like jails are still stuck with being described using older terms even after Linux containers and the distinction between them and VMs were mainstreamed.
I don't know about any of that. LXC containers are also not VM-like.
First of all - the Namespaces and Cgroups on Linux are not about security. They are about features.
FreeBSD Jails provide both secure isolation from the host and also provide the separated network namespace and resources with rctl(8) when needed.
To have Linux containers separated and secured you need additional layer for that - like SELinux or AppArmor. Only then Docker/Podman/other/... Linux containers are isolated and secure.
Regards.
FreeBSD Jails provide both secure isolation from the host and also provide the separated network namespace and resources with rctl(8) when needed.
To have Linux containers separated and secured you need additional layer for that - like SELinux or AppArmor. Only then Docker/Podman/other/... Linux containers are isolated and secure.
Regards.
You're telling those namespace / cgroup features have nothing to do with security? Not in any way, shape, or form? I'm a little skeptical.
Namespaces and cgroups provide resource accounting and some limited isolation between trusted workloads. They're not considered a sandbox or security boundary because the processes have full access to the Linux kernel APIs, which are not well-hardened.
To provide a security boundary between Linux processes, it's currently considered best practice to use something like Firecracker or gVisor.
To provide a security boundary between Linux processes, it's currently considered best practice to use something like Firecracker or gVisor.
I agree containers are more for ease of deployment, not security. But security is about shades of gray… limited isolation is one of those.
Underrated post.
Hate to nitpick, but a comparison to bare namespaces wouldn't be apt, either, as Docker et al use more features than just namespaces in their container implementations, like control groups, union/overlay/snapshot file systems, nftables, etc.
For several of those questions about jails, the answer would be the same if you were just evaluating the questions against namespaces by itself.
For several of those questions about jails, the answer would be the same if you were just evaluating the questions against namespaces by itself.
> Can live processes be moved into or out of a jail?
That might be possible on DragonflyBSD. They have support for freezing a running process to disk and restoring the process again (potentially on another machine) [1], and they have support for jails, so I don't see why you wouldn't be able to freeze a running process on the host and restoring it in the jail or vice versa.
[1] https://www.dragonflybsd.org/features/#index9h2
That might be possible on DragonflyBSD. They have support for freezing a running process to disk and restoring the process again (potentially on another machine) [1], and they have support for jails, so I don't see why you wouldn't be able to freeze a running process on the host and restoring it in the jail or vice versa.
[1] https://www.dragonflybsd.org/features/#index9h2
The article is about how to set up jails on FreeBSD.
The article has nothing to do with anything you are talking or asking about.
> * Can jails be used to run subprocesses in the normal filesystem, but with a different network environment (for example making a given command run its net traffic through TAP)?
Yes, you can run a vnet jail chrooted to /; same filesystem as the host, but a separate network system (you have to setup the network for the jail at this point; but I imagine the jail tools help with that). This is a fine use for jails. I currently run a vnet jail chrooted to / to do some crazy network stuff, but I just need a separate network, no other separation.
> * Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process?
I haven't used it, but it looks like yes/maybe with the rctl subsystem. This allows limits on lots of things, include memory of several types, cpu of a few types, filesystem io in bytes per second and operations per second, number of threads. Valid subjects for resource limits are process, user, loginclass and jail. Based on the manual, this doesn't let you limit network as you wanted, or place limits on threads within a process like you wanted. If you run a vnet jail, you can potentially set networking limits in other ways, but only if you pass virtual interfaces (such as epair, or taptun) to the jail rather than physical interfaces. This is a reasonable use for jails, but you might not need a jail for this?
> * Can live processes be moved into or out of a jail?
A process can move itself into a jail, but only if it's superuser. This is generally used administratively. Jailed processes can not be moved out of the jail, afaik; when the jail is destroyed, all processes within are killed. This is not within the design scope of jails.
> * Can jails be used to make a process think it's running as a different user?
I don't think so, but I'd use something with LD_PRELOAD to override getuid/geteuid for this; but I think I must not understand the question. You can certainly have entirely different users inside jail vs on the host?
Yes, you can run a vnet jail chrooted to /; same filesystem as the host, but a separate network system (you have to setup the network for the jail at this point; but I imagine the jail tools help with that). This is a fine use for jails. I currently run a vnet jail chrooted to / to do some crazy network stuff, but I just need a separate network, no other separation.
> * Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process?
I haven't used it, but it looks like yes/maybe with the rctl subsystem. This allows limits on lots of things, include memory of several types, cpu of a few types, filesystem io in bytes per second and operations per second, number of threads. Valid subjects for resource limits are process, user, loginclass and jail. Based on the manual, this doesn't let you limit network as you wanted, or place limits on threads within a process like you wanted. If you run a vnet jail, you can potentially set networking limits in other ways, but only if you pass virtual interfaces (such as epair, or taptun) to the jail rather than physical interfaces. This is a reasonable use for jails, but you might not need a jail for this?
> * Can live processes be moved into or out of a jail?
A process can move itself into a jail, but only if it's superuser. This is generally used administratively. Jailed processes can not be moved out of the jail, afaik; when the jail is destroyed, all processes within are killed. This is not within the design scope of jails.
> * Can jails be used to make a process think it's running as a different user?
I don't think so, but I'd use something with LD_PRELOAD to override getuid/geteuid for this; but I think I must not understand the question. You can certainly have entirely different users inside jail vs on the host?
Let me chime in and say, Michael W Lucas has an awesome book about FreeBSD Jails. I bought it recently and I read the whole thing and it helped me a lot. I still had to figure some things out by myself because the book is for a slightly older version of FreeBSD. But it is an awesome book.
I use only the tools included in base system for setting up my jails. No “ezjail” or anything.
If you read his whole book you will see how it might be the correct choice to just do it yourself. Depending on what you want to do etc.
For me I am definitely much better off having set it up myself with the help of mwl’s book.
https://www.freebsdmall.com/cgi-bin/fm/bsdmjails
Buy the physical copy of the book.
PS: Use vnet interfaces for most of your jails.
I use only the tools included in base system for setting up my jails. No “ezjail” or anything.
If you read his whole book you will see how it might be the correct choice to just do it yourself. Depending on what you want to do etc.
For me I am definitely much better off having set it up myself with the help of mwl’s book.
https://www.freebsdmall.com/cgi-bin/fm/bsdmjails
Buy the physical copy of the book.
PS: Use vnet interfaces for most of your jails.
All of Michael W Lucas' books are great, they're so easy to digest.
I upvoted this because I have a lot of love for FreeBSD and I think Jails are great.
However, I really wish the FreeBSD folks would educate themselves a bit more about what is actually available on Linux and how those options compare to FreeBSD Jails.
However, I really wish the FreeBSD folks would educate themselves a bit more about what is actually available on Linux and how those options compare to FreeBSD Jails.
And vice versa. I really wish the Linux folk would educate themselves a bit more about what is actually available on FreeBSD.
Why would they? What difference would it make to them?
Well, they might realise their use case it’s better suited to FreeBSD. But probably not better-enough to switch OS.
But mainly they might make make more educated comments. Sometimes, and maybe this is more a historic thing, the Linux advocates can get a bit carried away.
But mainly they might make make more educated comments. Sometimes, and maybe this is more a historic thing, the Linux advocates can get a bit carried away.
Who are you referring to? Linux or FreeBsd people? I guess both should look to each others'. But it's common to see this in FLOSS, right? Goes beyond namespaces/ isolation.
I don't mean to sound snide, I'm just saying: if your stack is Linux and the container ecosystem (for instance, if your stack is K8s) what would be in it for you to investigate Jails?
The biggest advantage is an abstract one; it's usually a good idea to have some idea of different ways to solve a problem even if you're going to be mostly (or even exclusively) focused on one particular way. "Narrowness of experience leads to narrowness of imagination" and that kind of stuff.
For example years ago I started runit in a container to run multiple processes (multiple containers was a bit tricky for operational reasons) and some people were surprised: "oh wait, you can do that?" Who knows if my experience with jails contributed to this "idea" (if you can even call it that), but it probably didn't hurt.
For example years ago I started runit in a container to run multiple processes (multiple containers was a bit tricky for operational reasons) and some people were surprised: "oh wait, you can do that?" Who knows if my experience with jails contributed to this "idea" (if you can even call it that), but it probably didn't hurt.
If your stack is K8s and you actually need K8s then none. Most people using K8s don’t actually need it and are practicing resume driven development.
You get that resume-driven development is a real thing, right? It's not irrational. Improving your future prospects is a very legitimate reason to invest in K8s development.
I'm speaking against my own interests here; I don't like K8s.
I'm speaking against my own interests here; I don't like K8s.
I think the point was aimed at people that want to debate about jails vs. linux namespaces. Not much of a debate if you don't understand the strengths and weaknesses of both sides.
If they're invested in an existing linux container ecosystem and don't intend to change then yeah probably not much gained.
If they're invested in an existing linux container ecosystem and don't intend to change then yeah probably not much gained.
[deleted]
In the aughts (2000s) I rented a "managed" VPS that was based on FreeBSD jails and an overlay filesystem. It was great in that you didn't have to maintain most of the software yourself (e.g. security updates) because it was maintained by the hoster on the base filesystem layer, while at the same time you were still root on your own filesystem overlay, where you could add or modify any file from the base filesystem. If you messed up somehow, you could simply revert to the base filesystem version of the respective files. Furthermore only the files on your overlay counted against your filesystem quota. Unfortunately the hoster discontinued that offering after a couple of years.
> Unfortunately the hoster discontinued that offering after a couple of years.
I’m planning on starting a FreeBSD based hosting service, but I am not sure if anyone actually wants what I have in mind.
The thing is that I have some couple of very specific ideas in mind.
I will offer a small amount of storage, and not general internet connectivity.
Users will have 25GB space and the idea is that they will be able to connect over Wireguard, but they cannot make outbound connections from the host.
It’ll be like a /home/user in the cloud.
So you can keep some files there and connect from wherever in the world and use the cli tools we all know and love. But idk if there is an actual market for that or not.
The selling point is that I will be focusing on the storage of that data. ZFS with redundancy and offsite backups – the works. That’s why the amount of data should be low. Only keep the most important files there.
Also I’m gonna accept payments in Bitcoin only, and people will have to sign up for many years upfront.
I think maybe the number of people that want this can be counted on one hand heh
I’m planning on starting a FreeBSD based hosting service, but I am not sure if anyone actually wants what I have in mind.
The thing is that I have some couple of very specific ideas in mind.
I will offer a small amount of storage, and not general internet connectivity.
Users will have 25GB space and the idea is that they will be able to connect over Wireguard, but they cannot make outbound connections from the host.
It’ll be like a /home/user in the cloud.
So you can keep some files there and connect from wherever in the world and use the cli tools we all know and love. But idk if there is an actual market for that or not.
The selling point is that I will be focusing on the storage of that data. ZFS with redundancy and offsite backups – the works. That’s why the amount of data should be low. Only keep the most important files there.
Also I’m gonna accept payments in Bitcoin only, and people will have to sign up for many years upfront.
I think maybe the number of people that want this can be counted on one hand heh
One question you may want to answer is why anyone would use your solution vs something like rsync.net or a small VM in DigitalOcean, etc. You said the selling point would be storage. What would be the differentiator in that area? Would price be it?
Just trying to help with question to validate the idea. Best of luck.
Just trying to help with question to validate the idea. Best of luck.
> something like rsync.net
My impression of rsync.net is that it is for backup. Whereas my service would be a live service you ssh into. Basically, connect to Wireguard VPN and then ssh into your cloud home.
The idea being that this is where you keep, and interact with, your files that are important.
> DigitalOcean
My service would have less system administration for the user, and high level of storage redundancy and offsite backups
> Just trying to help with question to validate the idea. Best of luck.
Thank you, I appreciate it :)
My impression of rsync.net is that it is for backup. Whereas my service would be a live service you ssh into. Basically, connect to Wireguard VPN and then ssh into your cloud home.
The idea being that this is where you keep, and interact with, your files that are important.
> DigitalOcean
My service would have less system administration for the user, and high level of storage redundancy and offsite backups
> Just trying to help with question to validate the idea. Best of luck.
Thank you, I appreciate it :)
Got it! This: less system administration, that's a good selling point!
No outbound connections would likely be a deal-breaker for many.
Verio VPS FTW! That was the first proper hosting I had.
Since this is gaining some traction. I'm curious if you are using jails and a jail manager, what are you're using now? I'm still using IOCAGE (Version 1.2) on freeBSD 13, but it seems like its not the best way to do it anymore? Documentation is all over the place on what is the best way to manage jails now.
I use BastilleBSD, it has a simple template system which I really like and decent docs.
https://github.com/BastilleBSD/bastille
Bastille also has a sister project 'rocinante' which allows you to use Bastille templates on the host. I converted my ansible scripts to bastille templates and it works a lot better for *ME*. I found I spend more time updating ansible scripts whenever I needed to use them, it costed more time then just using a setup.sh script, which rocianate basically is. https://github.com/BastilleBSD/rocinante
Another new kid on the block for jails is AppJail, it has some interesting features. I have not played with it enough to say how stable it is. https://github.com/DtxdF/AppJail
Bastille also has a sister project 'rocinante' which allows you to use Bastille templates on the host. I converted my ansible scripts to bastille templates and it works a lot better for *ME*. I found I spend more time updating ansible scripts whenever I needed to use them, it costed more time then just using a setup.sh script, which rocianate basically is. https://github.com/BastilleBSD/rocinante
Another new kid on the block for jails is AppJail, it has some interesting features. I have not played with it enough to say how stable it is. https://github.com/DtxdF/AppJail
iocage as well... I've been trying to sucker someone into writing a new jail manager in flua, since we have that in base. It hasn't worked yet.
I'm in the same boat.
Started with ezjail, switched to iocage, now thinking about bastille or roll-my-own.
Started with ezjail, switched to iocage, now thinking about bastille or roll-my-own.
Still on ezjail myself. Set my NAS/home server up several years ago with a blog post tutorial centered around ezjail and have just carried that forward. Eventually will need to do a fresh reinstall though and would be interested to know what's considered best practice these days.
There are a lot of jail tools these days, it would be nice if we posted more modern FreeBSD guides.
Myself I use BastilleBSD heavily for personal and small business use.
https://github.com/BastilleBSD/bastille
https://github.com/BastilleBSD/bastille
Is there a "best article" out there comparing BSD Jails to Docker and the strengths and weaknesses of choosing one or the other?
(I'm also curious if BSD Jails are the same thing as Solaris Zones but with a different name or if there is significant nuance making them different).
(I'm also curious if BSD Jails are the same thing as Solaris Zones but with a different name or if there is significant nuance making them different).
Use whichever one you're more comfortable with and that suits the rest of your stack best. There's nothing Jails are going to allow you to do that Linux containers can't do. All the shared-kernel workload isolation schemes are comparably safe.
For starters, Docker runs as a daemon.
For starters starters, Docker is a few layers up in the stack from jails. You would be better off comparing runc to jails or just cgroups+namespaces to jails.
No.
Docker and runc are very similar.
Docker run Docker containers. runc runs OCI containers.
And jails operate at a similar level, though without an image format.
Docker and runc are very similar.
Docker run Docker containers. runc runs OCI containers.
And jails operate at a similar level, though without an image format.
Right, so jails do not have images or any of the tooling around building and distributing images which is the vast majority of what makes up docker. This is why cgroups+namespaces are much closer to jails than docker is.
There are systems which build upon jails such as iocage and ezjail which are more similar to docker.
There are systems which build upon jails such as iocage and ezjail which are more similar to docker.
Just one nit, FreeBSD had jails before March of 2000. I know this because I was using them in 1999. And in fact they were merged into CURRENT in early 1999 and STABLE in mid/late 1999. It was merged into RELEASE in March 2000.
They were actually developed a year before, but the person who paid for development had a year of exclusive use.
http://freebsd.dk/sagas/jails/
http://freebsd.dk/sagas/jails/
Very nice
I used jails on FreeBSD and nothing in Linux comes close. Yes, it is not a pointy-clicky setup like Linux likes to do. But IMHO Jails are far more secure, in a way: you get what you 'pay' for.
I used jails on FreeBSD and nothing in Linux comes close. Yes, it is not a pointy-clicky setup like Linux likes to do. But IMHO Jails are far more secure, in a way: you get what you 'pay' for.
Jails are not, in fact, far more secure than Linux containers. Arguably they're a better, more coherent design, but both are shared-kernel isolation schemes, and the high-order bit of their security is exactly the same. The best argument you'll come up with is that FreeBSD kernel LPEs are rarer than those of Linux, but now we're just back to the standard Linux vs. FreeBSD security debate, which never goes anywhere.
I agree with all of this, with perhaps one tiny addendum. Defaults matter, and last I checked Docker/container defaults are miserable for security. I'm guessing if you randomly inspected production container vs jails out in the world, way more container solutions would have giant holes in them compared to jails.
Defaults on Docker specifically are pretty locked down.
Even if if the container process has the root uid it is not able to do most things normal root can do, and is pretty much limited to being able to access root owned files and binding to privileged ports (which in a container doesn't tend to matter).
> Defaults on Docker specifically are pretty locked down.
That is strongly dependent on your threat model. The default docker configuration completely bypasses the firewall, making it trivial for containers to be exposed to the open internet with no way for admins to prevent it[0]. Likewise, I hesitate to call docker's default of running as root safe since it means anyone with access to the docker socket immediately has root on the host.
[0] It is quite easy for someone even slightly inexperienced to accidentally write, say `-p 1234:1234` instead of `-p 127.0.0.1:1234:1234` and thereby cause a security incident or near-miss; ask me how I know.
That is strongly dependent on your threat model. The default docker configuration completely bypasses the firewall, making it trivial for containers to be exposed to the open internet with no way for admins to prevent it[0]. Likewise, I hesitate to call docker's default of running as root safe since it means anyone with access to the docker socket immediately has root on the host.
[0] It is quite easy for someone even slightly inexperienced to accidentally write, say `-p 1234:1234` instead of `-p 127.0.0.1:1234:1234` and thereby cause a security incident or near-miss; ask me how I know.
> Likewise, I hesitate to call docker's default of running as root safe since it means anyone with access to the docker socket immediately has root on the host.
I never got why this is commonly used as an argument against Docker, TBH. You just don't give out access to the Docker socket to anything untrusted. Doesn't pretty much everyone know that by now?
I feel like people always say Docker is awfully insecure, but then the proofs-of-concept include flags like `--privileged`, or the socket is mounted, or / is mounted, or --net=host is set... etc. Docker by default always seemed pretty good to me, but I'm not very experienced in that realm, so I'm just wondering what I'm missing.
I never got why this is commonly used as an argument against Docker, TBH. You just don't give out access to the Docker socket to anything untrusted. Doesn't pretty much everyone know that by now?
I feel like people always say Docker is awfully insecure, but then the proofs-of-concept include flags like `--privileged`, or the socket is mounted, or / is mounted, or --net=host is set... etc. Docker by default always seemed pretty good to me, but I'm not very experienced in that realm, so I'm just wondering what I'm missing.
Docker has to run as root, or use otherwise insecure methods ("rootless" is a sham, it requires suid binaries and CVE ridden unprivileged user namespaces).
I agree with ports, working[0][1][2] on it.
[0] https://github.com/moby/moby/discussions/45524
[1] https://github.com/moby/moby/issues/45532
[2] https://github.com/moby/moby/pull/45076
I agree with ports, working[0][1][2] on it.
[0] https://github.com/moby/moby/discussions/45524
[1] https://github.com/moby/moby/issues/45532
[2] https://github.com/moby/moby/pull/45076
That said, docker does have support for running "rootless"
Please provide technical details of `Jails are far more secure`.
Multiple CVEs for escaping? E.g.:
* https://blog.aquasec.com/cve-2022-0185-linux-kernel-containe...
* https://securitylabs.datadoghq.com/articles/dirty-pipe-conta...
* https://snyk.io/learn/docker-security/top-5-vulnerabilities/
* https://www.container-security.site/attackers/container_brea...
The only escaping of jails that I've heard of in the last ~20 years is one not in the jails code, but tunnelling out through devfs:
* https://www.freebsd.org/security/advisories/FreeBSD-SA-14:07...
* https://blog.aquasec.com/cve-2022-0185-linux-kernel-containe...
* https://securitylabs.datadoghq.com/articles/dirty-pipe-conta...
* https://snyk.io/learn/docker-security/top-5-vulnerabilities/
* https://www.container-security.site/attackers/container_brea...
The only escaping of jails that I've heard of in the last ~20 years is one not in the jails code, but tunnelling out through devfs:
* https://www.freebsd.org/security/advisories/FreeBSD-SA-14:07...
No? Because many of them Docker specific. Neither I mentioned Docker anywhere, not Docker is a good example of software with a good security record. Another link lumps together k8s, runc and so on. I specifically mentioned systemd-nspaw in the beginning as contender to jails, not other solutions.
Misleading: the most important vulnerabilities that break shared-kernel isolation systems like Jails, Zones, and containers are kernel LPEs, and they aren’t reported as “escapes”.
What about `firejail` on Linux? Is it not much more secure than FreeBSD Jails which is just based on chroot?
FreeBSD Jails are conceptually just based on chroot. Their actual design and implementation are a few orders of magnitude more secure.
A few order of magnitudes? Can you be more specific? I'm genuinely wondering what you could be referring to that I've missed.
chroot more or less isn't secure at all. All they're really saying is that Jails aren't built on top of chroot, but rather are a ground-up reimplementation motivated by improving chroot.
Ah! For some reason I thought jails were built on something similar to chroot at the kernel level, meaning that it also came with the same security caveats.
(to be more accurate, I knew that it was not "on top" in the same way as Linux containers are built on top of chroot. )
(to be more accurate, I knew that it was not "on top" in the same way as Linux containers are built on top of chroot. )
Thanks, I'll check out the specifics.
What makes Jails so far ahead of containers in your opinion?
Not op, but for me:
You can setup Linux in a jail.
You can nest bhyve virtual machines in a jail.
You can assign individual NICs to a jail.
You can encrypt with ZFS a jail.
You can run browsers in a jail with full set of features.
You can setup Linux in a jail.
You can nest bhyve virtual machines in a jail.
You can assign individual NICs to a jail.
You can encrypt with ZFS a jail.
You can run browsers in a jail with full set of features.
Most of those are supported by "containers", which are after all just a chroot and a process tree with restricted visibility into certain kernel objects. Assigning individual NICs or running a browser with extra isolation is one of the classic Linux namespace tutorials.
edit: I'm not talking about running binaries compiled for Linux under FreeBSD. The parent said it's possible to set up Linux within a jail, so I want to see instructions to boot an actual Linux kernel as a FreeBSD process.
> You can setup Linux in a jail.
edit: I'm not talking about running binaries compiled for Linux under FreeBSD. The parent said it's possible to set up Linux within a jail, so I want to see instructions to boot an actual Linux kernel as a FreeBSD process.
> Assigning individual NICs or running a browser with extra isolation is one of the classic Linux namespace tutorials.
With FreeBSD you can set up a standalone virtualized network stack:
* https://klarasystems.com/articles/virtualize-your-network-on...
* https://man.freebsd.org/cgi/man.cgi?vnet
* https://wiki.freebsd.org/Jails/VNET
With FreeBSD you can set up a standalone virtualized network stack:
* https://klarasystems.com/articles/virtualize-your-network-on...
* https://man.freebsd.org/cgi/man.cgi?vnet
* https://wiki.freebsd.org/Jails/VNET
I read the quote you're replying to as "you can also do this under linux", i.e. it didn't dispute that freebsd jails can do it. But it's not a distinguishing feature of jails.
You use FreeBSD kernel and Linux userland. FreeBSD can emulate Linux system calls.
There is no such thing as “Linux userland”. Obviously you can port most GNU utilities and other things that are typically run on Linux on FreeBSD but that’s a far cry from “running Linux”.
You could maybe build and run User-Mode Linux on FreeBSD, and run that in a jail. That's pretty close to your own interpretation of "Linux kernel in a jail"
But to be real: that's obviously not what was meant. If someone wants to install and run Ubuntu inside a FreeBSD jail, it is well possible to do that. Nobody really cares if it's running kernel.org code, just that the binaries are running as expected.
But to be real: that's obviously not what was meant. If someone wants to install and run Ubuntu inside a FreeBSD jail, it is well possible to do that. Nobody really cares if it's running kernel.org code, just that the binaries are running as expected.
Using the linuxulator.
https://wiki.freebsd.org/LinuxJails
https://wiki.freebsd.org/LinuxJails
[deleted]
All of those available on Linux and even more, see https://www.man7.org/linux/man-pages/man1/systemd-nspawn.1.h...
I don't see the guide to run FreeBSD in a container from that man page.
And what about if my Linux Distro doesn't use system-md, like Slackware.
And what about if my Linux Distro doesn't use system-md, like Slackware.
I wonder why would anyone spend time on developing and maintaining that. FreeBSD needs Linux userland and syscall support to get roughly the same level of software compatibility. The opposite isn't true (maybe it is for your special snowlake internal application developed specifically for FreeBSD, but it isn't so in general case).
All other things you have mentioned are possible with pretty much any of the Linux container runtimes. systemd-nspawn is just one of them, and I don't think it is actually used very much compared to the alternatives.
All other things you have mentioned are possible with pretty much any of the Linux container runtimes. systemd-nspawn is just one of them, and I don't think it is actually used very much compared to the alternatives.
> And what about if my Linux Distro doesn't use system-md, like Slackware.
It's a choice of Slackware not use systemd infra. However on Linuxes you have more that one choice to run containers, your next option would be LXC/LDX https://docs.slackware.com/howtos:misc:lxc
It's a choice of Slackware not use systemd infra. However on Linuxes you have more that one choice to run containers, your next option would be LXC/LDX https://docs.slackware.com/howtos:misc:lxc
Sure and my choice not to use it too. Well aware of LXC too.
I'm not claiming Linux can't. I was stating why. I know Linux can do what FreeBSD can do, but BSD does it better.
However both are pretty much mimicked concept from Solaris Zones
I'm not claiming Linux can't. I was stating why. I know Linux can do what FreeBSD can do, but BSD does it better.
However both are pretty much mimicked concept from Solaris Zones
> However both are pretty much mimicked concept from Solaris Zones
Your timeline is off.
FreeBSD and Linux were within a year of each other (not in-tree on linux, but Debian packaged VServer kernels and VServer + GRSec kernels; we used vserver+grsec debian packaged kernels at work in the mid through late '00s).
Solaris containers came years later.
2000 - FreeBSD jails
2001 - Linux VServer
2004 - Solaris Containers
https://blog.aquasec.com/a-brief-history-of-containers-from-...
Your timeline is off.
FreeBSD and Linux were within a year of each other (not in-tree on linux, but Debian packaged VServer kernels and VServer + GRSec kernels; we used vserver+grsec debian packaged kernels at work in the mid through late '00s).
Solaris containers came years later.
2000 - FreeBSD jails
2001 - Linux VServer
2004 - Solaris Containers
https://blog.aquasec.com/a-brief-history-of-containers-from-...
I don’t know about the low level stuff but LXC is pretty much the same end result as FreeBSD jails. At least for my very basic needs that is.
Instruction are for FreeBSD 12, I wonder if it's still current for 13 and 14...?
In my limited experience with BSD over the years (I tinker with it every year or so), a lot of the core experience remains the same. Additions to rc.conf should keep working, as well as configuration files for things like jails.
Not really.
Specifically the suggestion to use lo# instead of vnet.
Specifically the suggestion to use lo# instead of vnet.
Every time I think of freebsd and how cool it was and is I wondering what caused linux to overtake it. I like linux as well, but I just don't understand the mechanics behind freebsd not gaining traction.
For me it was Java. I think there was something with mmap().
I ran FreeBSD servers with jails around ‘00. They worked fine.
Not for me, but in hindsight, maybe the FreeBSD daemon mascotte had something to do with it.
But around that time bea weblogic and ibm websphere also became popular, and I don’t think Java was officially supported. It was also a period where people tended to run oracle, but I’m not sure if that tab on Linux or something else. I think it was some custom Linux “unbreakable oracle”
I ran FreeBSD servers with jails around ‘00. They worked fine.
Not for me, but in hindsight, maybe the FreeBSD daemon mascotte had something to do with it.
But around that time bea weblogic and ibm websphere also became popular, and I don’t think Java was officially supported. It was also a period where people tended to run oracle, but I’m not sure if that tab on Linux or something else. I think it was some custom Linux “unbreakable oracle”
AT&T v BSDi had a large hand in that outcome of fates. It put the legality of BSD code in all its places into doubt for many years.
Ironically, just a few years later, SCO thought "hey that was a good idea" and did it to Linux too.
Ironically, just a few years later, SCO thought "hey that was a good idea" and did it to Linux too.
Off topic: I miss the days when web pages looked like this link.
High info density, consistent look, “responsive” without being responsive.
High info density, consistent look, “responsive” without being responsive.
Offtopic, but isn't it funny that the system that calls itself FreeBSD has a concept of "jails"? Couldn't they come up with better name? :)
This may be me shouting into the void, but I wish there were an article directly comparing jails with namespaces, which is the Linux functionality that Docker uses. I can totally believe that FreeBSD jails provide a better / more unified / more secure experience than Docker, but to extend that into saying "FreeBSD jails are better than Linux namespaces" feels like a category error.
Questions I would like to see answered in that article:
* Can jails be used to run subprocesses in the normal filesystem, but with a different network environment (for example making a given command run its net traffic through TAP)?
* Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process?
* Can live processes be moved into or out of a jail?
* Can jails be used to make a process think it's running as a different user?
I feel like the answer to these questions is generally "no, that's not what jails are for", which is (1) a fine answer given the apparent goal of being a better chroot(), and (2) reinforces that jails and namespaces are addressing different problem domains.