LXD and Docker containers nesting - Tutorial

Updated: June 17, 2021

Welcome to Arcane Weekly. Today, I want to talk to you about a problem you're not likely to encounter, but if you do, then you'd want a quick and handy solution. Tools at hand: LXD containers, Docker. Mission at hand: You want to run the two at the same time. More specifically, you want to spawn a LXD container, and then inside it, start a Docker container. Why? Why not.

Then, the problem you have is as follows. The Docker execution fails with the following error: docker: Error response from daemon: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: rootfs_linux.go:60: mounting "proc" to rootfs at "/proc" caused: permission denied: unknown. Lots of text there. Let's debug this, shall we?

Problem in more detail

Normally, when you run Docker containers, you shouldn't have any issues accessing /proc. But the above error tells us this isn't doable. It's a rather confusing message, because it has a lot of information, which creates a misleading trail. True, there could be tons of reasons for why you may encounter the issue. In my case, the setup was brand new, vanilla, without any crazy customizations. There were no other errors, and for all practical purposes, Docker has been installed and configured correctly.

Then I thought, I might want to look at how LXD does things, and perhaps this could give me a better understanding of why I was seeing this issue. Indeed, I soon found an old, semi-documented article about LXD, which explains the situation. For security reasons, containers are not allowed to access /proc and /sys in unsafe ways. A workaround for this - with the relevant security implications - is to permit such access.

Solution

What you need to do is allow nesting for the affected container.

lxc launch "image" "container name" -c security.nesting=true

If the container exists, then before you start it:

lxc config set "container name" security.nesting true

And then start the container:

lxc start "container name"

If it's already running, stop the container, set the policy, then start it again. The operation is non-destructive, so if you encounter the problem only after you've already spawned the container, you won't lose any work or data inside it. The setting is also full reversible. And our job be done here, fellas.

Conclusion

It's interesting to see that I've only now encountered an issue that's been around for about five odd years. Or that so many people are encountering this problem, and yet, the information is scarce and scattered about. Well, if you are going for nested containers, then you want to edit the security policy for affected containers in LXD. This is a relatively simple trick. There's more to it of course, like the concepts of privileged and unprivileged containers, and other security considerations. But if your work isn't about security, there aren't any downsides to this change.

Better documentation would help. Less cryptic error messages, too. Meanwhile, try to think why, in your setup, there could be permissions errors for what you're trying to do, and this should lead you in the right direction. Once you know how to frame the problem statement, you're 80% done. Now you can do containers within containers, the whole: Yo Dawn, I herd u like containers, so we put containers in your containers ... Inception. Bonus: try to see how many levels of nesting you can do before the universe implodes.

Cheers.

You may also like: