Our goal is to get working a simple example of unprivileged containers. (Understanding what these steps do and customizing further is an exercise for the reader.)
We start from a fresh install of Debian Buster. Then,
As root:
apt install --no-install-recommends lxc uidmap gnupg dnsmasq-base libpam-cgfs
adduser my_username
Create the following files with the following content:
/etc/default/lxc-net :
USE_LXC_BRIDGE="true"
/etc/lxc/lxc-usernet :
my_username veth lxcbr0 10
/etc/sysctl.d/80-lxc-userns.conf :
kernel.unprivileged_userns_clone=1
Reboot.
Log in as my_username .
Verify that the bridge is up with ip a
. You should see lxcbr0.
Do grep ^my_username: /etc/sub[gu]id to get the initial and total number of subuids and subgids available for your user. You should see something like this:
/etc/subgid:my_username:100000:65536
/etc/subuid:my_username:100000:65536
Create the following file, making parent directories if necessary. Copy the idmap numbers from /etc/subgid and /etc/subuid above.
~/.config/lxc/default.conf :
lxc.idmap = g 0 100000 65536
lxc.idmap = u 0 100000 65536
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
Check the permissions on ~/.local/share . If the directory does not exist, you are fine; lxc will create it with the right permissions.
ls -ld ~/.local/share
If it exists, make sure it is (at least) group and world executable. If not:
chmod go+x ~/.local/share
(optional) Create the ~/.cache directory if it does not already exist:
mkdir ~/.cache
You should now be all set. Here is a quick test that runs apt update inside a Debian Buster container (that is, this demonstrates a Debian Buster container inside a Debian Buster host). We create log files lxc-create.log and lxc-start.log to which we refer in the additional commentary below. Verbosity in the log files is set to the maximum, "DEBUG".
DOWNLOAD_KEYSERVER=hkp://keyserver.ubuntu.com lxc-create -n mycontainer -o lxc-create.log -l DEBUG -t download -- -d debian -r buster -a amd64
lxc-start -n mycontainer -o lxc-start.log --logpriority=DEBUG
lxc-attach -n mycontainer -- env -i TERM="$TERM" apt update
lxc-stop -n mycontainer
lxc-destroy -n mycontainer
Additional commentary:
In general, LXC feels like it still has a lot of rough edges. The steps needed to get things working are far from obvious. There are inscrutable error messages, weird behavior that needs to be worked around, and lots of gotchas.
Initial set up was Debian Buster 10.3.0 "Buster" - Official amd64 NETINST 20200208-12:07, fresh install, UEFI, no "normal user" account, just root, GPT, ext4 on LVM, select only "standard system utilities" at the "Software selection" step in the installer.
The version of the lxc package was 1:3.1.0+really3.0.3-8 . This corresponds to upstream 3.0.3; the "3.1.0+really" prefix is a workaround for a Debian packaging misstep (9e0999289ae52dd50f915b71724fd3d9d64fd284) in 2019 that we will have to live with for a while, in order to preserve strictly increasing version numbers. This prefix should be ignored. The epoch 1: should also be ignored. (It was introduced at 6e451ce51f0fd632062546a3d6ee8cc72dc59723 in 2014.) These hexadecimal constants are Git commits to the Debian lxc package. (Surprisingly, these constants are not indexed by Google.)
We used the --no-install-recommends flag because we want to demonstrate a minimal set of packages to install.
When creating a new local non-root user, Debian Buster automatically creates entries in /etc/subuid and /etc/subgid , so there was no need to edit them. (Opinionated side note: the default allocation ought to be in blocks of size 100000 or 10000, not 65536. We humans are base-10 creatures.) (Blue sky: rearchitect Linux so that uids and subuids are in different namespaces.)
Future work: demonstrate error if user is not present in /etc/subuid or /etc/subgid .
Future work: demonstrate error if ~/.config/lxc/default.conf does not match /etc/sub?id .
Omitting the idmap lines in ~/.config/lxc/default.conf results in the following error message in lxc-create.log:
lxc-create mycontainer ... ERROR conf - conf.c:chown_mapped_root:3150 - No uid mapping for container root
lxc-create mycontainer ... ERROR lxccontainer - lxccontainer.c:do_storage_create:1288 - Error chowning "/home/my_username/.local/share/lxc/mycontainer/rootfs" to container root
lxc-create mycontainer ... ERROR conf - conf.c:suggest_default_idmap:4777 - You must either run as root, or define uid mappings
lxc-create mycontainer ... ERROR conf - conf.c:suggest_default_idmap:4778 - To pass uid mappings to lxc-create, you could create
lxc-create mycontainer ... ERROR conf - conf.c:suggest_default_idmap:4779 - ~/.config/lxc/default.conf:
lxc-create mycontainer ... ERROR conf - conf.c:suggest_default_idmap:4780 - lxc.include = /etc/lxc/default.conf
lxc-create mycontainer ... ERROR conf - conf.c:suggest_default_idmap:4781 - lxc.idmap = u 0 100000 65536
lxc-create mycontainer ... ERROR conf - conf.c:suggest_default_idmap:4782 - lxc.idmap = g 0 100000 65536
lxc-create mycontainer ... ERROR lxccontainer - lxccontainer.c:do_lxcapi_create:1869 - Failed to create (none) storage for mycontainer
lxc-create mycontainer ... ERROR lxc_create - tools/lxc_create.c:main:327 - Failed to create container mycontainer
In LXC 2 (Debian Stretch), the configuration variable was called lxc.id_map , with an underbar. (Opinionated side note: such a cosmetic change in the variable name was silly. The migration experience could have been smoother.) Using the old variable name results in the following error in lxc-create.log:
lxc-create mycontainer ... ERROR confile - confile.c:parse_line:2312 - Unknown configuration key "lxc.id_map"
lxc-create mycontainer ... ERROR parse - parse.c:lxc_file_for_each_line_mmap:142 - Failed to parse config file "/home/my_username/.config/lxc/default.conf" at line "lxc.id_map = u 0 100000 65536"
The LXC 2 variables lxc.network.type and lxc.network.link also have new names in LXC 3, lxc.net.0.type and lxc.net.0.link . Future work: demonstrate errors if one uses those LXC 2 variables in LXC 3.
Omitting the uidmap package, or not creating /etc/sysctl.d/80-lxc-userns.conf , or not rebooting result in the error messages in lxc-create.log which may look like some of the following. There is a bug in lxc-create that causes these "lxc-usernsexec" error messages to be broken in 3 simultaneous ways. They are (1) interleaved, (2) truncated, and (3) non-deterministic about which error message you will see. It would have been nice if, in addition to "No such file or directory", it printed what file or directory it was trying to access.
lxc-create mycontainer ... ERROR conf - conf.c:chown_mapped_root:3250 - lxc-usernsexec failed: No such file or directory - Failed to open ttyNo such file or directory - Failed to open ttyOperation not permitted - Failed to unshare mount and user namespac
lxc-create mycontainer ... ERROR conf - conf.c:chown_mapped_root:3250 - lxc-usernsexec failed: No such file or directory - Failed to open tt
lxc-create mycontainer ... ERROR lxc_create - tools/lxc_create.c:main:327 - Failed to create container mycontainer
You can check whether the kernel variable has been set properly (in /etc/sysctl.d/80-lxc-userns.conf) with the command /usr/sbin/sysctl kernel.unprivileged_userns_clone . This command does not require root, but /usr/sbin is not in the PATH by default. It should print the following:
kernel.unprivileged_userns_clone = 1
Omitting the gnupg package results in the following error message only on the console after the lxc-create command. This error does not show up in the log file. Bizarrely, you cannot even redirect this error to a file as you normally can by doing 2> file.log . It just disappears if you try.
ERROR: Missing recommended tool: gpg
You can workaround this by using --no-validate
lxc-create: mycontainer: lxccontainer.c: create_run_template: 1617 Failed to create container from template
lxc-create: mycontainer: tools/lxc_create.c: main: 327 Failed to create container mycontainer
The log file lxc-create.log is completely useless in this situation, but we give it below to make some additional commentary.
lxc-create mycontainer ... INFO confile - confile.c:set_config_idmaps:1605 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-create mycontainer ... INFO confile - confile.c:set_config_idmaps:1605 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-create mycontainer ... DEBUG conf - conf.c:chown_mapped_root:3190 - trying to chown "/home/my_username/.local/share/lxc/mycontainer" to 1000
lxc-create mycontainer ... DEBUG conf - conf.c:chown_mapped_root:3190 - trying to chown "/home/my_username/.local/share/lxc/mycontainer/rootfs" to 1000
lxc-create mycontainer ... DEBUG conf - conf.c:chown_mapped_root:3190 - trying to chown "/home/my_username/.local/share/lxc/mycontainer" to 1000
lxc-create mycontainer ... INFO confile - confile.c:set_config_idmaps:1605 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-create mycontainer ... INFO confile - confile.c:set_config_idmaps:1605 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-create mycontainer ... DEBUG storage - storage/storage.c:get_storage_by_name:231 - Detected rootfs type "dir"
lxc-create mycontainer ... ERROR lxccontainer - lxccontainer.c:create_run_template:1617 - Failed to create container from template
lxc-create mycontainer ... DEBUG conf - conf.c:idmaptool_on_path_and_privileged:2860 - The binary "/usr/bin/newuidmap" does have the setuid bit set
lxc-create mycontainer ... DEBUG conf - conf.c:idmaptool_on_path_and_privileged:2860 - The binary "/usr/bin/newgidmap" does have the setuid bit set
lxc-create mycontainer ... DEBUG conf - conf.c:lxc_map_ids:2952 - Functional newuidmap and newgidmap binary found
lxc-create mycontainer ... DEBUG storage - storage/storage.c:get_storage_by_name:231 - Detected rootfs type "dir"
lxc-create mycontainer ... INFO lxccontainer - lxccontainer.c:container_destroy:2977 - Destroyed rootfs for mycontainer
lxc-create mycontainer ... DEBUG conf - conf.c:idmaptool_on_path_and_privileged:2860 - The binary "/usr/bin/newuidmap" does have the setuid bit set
lxc-create mycontainer ... DEBUG conf - conf.c:idmaptool_on_path_and_privileged:2860 - The binary "/usr/bin/newgidmap" does have the setuid bit set
lxc-create mycontainer ... DEBUG conf - conf.c:lxc_map_ids:2952 - Functional newuidmap and newgidmap binary found
lxc-create mycontainer ... INFO lxccontainer - lxccontainer.c:container_destroy:3042 - Destroyed directory "/home/my_username/.local/share/lxc/mycontainer" for "mycontainer"
lxc-create mycontainer ... ERROR lxc_create - tools/lxc_create.c:main:327 - Failed to create container mycontainer
With the uidmap package properly installed, we get message "Functional newuidmap and newgidmap binary found" as seen above. It sure would have been helpful if an error indicating the opposite were reported if the uidmap package is not installed! Then, we would have had a clue of what to look for (e.g., apt-file search newuidmap). When debugging the problem of missing uidmap, the big breakthrough was finding this comment https://github.com/lxc/lxc/issues/2764#issuecomment-450339282.
(At this point, lxc-create runs successfully.)
Omitting the libpam-cgfs package results in the following error message in lxc-start.log. This also happens if you have installed libpam-cgfs but have not rebooted. The error message is (again) completely useless for figuring out how to fix it. It might have been helpful if "No such file or directory" reported what file or directory it had tried to access.
lxc-start mycontainer ... DEBUG terminal - terminal.c:lxc_terminal_peer_default:707 - No such device - The process does not have a controlling terminal
lxc-start mycontainer ... DEBUG conf - conf.c:chown_mapped_root:3190 - trying to chown "/dev/pts/0" to 1000
lxc-start mycontainer ... INFO start - start.c:lxc_init:904 - Container "mycontainer" is initialized
lxc-start mycontainer ... DEBUG lxccontainer - lxccontainer.c:wait_on_daemonized_start:830 - First child 1306 exited
lxc-start mycontainer ... ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:833 - No such file or directory - Failed to receive the container state
lxc-start mycontainer ... ERROR lxc_start - tools/lxc_start.c:main:330 - The container failed to start
lxc-start mycontainer ... ERROR lxc_start - tools/lxc_start.c:main:333 - To get more details, run the container in foreground mode
lxc-start mycontainer ... ERROR lxc_start - tools/lxc_start.c:main:336 - Additional information can be obtained by setting the --logfile and --logpriority options
Additional information cannot be obtained by setting the --logfile and --logpriority options; we are already using those options!
We check, and change if necessary, the permissions of ~/.local/share to work around the following problem. The "nano" command may be first to create ~/.local/share/ in the process of creating ~/.local/share/nano/ . This can happen if you used the sudoedit command to create files in /etc , or if you used nano to create ~/.config/lxc/default.conf . nano creates that directory with permissions too strict, namely 0700, for lxc. lxc puts the filesystems of its containers in ~/.local/share/lxc , so subuids need to be able to access it. It's curious that nano creates ~/.local/share with 0700 permissions but it creates ~/.local with 0755. This is probably a bug in nano.
If permissions of ~/.local/share are too strict, one will get the following error message in lxc-start.log.
lxc-start mycontainer ... ERROR start - start.c:print_top_failing_dir:125 - Permission denied - Could not access /home/my_username/.local/share. Please grant it x access, or add an ACL for the container root
lxc-start mycontainer ... ERROR sync - sync.c:__sync_wait:62 - An error occurred in another process (expected sequence number 3)
lxc-start mycontainer ... DEBUG network - network.c:lxc_delete_network:3180 - Deleted network devices
lxc-start mycontainer ... ERROR start - start.c:__lxc_start:1951 - Failed to spawn container "mycontainer"
lxc-start mycontainer ... DEBUG lxccontainer - lxccontainer.c:wait_on_daemonized_start:830 - First child 834 exited
lxc-start mycontainer ... ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:842 - Received container state "ABORTING" instead of "RUNNING"
lxc-start mycontainer ... ERROR lxc_start - tools/lxc_start.c:main:330 - The container failed to start
lxc-start mycontainer ... ERROR lxc_start - tools/lxc_start.c:main:333 - To get more details, run the container in foreground mode
lxc-start mycontainer ... ERROR lxc_start - tools/lxc_start.c:main:336 - Additional information can be obtained by setting the --logfile and --logpriority options
The Debian Wiki suggests the setfacl command (which is part of the "acl" package) as an alternative to chmod to make ~/.local/share accessible to unprivileged containers. Although it does work, it seems that, in order to do it correctly, you needs to individually add all your subuids, typically 65536 of them, to the ACL of that directory. Though I have yet to see lxc use more than 1 subuid, even if creating multiple containers. Non-root users inside the container use more UIDs, e.g., starting with 101000, but the container still works without those UIDs on the ACL for ~/.local/share .
Future work: what happens if you have an insufficient number of subuids or subgids?
Not creating and populating /etc/lxc/lxc-usernet results in the following error message in lxc-start.log :
lxc-start mycontainer ... WARN start - start.c:lxc_spawn:1758 - Operation not permitted - Failed to allocate new network namespace id
lxc-start mycontainer ... INFO network - network.c:lxc_create_network_unpriv_exec:2150 - Execing lxc-user-nic create /home/my_username/.local/share/lxc mycontainer 893 veth lxcbr0 (null)
lxc-start mycontainer ... ERROR network - network.c:lxc_create_network_unpriv_exec:2178 - lxc-user-nic failed to configure requested network: No such file or directory - Failed to open "/etc/lxc/lxc-usernet"
cmd/lxc_user_nic.c: 1296: main: Quota reached
lxc-start mycontainer ... ERROR start - start.c:lxc_spawn:1777 - Failed to create the configured network
lxc-start mycontainer ... DEBUG network - network.c:lxc_delete_network:3180 - Deleted network devices
lxc-start mycontainer ... DEBUG lxccontainer - lxccontainer.c:wait_on_daemonized_start:830 - First child 884 exited
lxc-start mycontainer ... ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:842 - Received container state "ABORTING" instead of "RUNNING"
lxc-start mycontainer ... ERROR start - start.c:__lxc_start:1951 - Failed to spawn container "mycontainer"
lxc-start mycontainer ... DEBUG conf - conf.c:idmaptool_on_path_and_privileged:2860 - The binary "/usr/bin/newuidmap" does have the setuid bit set
lxc-start mycontainer ... DEBUG conf - conf.c:idmaptool_on_path_and_privileged:2860 - The binary "/usr/bin/newgidmap" does have the setuid bit set
lxc-start mycontainer ... DEBUG conf - conf.c:lxc_map_ids:2952 - Functional newuidmap and newgidmap binary found
lxc-start mycontainer ... ERROR lxc_start - tools/lxc_start.c:main:330 - The container failed to start
lxc-start mycontainer ... ERROR lxc_start - tools/lxc_start.c:main:333 - To get more details, run the container in foreground mode
lxc-start mycontainer ... ERROR lxc_start - tools/lxc_start.c:main:336 - Additional information can be obtained by setting the --logfile and --logpriority options
If the network bridge is not properly set up, one gets the following error message in lxc-start.log :
lxc-start mycontainer ... WARN start - start.c:lxc_spawn:1758 - Operation not permitted - Failed to allocate new network namespace id
lxc-start mycontainer ... INFO network - network.c:lxc_create_network_unpriv_exec:2150 - Execing lxc-user-nic create /home/my_username/.local/share/lxc mycontainer 697 veth lxcbr0 (null)
lxc-start mycontainer ... ERROR network - network.c:lxc_create_network_unpriv_exec:2178 - lxc-user-nic failed to configure requested network: cmd/lxc_user_nic.c: 576: create_nic: Error attaching vethKL4MYM to lxcbr0
lxc-start mycontainer ... ERROR start - start.c:lxc_spawn:1777 - Failed to create the configured network
lxc-start mycontainer ... DEBUG network - network.c:lxc_delete_network:3180 - Deleted network devices
lxc-start mycontainer ... DEBUG lxccontainer - lxccontainer.c:wait_on_daemonized_start:830 - First child 688 exited
lxc-start mycontainer ... ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:842 - Received container state "ABORTING" instead of "RUNNING"
lxc-start mycontainer ... ERROR lxc_start - tools/lxc_start.c:main:330 - The container failed to start
lxc-start mycontainer ... ERROR start - start.c:__lxc_start:1951 - Failed to spawn container "mycontainer"
lxc-start mycontainer ... ERROR lxc_start - tools/lxc_start.c:main:333 - To get more details, run the container in foreground mode
lxc-start mycontainer ... ERROR lxc_start - tools/lxc_start.c:main:336 - Additional information can be obtained by setting the --logfile and --logpriority options
lxc-start mycontainer ... DEBUG conf - conf.c:idmaptool_on_path_and_privileged:2860 - The binary "/usr/bin/newuidmap" does have the setuid bit set
lxc-start mycontainer ... DEBUG conf - conf.c:idmaptool_on_path_and_privileged:2860 - The binary "/usr/bin/newgidmap" does have the setuid bit set
lxc-start mycontainer ... DEBUG conf - conf.c:lxc_map_ids:2952 - Functional newuidmap and newgidmap binary found
The type of networking we set up is called "independent bridge (masqueraded bridge)" at LXC/SimpleBridge on the Debian Wiki . Essentially it puts the container behind a NAT. (This means you will not be able to access the container from outside the host. Consider instead a Host-Shared Bridge, described on the same page, if you want your container to be visible to others on your network.)
You can check whether the network bridge is set up with the ip a command. If the bridge is set up, you should see an entry for lxcbr0. If not, become root, and use the command journalctl -u lxc-net to see error messages.
Not creating and populating the file /etc/default/lxc-net causes the following in journalctl -u lxc-net . It seems to have no errors despite not setting up the bridge.
... systemd[1]: Starting LXC network bridge setup...
... systemd[1]: Started LXC network bridge setup.
Here is what is should look like with /etc/default/lxc-net populated:
... systemd[1]: Starting LXC network bridge setup...
... dnsmasq[550]: started, version 2.80 cachesize 150
... dnsmasq[550]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth DNSSEC loop-detect inotify dumpfile
... dnsmasq-dhcp[550]: DHCP, IP range 10.0.3.2 -- 10.0.3.254, lease time 1h
... dnsmasq-dhcp[550]: DHCP, sockets bound exclusively to interface lxcbr0
... dnsmasq[550]: reading /etc/resolv.conf
... dnsmasq[550]: using nameserver (...)#53
... dnsmasq[550]: read /etc/hosts - 5 addresses
... systemd[1]: Started LXC network bridge setup.
... dnsmasq[550]: reading /etc/resolv.conf
... dnsmasq[550]: using nameserver (...)#53
Omitting the dnsmasq-base package results in the following error message in journalctl -u lxc-net
... lxc-net[510]: /usr/lib/x86_64-linux-gnu/lxc/lxc-net: 136: /usr/lib/x86_64-linux-gnu/lxc/lxc-net: dnsmasq: not found
... lxc-net[510]: Failed to setup lxc-net.
... lxc-net[510]: Failed to setup lxc-net.
... systemd[1]: lxc-net.service: Main process exited, code=exited, status=1/FAILURE
... systemd[1]: lxc-net.service: Failed with result 'exit-code'.
... systemd[1]: Failed to start LXC network bridge setup.
Although the error message "dnsmasq: not found" might suggest installing the dnsmasq package instead of dnsmasq-base, that results in the following error message in journalctl -u lxc-net
... systemd[1]: Starting LXC network bridge setup...
... lxc-net[528]: dnsmasq: failed to create listening socket for 10.0.3.1: Address already in use
... lxc-net[528]: Failed to setup lxc-net.
... lxc-net[528]: Failed to setup lxc-net.
... systemd[1]: lxc-net.service: Main process exited, code=exited, status=1/FAILURE
... systemd[1]: lxc-net.service: Failed with result 'exit-code'.
... systemd[1]: Failed to start LXC network bridge setup.
I don't know what you are supposed to do if you actually need the dnsmasq package installed for other reasons.
If you've installed everything except uidmap and have rebooted, you do not need to reboot after installing uidmap.
At some point, I think I got in a state in which I seemed to need to add to ~/.config/lxc/default.conf :
lxc.apparmor.profile=unconfined
However, I was unable to replicate this problem.
If lxc-create is the first process to create the ~/.cache directory, it creates it with the owner and group of a subuid / subgid instead of your username and group.
$ ls -ld .cache/
drwxr-xr-x 1 100000 100000 6 Feb 14 12:10 .cache/
This may eventually cause problems down the line (e.g., other applications which use ~/.cache), but we didn't see any in brief testing. Nevertheless, this is why we recommended manually creating the .cache directory.
The Debian wiki directs adding this to ~/.config/lxc/default.conf:
# "Secure" mounting
lxc.mount.auto = proc:mixed sys:ro cgroup:mixed
and it gives as justification: "Warning: Bad settings with lxc.mount.auto option can lead to security risk and data loss!"
Is there more explanation of this security risk? My understanding of unprivileged containers, in fact, the point of unprivileged containers, is that they cannot break the host system beyond how much an unprivileged user can break the host system. Maybe the "security risk" is the container messing with the non-root user running the container, or non-root users within the container gaining root within the container. For the purposes of demonstrating a minimal setup that just gets things working, we have omitted this line and disregarded the warning.
We use env -i in lxc-attach so that environment variables such as HOME, USER, LOGNAME, and MAIL inside the container do not inherit values from outside. We pass TERM so we can potentially use applications which do terminal manipulation, e.g., use ncurses.
An alternative to env -i TERM=$TERM bash to get a root shell in the container is
lxc-attach -n mycontainer -- su -
This results in the following message before getting a root shell prompt:
mesg: ttyname failed: No such device
This message comes from mesg n in /root/.profile and can be ignored. I don't know why it happens or the best way to prevent it.
I typically modify the entry for the ethernet device (e.g., eno1) in /etc/network/interfaces on the host from "allow-hotplug" to "auto". This forces the boot process to wait until internet is working before continuing, so avoids potential weirdness of creating a bridge, or a container, before host networking is up. Future work: investigate what happens if things happen in the wrong order.
In the instructions given above, it was not necessary to modify /etc/lxc/default.conf , which is nice, because that avoids potential conflicts with future Debian package upgrades.
The instructions above require a reboot. I have not investigated how to get LXC unprivileged containers working without a reboot.
Update (2021-07): The DOWNLOAD_KEYSERVER variable is for /usr/share/lxc/templates/lxc-download . The value was taken from the fix in upstream LXC. The old value hkp://pool.sks-keyservers.net is no longer being maintained (RIP Web Of Trust due to attacks) and now returns a certificate error causing downloads to fail.
To try lxc many times under many different settings for the preparation of this document, to see which things caused and fixed which errors, we used LVM snapshots. This avoided having to having to repeatedly reinstall Debian from scratch.
Some useful websites:
https://wiki.debian.org/LXC
https://wiki.debian.org/LXC/SimpleBridge
Finally, below are logs of successful lxc-create and lxc-start. These are the messages that are ignorable. The lxc-start log also includes messages produced by lxc-stop.
~/lxc-create.log :
lxc-create mycontainer ... INFO confile - confile.c:set_config_idmaps:1605 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-create mycontainer ... INFO confile - confile.c:set_config_idmaps:1605 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-create mycontainer ... DEBUG conf - conf.c:chown_mapped_root:3190 - trying to chown "/home/my_username/.local/share/lxc/mycontainer" to 1000
lxc-create mycontainer ... DEBUG conf - conf.c:chown_mapped_root:3190 - trying to chown "/home/my_username/.local/share/lxc/mycontainer/rootfs" to 1000
lxc-create mycontainer ... DEBUG conf - conf.c:chown_mapped_root:3190 - trying to chown "/home/my_username/.local/share/lxc/mycontainer" to 1000
lxc-create mycontainer ... INFO confile - confile.c:set_config_idmaps:1605 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-create mycontainer ... INFO confile - confile.c:set_config_idmaps:1605 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-create mycontainer ... DEBUG storage - storage/storage.c:get_storage_by_name:231 - Detected rootfs type "dir"
lxc-create mycontainer ... INFO confile - confile.c:set_config_idmaps:1605 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-create mycontainer ... INFO confile - confile.c:set_config_idmaps:1605 - Read uid map: type g nsid 0 hostid 100000 range 65536
~/lxc-start.log :
lxc-start mycontainer ... INFO confile - confile.c:set_config_idmaps:1605 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start mycontainer ... INFO confile - confile.c:set_config_idmaps:1605 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start mycontainer ... INFO lxccontainer - lxccontainer.c:do_lxcapi_start:961 - Set process title to [lxc monitor] /home/my_username/.local/share/lxc mycontainer
lxc-start mycontainer ... INFO lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:759 - Processing "reject_force_umount # comment this to allow umount -f; not recommended"
lxc-start mycontainer ... INFO seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start mycontainer ... INFO seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start mycontainer ... INFO seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start mycontainer ... INFO seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:759 - Processing "[all]"
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:759 - Processing "kexec_load errno 1"
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:759 - Processing "open_by_handle_at errno 1"
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errno)
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:759 - Processing "init_module errno 1"
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:759 - Processing "finit_module errno 1"
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:759 - Processing "delete_module errno 1"
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start mycontainer ... INFO seccomp - seccomp.c:parse_config_v2:970 - Merging compat seccomp contexts into main context
lxc-start mycontainer ... DEBUG terminal - terminal.c:lxc_terminal_peer_default:707 - No such device - The process does not have a controlling terminal
lxc-start mycontainer ... DEBUG conf - conf.c:chown_mapped_root:3190 - trying to chown "/dev/pts/0" to 1000
lxc-start mycontainer ... INFO start - start.c:lxc_init:904 - Container "mycontainer" is initialized
lxc-start mycontainer ... INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWUSER
lxc-start mycontainer ... INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWNS
lxc-start mycontainer ... INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWPID
lxc-start mycontainer ... INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWUTS
lxc-start mycontainer ... INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWIPC
lxc-start mycontainer ... DEBUG start - start.c:lxc_try_preserve_namespaces:196 - Preserved user namespace via fd 14
lxc-start mycontainer ... DEBUG start - start.c:lxc_try_preserve_namespaces:196 - Preserved mnt namespace via fd 15
lxc-start mycontainer ... DEBUG start - start.c:lxc_try_preserve_namespaces:196 - Preserved pid namespace via fd 16
lxc-start mycontainer ... DEBUG start - start.c:lxc_try_preserve_namespaces:196 - Preserved uts namespace via fd 17
lxc-start mycontainer ... DEBUG start - start.c:lxc_try_preserve_namespaces:196 - Preserved ipc namespace via fd 18
lxc-start mycontainer ... DEBUG conf - conf.c:idmaptool_on_path_and_privileged:2860 - The binary "/usr/bin/newuidmap" does have the setuid bit set
lxc-start mycontainer ... DEBUG conf - conf.c:idmaptool_on_path_and_privileged:2860 - The binary "/usr/bin/newgidmap" does have the setuid bit set
lxc-start mycontainer ... DEBUG conf - conf.c:lxc_map_ids:2952 - Functional newuidmap and newgidmap binary found
lxc-start mycontainer ... INFO start - start.c:do_start:1148 - Unshared CLONE_NEWNET
lxc-start mycontainer ... DEBUG conf - conf.c:idmaptool_on_path_and_privileged:2860 - The binary "/usr/bin/newuidmap" does have the setuid bit set
lxc-start mycontainer ... DEBUG conf - conf.c:idmaptool_on_path_and_privileged:2860 - The binary "/usr/bin/newgidmap" does have the setuid bit set
lxc-start mycontainer ... DEBUG conf - conf.c:lxc_map_ids:2952 - Functional newuidmap and newgidmap binary found
lxc-start mycontainer ... DEBUG start - start.c:lxc_spawn:1754 - Preserved net namespace via fd 10
lxc-start mycontainer ... WARN start - start.c:lxc_spawn:1758 - Operation not permitted - Failed to allocate new network namespace id
lxc-start mycontainer ... INFO network - network.c:lxc_create_network_unpriv_exec:2150 - Execing lxc-user-nic create /home/my_username/.local/share/lxc mycontainer 628 veth lxcbr0 (null)
lxc-start mycontainer ... NOTICE utils - utils.c:lxc_switch_uid_gid:1378 - Switched to gid 0
lxc-start mycontainer ... NOTICE utils - utils.c:lxc_switch_uid_gid:1387 - Switched to uid 0
lxc-start mycontainer ... NOTICE utils - utils.c:lxc_setgroups:1400 - Dropped additional groups
lxc-start mycontainer ... INFO start - start.c:do_start:1254 - Unshared CLONE_NEWCGROUP
lxc-start mycontainer ... DEBUG storage - storage/storage.c:get_storage_by_name:231 - Detected rootfs type "dir"
lxc-start mycontainer ... DEBUG conf - conf.c:lxc_mount_rootfs:1332 - Mounted rootfs "/home/my_username/.local/share/lxc/mycontainer/rootfs" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs" with options "(null)"
lxc-start mycontainer ... INFO conf - conf.c:setup_utsname:791 - Set hostname to "mycontainer"
lxc-start mycontainer ... DEBUG network - network.c:lxc_setup_netdev_in_child_namespaces:3032 - Network device "eth0" has been setup
lxc-start mycontainer ... INFO network - network.c:lxc_setup_network_in_child_namespaces:3053 - network has been setup
lxc-start mycontainer ... INFO conf - conf.c:mount_autodev:1118 - Preparing "/dev"
lxc-start mycontainer ... INFO conf - conf.c:mount_autodev:1165 - Prepared "/dev"
lxc-start mycontainer ... INFO conf - conf.c:lxc_fill_autodev:1209 - Populating "/dev"
lxc-start mycontainer ... DEBUG conf - conf.c:lxc_fill_autodev:1282 - Bind mounted host device node "/dev/full" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/full"
lxc-start mycontainer ... DEBUG conf - conf.c:lxc_fill_autodev:1282 - Bind mounted host device node "/dev/null" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/null"
lxc-start mycontainer ... DEBUG conf - conf.c:lxc_fill_autodev:1282 - Bind mounted host device node "/dev/random" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/random"
lxc-start mycontainer ... DEBUG conf - conf.c:lxc_fill_autodev:1282 - Bind mounted host device node "/dev/tty" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/tty"
lxc-start mycontainer ... DEBUG conf - conf.c:lxc_fill_autodev:1282 - Bind mounted host device node "/dev/urandom" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/urandom"
lxc-start mycontainer ... DEBUG conf - conf.c:lxc_fill_autodev:1282 - Bind mounted host device node "/dev/zero" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/zero"
lxc-start mycontainer ... INFO conf - conf.c:lxc_fill_autodev:1286 - Populated "/dev"
lxc-start mycontainer ... INFO conf - conf.c:mount_entry:2014 - No such file or directory - Failed to mount "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" (optional)
lxc-start mycontainer ... INFO conf - conf.c:mount_file_entries:2333 - Finished setting up mounts
lxc-start mycontainer ... DEBUG conf - conf.c:lxc_setup_dev_console:1771 - Mounted pts device "/dev/pts/0" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/console"
lxc-start mycontainer ... INFO utils - utils.c:lxc_mount_proc_if_needed:1231 - I am 1, /proc/self points to "1"
lxc-start mycontainer ... WARN conf - conf.c:lxc_setup_devpts:1616 - Invalid argument - Failed to unmount old devpts instance
lxc-start mycontainer ... DEBUG conf - conf.c:lxc_setup_devpts:1653 - Mount new devpts instance with options "gid=5,newinstance,ptmxmode=0666,mode=0620,max=1024"
lxc-start mycontainer ... DEBUG conf - conf.c:lxc_setup_devpts:1672 - Created dummy "/dev/ptmx" file as bind mount target
lxc-start mycontainer ... DEBUG conf - conf.c:lxc_setup_devpts:1677 - Bind mounted "/dev/pts/ptmx" to "/dev/ptmx"
lxc-start mycontainer ... DEBUG conf - conf.c:lxc_allocate_ttys:989 - Created tty "/dev/pts/0" with master fd 11 and slave fd 14
lxc-start mycontainer ... DEBUG conf - conf.c:lxc_allocate_ttys:989 - Created tty "/dev/pts/1" with master fd 15 and slave fd 16
lxc-start mycontainer ... DEBUG conf - conf.c:lxc_allocate_ttys:989 - Created tty "/dev/pts/2" with master fd 17 and slave fd 18
lxc-start mycontainer ... DEBUG conf - conf.c:lxc_allocate_ttys:989 - Created tty "/dev/pts/3" with master fd 19 and slave fd 20
lxc-start mycontainer ... INFO conf - conf.c:lxc_allocate_ttys:1005 - Finished creating 4 tty devices
lxc-start mycontainer ... DEBUG conf - conf.c:lxc_setup_ttys:940 - Bind mounted "/dev/pts/0" onto "/dev/tty1"
lxc-start mycontainer ... DEBUG conf - conf.c:lxc_setup_ttys:940 - Bind mounted "/dev/pts/1" onto "/dev/tty2"
lxc-start mycontainer ... DEBUG conf - conf.c:lxc_setup_ttys:940 - Bind mounted "/dev/pts/2" onto "/dev/tty3"
lxc-start mycontainer ... DEBUG conf - conf.c:lxc_setup_ttys:940 - Bind mounted "/dev/pts/3" onto "/dev/tty4"
lxc-start mycontainer ... INFO conf - conf.c:lxc_setup_ttys:949 - Finished setting up 4 /dev/tty<N> device(s)
lxc-start mycontainer ... INFO conf - conf.c:setup_personality:1716 - Set personality to "0x0"
lxc-start mycontainer ... DEBUG conf - conf.c:setup_caps:2529 - Capabilities have been setup
lxc-start mycontainer ... NOTICE conf - conf.c:lxc_setup:3716 - The container "mycontainer" is set up
lxc-start mycontainer ... INFO lsm - lsm/lsm.c:lsm_process_label_set_at:178 - Set AppArmor label to "lxc-container-default-cgns"
lxc-start mycontainer ... INFO apparmor - lsm/apparmor.c:apparmor_process_label_set:1101 - Changed AppArmor profile to lxc-container-default-cgns
lxc-start mycontainer ... DEBUG start - start.c:lxc_spawn:1829 - Preserved cgroup namespace via fd 19
lxc-start mycontainer ... NOTICE start - start.c:start:2037 - Exec'ing "/sbin/init"
lxc-start mycontainer ... NOTICE start - start.c:post_start:2048 - Started "/sbin/init" with pid "628"
lxc-start mycontainer ... DEBUG lxccontainer - lxccontainer.c:wait_on_daemonized_start:830 - First child 619 exited
lxc-start mycontainer ... NOTICE start - start.c:signal_handler:430 - Received 17 from pid 621 instead of container init 628
lxc-start mycontainer ... DEBUG start - start.c:signal_handler:447 - Container init process 628 exited
lxc-start mycontainer ... DEBUG start - start.c:__lxc_start:1984 - Container "mycontainer" is halting
lxc-start mycontainer ... INFO error - error.c:lxc_error_set_and_log:54 - Child <628> ended on signal (2)
lxc-start mycontainer ... DEBUG network - network.c:lxc_delete_network:3180 - Deleted network devices
lxc-start mycontainer ... DEBUG conf - conf.c:idmaptool_on_path_and_privileged:2860 - The binary "/usr/bin/newuidmap" does have the setuid bit set
lxc-start mycontainer ... DEBUG conf - conf.c:idmaptool_on_path_and_privileged:2860 - The binary "/usr/bin/newgidmap" does have the setuid bit set
lxc-start mycontainer ... DEBUG conf - conf.c:lxc_map_ids:2952 - Functional newuidmap and newgidmap binary found
No comments :
Post a Comment