LXC

Some hints:

• Docker is recommended to be used inside VM.
• overlay/overlay2 is not possible on ZFS (as documented by docker).
• In priviledged container it works with ZFS using AUFS. See below.
  • From LXC-NEWS: The aufs storage driver has been deprecated since LXC 2.1 and is now officially removed.
• ZFS over ZFS is not possible (ZFS nesting)

From documentation: https://pve.proxmox.com/wiki/Linux_Container#pct_configuration

• edit LXC container config

  /etc/pve/local/lxc/contained_id.conf

  features:  keyctl=1,nesting=1

• stop/start LXC container

  docker run hello-world

The vfs backend is a very simple fallback that has no copy-on-write support. Each layer is just a separate directory. Creating a new layer based on another layer is done by making a deep copy of the base layer into a new directory.
Since this backend doesn’t share diskspace use between layers, and since creating a new layer is a slow operation this is not a very practical backend. However, it still has its uses, for instance to verify other backends against, or if you need a super robust (if slow) backend that works everywhere.

docker info
...
 Server Version: 19.03.8
 Storage Driver: vfs
...

When restarted in privileged container: NOTE: restarting in privileged container do mess with user permission. Make backup/clone before.

docker info
...
 Storage Driver: aufs
  Root Dir: /var/lib/docker/aufs
  Backing Filesystem: zfs
  Dirs: 0
  Dirperm1 Supported: true
...

Edit existing local storage and add following content types:

• Disk image
• Container

Add mount point to LXC:

• Resources –> Add –> Mount Point
  • Storage: local
  • Backup: NO
  • Path: /var/lib/docker
  • Mount options: noatime

And new RAW disc will be created, with EXT4 FS. Docker will use overlay2 driver with this FS.

• Only directory hierarchy under /mnt/bindmounts are allowed to be bind-mounterd inside LXC containers.
• Host permission and ACL will be used. To play with permission to shared folder please read: Unprivileged LXC containers
• Container Settings)
• CONS: no disk size control inside guest. Possible to use btrfs subvolume quotas

Login to host using SSH node

btrfs subvol create /mnt/bindmounts
btrfs quota enable /mnt/bindmounts
btrfs subvol create /mnt/bindmounts/<cid>-docker
btrfs qgroup limit 50G /mnt/bindmounts/<cid>-docker
chown 100000.100000 /mnt/bindmounts/<cid>-docker
# setfacl -Rm user:100000:rwx,default:user:100000:rwx
 
pct set <cid> -mp0 /mnt/bindmounts/<cid>-docker,mp=/var/lib/docker

btrfs quota rescan /
btrfs qgroup show -pcre 

dockerd -D --storage-driver zfs
...
zfs command is not available: exec: "zfs": executable file not found in $PATH  storage-driver=zfs
...
apt-get install zfsutils-linux
 
dockerd -D --storage-driver zfs
...
DEBU[2020-05-06T16:37:51.502473451Z] cannot open /dev/zfs: open /dev/zfs: no such file or directory  storage-driver=zfs
...

/etc/pve/lxc/${container_id}.conf

lxc.mount.entry: /dev/zfs dev/zfs none bind,create=file

Exposing /dev/zfs works. NOTE: it gives too wide permisions, like zfs list shows ALL info about host ZFS. After this trick, Docker starts and detects ZFS without any additional configuration.

Solution:

cp /etc/apparmor.d/lxc/lxc-default-with-nesting /etc/apparmor.d/lxc/lxc-default-with-nesting-docker

Edit new file and update profile name and add some mount permissions:

# Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
# will source all profiles under /etc/apparmor.d/lxc
 
profile lxc-container-default-with-nesting-docker flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/lxc/container-base>
  #include <abstractions/lxc/start-container>
 
  deny /dev/.lxc/proc/** rw,
  deny /dev/.lxc/sys/** rw,
  mount fstype=proc -> /var/cache/lxc/**,
  mount fstype=sysfs -> /var/cache/lxc/**,
  mount options=(rw,bind),
  mount fstype=cgroup -> /sys/fs/cgroup/**,
  mount fstype=cgroup2 -> /sys/fs/cgroup/**,
  mount fstype=aufs,
  mount fstype=overlay,
}

systemctl reload apparmor

Edit /etc/pve/lxc/${container_id}.conf and append this line:

/etc/pve/lxc/${container_id}.conf

lxc.apparmor.profile: lxc-container-default-with-nesting-docker

/etc/pve/lxc/${container_id}.conf

lxc.apparmor.profile = unconfined

systemctl reload apparmor

Error: AUFS cannot be used in non-init user namespace

On host log:

kernel: overlayfs: filesystem on '/var/lib/docker/check-overlayfs-support244358035/upper' not supported as upperdir
kernel: overlayfs: filesystem on '/var/lib/docker/check-overlayfs-support445538983/upper' not supported as upperdir

There are some requirements to use overlayfs. It is not possible to use it over ZFS.