meta data for this page
  •  

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
vm:proxmox:lxc [2020/05/06 19:37] – niziakvm:proxmox:lxc [2021/02/22 08:18] (current) – niziak
Line 1: Line 1:
 ====== LXC ====== ====== LXC ======
  
 +===== rename CT =====
 +<code bash>pct set <VMID> --hostname <newname></code>
  
-===== docker inside unprivileged LXC ===== +===== update CT templates =====
- +
-Some hints: +
-  * Docker is recommended to be used inside VM. +
-  * ''overlay''/''overlay2'' is not possible on ZFS (as documented by docker). +
-  * In priviledged container it works with ''ZFS'' using ''AUFS''. See below. +
-      * From [[https://linuxcontainers.org/lxc/news/|LXC-NEWS]]: The aufs storage driver has been deprecated since LXC 2.1 and is now officially removed. +
-  * ZFS over ZFS is not possible (ZFS nesting) +
- +
-From documentation: [[https://pve.proxmox.com/wiki/Linux_Container#pct_configuration]] +
- +
-  * edit LXC container config <file | /etc/pve/local/lxc/contained_id.conf> +
-features:  keyctl=1,nesting=1 +
-</file> +
-  * stop/start LXC container <code bash>docker run hello-world</code> +
- +
-==== VFS FS is used by docker. ==== +
- +
- +
-  The vfs backend is a very simple fallback that has no copy-on-write support. Each layer is just a separate directory. Creating a new layer based on another layer is done by making a deep copy of the base layer into a new directory. +
-  Since this backend doesn’t share diskspace use between layers, and since creating a new layer is a slow operation this is not a very practical backend. However, it still has its uses, for instance to verify other backends against, or if you need a super robust (if slow) backend that works everywhere. +
-  +
 <code bash> <code bash>
-docker info +# pveam - Proxmox VE Appliance Manager 
-... +pveam update
- Server Version: 19.03.8 +
- Storage Driver: vfs +
-...+
 </code> </code>
  
-When restarted in privileged container: 
-NOTE: restarting in privileged container do mess with user permission. Make backup/clone before. 
  
-<code bash> +===== Shrink container disc =====
-docker info +
-... +
- Storage Driver: aufs +
-  Root Dir: /var/lib/docker/aufs +
-  Backing Filesystem: zfs +
-  Dirs: 0 +
-  Dirperm1 Supported: true +
-... +
-</code>+
  
 +It is not supported. Command <code bash>pct resize <VMID> rootfs <newsize></code> cannot be used.
  
-==== Use RAW image for Docker ====+Workaround 1: 
 +  * Stop container 
 +  * Edit ''<vmid>conf'' and set new disk size 
 +  * Perform backup 
 +  * Restore LXC from backup
  
-Edit existing ''local'' storage and add following content types: +Workaround 2: 
-  * ''Disk image'' +  * Change ZFS volume size <code bash>zfs set refquota=8G rpool/data/subvol-810-disk-0</code> 
-  * ''Container''+  * Adjust LXC config: 
 +    * Edit ''<vmid>conf'' and set new disk size, **OR** 
 +    * ''pct rescan''
  
-Add mount point to LXC: 
-  * Resources --> Add --> Mount Point 
-    * Storage: ''local'' 
-    * Backup: NO 
-    * Path: ''/var/lib/docker'' 
-    * Mount options: ''noatime'' 
- 
-And new RAW disc will be created, with EXT4 FS.  
-Docker will use ''overlay2'' driver with this FS. 
- 
-==== Bind mount host btrfs subvolume ==== 
- 
-  * Only directory hierarchy under ''/mnt/bindmounts'' are allowed to be bind-mounterd inside LXC containers. 
-  * Host permission and ACL will be used. To play with permission to shared folder please read: [[https://pve.proxmox.com/wiki/Unprivileged_LXC_containers|Unprivileged LXC containers]] 
-  * [[https://pve.proxmox.com/wiki/Linux_Container#pct_settings|Container Settings]]) 
-  * CONS: no disk size control inside guest. Possible to use btrfs subvolume quotas 
- 
-Login to host using SSH node 
- 
-<code bash> 
-btrfs subvol create /mnt/bindmounts 
-btrfs quota enable /mnt/bindmounts 
-btrfs subvol create /mnt/bindmounts/<cid>-docker 
-btrfs qgroup limit 50G /mnt/bindmounts/<cid>-docker 
-chown 100000.100000 /mnt/bindmounts/<cid>-docker 
-# setfacl -Rm user:100000:rwx,default:user:100000:rwx 
- 
-pct set <cid> -mp0 /mnt/bindmounts/<cid>-docker,mp=/var/lib/docker 
-</code> 
- 
-<code bash> 
-btrfs quota rescan / 
-btrfs qgroup show -pcre  
-</code> 
- 
- 
-==== Enabling ZFS tools and access - no success ==== 
- 
-<code bash> 
-dockerd -D --storage-driver zfs 
-... 
-zfs command is not available: exec: "zfs": executable file not found in $PATH  storage-driver=zfs 
-... 
-apt-get install zfsutils-linux 
- 
-dockerd -D --storage-driver zfs 
-... 
-DEBU[2020-05-06T16:37:51.502473451Z] cannot open /dev/zfs: open /dev/zfs: no such file or directory  storage-driver=zfs 
-... 
-</code> 
- 
-<file | /etc/pve/lxc/${container_id}.conf> 
-lxc.mount.entry: /dev/zfs dev/zfs none bind,create=file 
-</file> 
-Exposing ''/dev/zfs'' works. NOTE: it gives too wide permisions, like ''zfs list'' shows ALL info about host ZFS. 
-After this trick, Docker starts and detects ZFS without any additional configuration. 
-But usage is not possible. Docker cannot create additional subvolumes 
- 
-<code bash> 
-~# docker run hello-world 
-Unable to find image 'hello-world:latest' locally 
-latest: Pulling from library/hello-world 
-0e03bdcc26d7: Extracting [==================================================>]  2.529kB/2.529kB 
-docker: failed to register layer: exit status 1: "/sbin/zfs fs create -o mountpoint=legacy rpool/data/subvol-803-disk-0/007d22d205263c9f89d2e53ab75787430a20a1b1b3b0270acf4eb67459de09ed" => cannot create 'rpool/data/subvol-803-disk-0/007d22d205263c9f89d2e53ab75787430a20a1b1b3b0270acf4eb67459de09ed': permission denied 
-</code> 
- 
-Try to add permission on host (note, not possible to use UID which not exists in /etc/passwd) 
-<code bash> 
-zfs allow -u 100000 create,destroy,mount rpool/data/subvol-803-disk-0 
-</code> 
- 
-Doesn't help. Give up and switch to RAW EXT4 image. 
- 
- 
- 
-==== Trying to run on ZFS - without success ==== 
- 
-Solution: 
-<code bash> 
-cp /etc/apparmor.d/lxc/lxc-default-with-nesting /etc/apparmor.d/lxc/lxc-default-with-nesting-docker 
-</code> 
- 
-Edit new file and update ''profile'' name and add some mount permissions: 
- 
-<file|/etc/apparmor.d/lxc/lxc-default-with-nesting-docker> 
-# Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which 
-# will source all profiles under /etc/apparmor.d/lxc 
- 
-profile lxc-container-default-with-nesting-docker flags=(attach_disconnected,mediate_deleted) { 
-  #include <abstractions/lxc/container-base> 
-  #include <abstractions/lxc/start-container> 
- 
-  deny /dev/.lxc/proc/** rw, 
-  deny /dev/.lxc/sys/** rw, 
-  mount fstype=proc -> /var/cache/lxc/**, 
-  mount fstype=sysfs -> /var/cache/lxc/**, 
-  mount options=(rw,bind), 
-  mount fstype=cgroup -> /sys/fs/cgroup/**, 
-  mount fstype=cgroup2 -> /sys/fs/cgroup/**, 
-  mount fstype=aufs, 
-  mount fstype=overlay, 
-} 
-</file> 
- 
-<code bash>systemctl reload apparmor</code> 
- 
-Edit ''/etc/pve/lxc/${container_id}.conf'' and append this line: 
-<file | /etc/pve/lxc/${container_id}.conf> 
-lxc.apparmor.profile: lxc-container-default-with-nesting-docker 
-</file> 
- 
-=== Disabling apparmor === 
- 
-<file | /etc/pve/lxc/${container_id}.conf> 
-lxc.apparmor.profile = unconfined 
-</file> 
- 
-<code bash>systemctl reload apparmor</code> 
- 
- 
- 
-=== Forcing ''aufs'' === 
- 
-Error: AUFS cannot be used in non-init user namespace 
- 
-=== Forcing ''overlay2'' === 
- 
-On host log: 
-<code> 
-kernel: overlayfs: filesystem on '/var/lib/docker/check-overlayfs-support244358035/upper' not supported as upperdir 
-kernel: overlayfs: filesystem on '/var/lib/docker/check-overlayfs-support445538983/upper' not supported as upperdir 
-</code> 
-There are some requirements to use overlayfs. It is not possible to use it over ''ZFS''.