meta data for this page

Proxmox

Upgrading

Always upgrade with dist-upgrade ! 

apt update && apt dist-upgrade

Installation

  • BIOS:
    • Enable VT
    • Enable IOMMU
  • Choose ZFS
    • Leave some free space on HDD for swap partition
      • Click “Advanced Options” and manually set “hdsize”
      • Installer 6.1-2 BUG: On system with 2x 2TB HDDs, Proxmox installer on ZFS RAID1 propose only hdsize=953MB. This issue gone with Proxmox 6.2.
    • Do not create swap on ZFS zvol!
  • reboot
  • manually configure VLAN network from console VLAN 802.1Q
    • check if it is still necessary. Perhaps enabling VLAN aware birdge from GUI is enough
  • https://ip_address:8006

issues

initramfs

Proxmox installed on ZFS RADI1. After reboot, Proxmox boots into initramfs. From initramfs console: 

zfs list
Try running '/sbin/modprobe zfs' as root to load them.

REASON: no root cmdline, because UEFI boot was used instead of grub loader. SOLUTION: force legacy boot in BIOS OR TRY: 

echo -n " rootdelay=5" >> /etc/kernel/cmdline
pve-efiboot-tool refresh

initramfs

REASON: ZFS invoked too soon: https://pve.proxmox.com/wiki/ZFS:_Tips_and_Tricks#Boot_fails_and_goes_into_busybox 

modprobe zfs
zpool import -N -f rpool
exit

post-installation tips

  • set MAC address prefix: `Datacenter` → `Options`. Locally Administered Address Ranges are: 
    x2-xx-xx-xx-xx-xx
x6-xx-xx-xx-xx-xx
xA-xx-xx-xx-xx-xx
xE-xx-xx-xx-xx-xx
  • Speedup migrations to set unencrypted network traffic (in secured networks!): 
    migration: network=first_cluster_host_ip/22,type=insecure
  • disable subscription repo:
    sed -i.bak 's|deb https://enterprise.proxmox.com/debian/pve buster pve-enterprise|\#deb https://enterprise.proxmox.com/debian/pve buster pve-enterprise|' /etc/apt/sources.list.d/pve-enterprise.list
  • add no subscription repo: 
    echo 'deb http://download.proxmox.com/debian/pve buster pve-no-subscription' > /etc/apt/sources.list.d/pve-no-sub.list
  • apt-get update
  • apt-get upgrade
  • apt-get install dropbear busybox-static --no-install-recommends
  • Update grub config:
    /etc/default/grub
    GRUB_CMDLINE_LINUX_DEFAULT="rootdelay=15 mitigations=off panic=30 nomodeset textonly video=vesafb:off video=efifb:off"
    update-grub
  • SMART disk monitoring
    /etc/smartd.conf
    #                             Short   1AM  Long  Sat    2AM
DEVICESCAN -a -o on -S on -s (S/../.././01|L/../../6/02) -m root
    systemctl restart smartmontools.service
  • Activate ZFS e-mail notification https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_zfs
    • systemctl enable --now zfs-zed
  • Use own internal CA signed cert for Web UI: Certificate Management
    • For clustering Proxmox is using own self signed certs! Do not replace them!
  • Postfix relay:
    apt-get install postfix bsd-mailx
wget https://raw.githubusercontent.com/Tontonjo/proxmox/master/ez_proxmox_mail_configurator.sh
bash ez_proxmox_mail_configurator.sh
  • Assign admin permissions: https://pve.proxmox.com/wiki/User_Management#_real_world_examples
  • For HDDs storage, limit Proxmox operation speed to 100000KiB: 
    pvesm set local --bwlimit default=100000
    • repeat in for local, local-zfs, hdd-zfs, hdd etc…
    • to remove limit: 
      pvesm set local --delete bwlimit
  • For HDDs storage - force volume big block size. OpenZFS default was 8k. With OpenZFS 2.2 it was changed to 16k. But to reduce fragmentation and HDD seeks - set it to 128k 
    pvesm set hdd-zfs --blocksize 128k
  • Enable IOMMU
    • in `GRUB_CMDLINE_LINUX_DEFAULT` and add kernel parameter `intel_iommu=on` or `amd_iommu=on`
    • vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd