====== HDD ======

[[https://github.com/openzfs/zfs/discussions/14916|ZFS Send & RaidZ - Poor performance on HDD #14916]]

All ''zfs_vdev'' read/write parameters are for number of parallel requests to one vdev (i.e. for 2x HDD mirror it is for each HDD).
So for very slow 2xHDD mirror it is good to limit reads to 2 (2xHDD = 4 parallel reads), and write to 2.

<code bash>
cat /sys/module/zfs/parameters/zfs_vdev_async_write_max_active
echo 2 > /sys/module/zfs/parameters/zfs_vdev_async_write_max_active
cat /sys/module/zfs/parameters/zfs_vdev_async_write_max_active
</code>

Use huge record size - it can help on SMR drives. Note: it only make sense for ZFS file system. Cannot be applied on ZVOL.

<code bash>
zfs set recordsize=16M hddpool/data
zfs set recordsize=16M hddpool/vz
</code>



For ZVOLs: [[https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Workload%20Tuning.html#zvol-volblocksize|zvol volblocksize]]

[[https://blog.guillaumematheron.fr/2023/500/change-zfs-volblock-on-a-running-proxmox-vm/|Change ZFS volblock on a running Proxmox VM]]

**Note:**  When no stripping is used (simple mirror) volblocksize should be 4kB (or at least the same as ashift).

**Note:**  Latest Proxmox default volblock size was increased form 8k to 16k. When 8k is used warning is shown:
<code>
Warning: volblocksize (8192) is less than the default minimum block size (16384).
To reduce wasted space a volblocksize of 16384 is recommended.
</code>

====== Change volblock (by copying to new) ======

<code bash>
zfs create -s -V 40G hddpool/data/vm-156-disk-0-16k -o volblock=16k
dd if=/dev/zvol/hddpool/data/vm-156-disk-0 of=/dev/zvol/hddpool/data/vm-156-disk-0-16k bs=1M status=progress conv=sparse
zfs rename hddpool/data/vm-156-disk-0 hddpool/data/vm-156-disk-0-backup
zfs rename hddpool/data/vm-156-disk-0-16k hddpool/data/vm-156-disk-0
</code>

<code bash>
# zfs list
rpool/data/vm-101-disk-0                       37.5G   854G  37.5G  -
rpool/data/vm-101-disk-0-backup                41.9G   854G  41.9G  -
</code>

====== bfq ======

Use of''bfq'' is mandatory. See [[iosched#my_findings|my findings]].

====== prefetch ======

By default prefetech is enabled. Enabling it generates additional ''async'' reads.

''/sys/module/zfs/parameters/zfs_prefetch_disable''
''/sys/module/zfs/parameters/zvol_prefetch_bytes''

NOTE: Disable it ! I was unable to peform full (dirty bitmap lost after restart) KVM backup to PBS due to huge sync read queue times (up to 25s) and backup times out with error.


====== SMR ======

  * SMR drives behaves correctly for sequential writes, but long working ZFS or LVM thin spread writes into lots of random location causing unusable IOPS. So never use SMR.
  * If SMR must be  used try to set ''spa_num_allocators=1'' (default is 4) [[https://openzfs.github.io/openzfs-docs/man/v2.3/4/zfs.4.html#spa_num_allocators|spa_num_allocators]]