swapping when pleny of free of ram

https://github.com/openzfs/zfs/issues/18893

Host swapping burst observed when ZFS ARC cache grows.

Scenario:

• spl_cache_grow_work → __vmalloc → alloc_pages, and it asks for an order-10

allocation, a 4 MiB contiguous block, with GFP_ATOMIC. That is ZFS (the SPL layer) growing one of its slab caches.

• Why it swaps with 75 GB free
  • The free memory is in small pieces. Normal has only 8 free blocks of order 10 (from /proc/buddyinfo), so kswapd is woken to build one.
  • To make a contiguous 4 MiB block it scans the LRU and swaps out cold VM pages. It also runs the ZFS shrinker,

which is why the ARC keeps shrinking (8–9 GB, down from about 17 GB).

• This also explains why lowering swappiness never stopped it, since this reclaim is driven by the failed contiguous allocation,

not by general memory pressure.

Workarounds:

• Raise vm.compaction_proactiveness from 20 to about 60, so kcompactd keeps order-10 blocks available.
  • can make things worst when on Proxmox (CPU spikes due to kcompactd heavy work)
  • can cause swapping loops in low mem conditions (kcompact wants to create continous 4MB block will throw to swap lots of small pages)
• Protect the ARC. Set zfs_arc_min (about 16 GB). It stops the kswapd runs from trimming it, but doesn't stop the swapping.
• zswap: enabling it on more hosts makes these bursts cheap, since the swap-outs would be compressed in RAM.