NFS

https://stackoverflow.com/questions/72493803/why-did-nfs-switch-from-udp-to-tcp-as-the-standard-protocol-in-version-4

A new feature, available for both 2.4 and 2.5 kernels but not yet integrated into the mainstream kernel at the time 
of this writing, is NFS over TCP. Using TCP has a   distinct advantage and a distinct disadvantage over UDP. 
The advantage is that it works far better than UDP on lossy networks. When using TCP, a single dropped packet 
can be retransmitted, without the retransmission of the entire RPC request, resulting in better performance 
on lossy networks. In addition, TCP will handle network speed differences better than UDP, due to the underlying 
flow control at the network level. 

Mounts inside NFS export. Crossmounts.

• nohide: This option prevents the NFS server from hiding the contents of a directory when it's mounted by a client. If you don't use nohide, clients will only see the directory itself, not the contents of any subdirectories that are themselves mounted.
• crossmnt: This option is similar to nohide but specifically allows clients to navigate and access filesystems that are mounted within the exported directory. It ensures that the NFS client can “see through” the mount point and access the filesystems mounted beneath it.

/etc/exports

/srv * (sync,no_subtree_check,crossmnt,nohide)

On server /etc/exports:

• no_subtree_check - If a subdirectory of a filesystem is exported, but the whole filesystem isn't then whenever a NFS request arrives, the server must check not only that the accessed file is in the appropriate filesystem (which is easy) but also that it is in the exported tree (which is harder). This check is called the subtree_check.
• async - This option allows the NFS server to violate the NFS protocol and reply to requests before any changes made by that request have been committed to stable storage (e.g. disc drive).
• all_squash - Map all uids and gids to the anonymous user. Useful for NFS-exported public FTP directories, news spool directories, etc. The opposite option is no_all_squash, which is the default setting.

On client /etc/fstab:

• homeserver:/srv/downloads /srv/downloads nfs4 _netdev,x-systemd.after=network-online.target,x-systemd.device-timeout=0,noatime,nodiratime,nfsvers=4.2 0 0

NFS/Troubleshooting

mountstats --nfs

Setup 3 identical VMs with Debian 11. On one of machine entering into NFGS directory is insanely slow.

• On “fast” machine
  • wireshark shows few READDIR aggregated request to get all content and attributes
  • /boot/config-5.10.0-33-amd64
  • 5.10.226-1
  • cat /proc/sys/sunrpc/nfs_debug 0x0000
• On “slow” machine
  • wireshark shows few READDIR requests and then lots of single GETATTR requests for each file
  • dmesg contains lots of NFD debug info - main cause of slow speed
  • /boot/config-5.10.0-18-amd64
  • 5.10.140-1
  • cat /proc/sys/sunrpc/nfs_debug 0xFFFF

Reason - debug enabled:

rpcdebug -m nfs -s all

Disable it:

rpcdebug -m nfs -c all