CEPH performance

• BlueStore Config Reference: Sizing
• https://yourcmc.ru/wiki/Ceph_performance
• Ceph Performance Tuning Checklist
• New to Ceph, HDD pool is extremely slow
• Ceph Storage Performance
• Ceph: A Journey to 1 TiB/s
• https://www.boniface.me/posts/pvc-ceph-tuning-adventures/

Ceph is build for scale and works great in large clusters. In small cluster every node will be heavily loaded.

• adapt PG to number of OSDs to spread traffic evenly
• use krbd
• more OSD = better parallelism
• enable writeback on VMs (possible data loss on consumer SSDs)
• MTU 9000 (jumbo frames) Ceph Loves Jumbo Frames
• net latency <200us (ping -s 1000 pve)
• Ceph: A Journey to 1 TiB/s
  • Ceph is incredibly sensitive to latency introduced by CPU c-state transitions. Set Max perf in BIOS to disable C-States or boot Linux with GRUB_CMDLINE_LINUX=“idle=poll intel_idle.max_cstate=0 intel_pstate=disable processor.max_cstate=1”
  • Disable IOMMU in kernel

• Configuring Small Ceph Clusters for Optimal Performance - Josh Salomon, Red Hat
• number of PG should be power of 2 (or middle between powers of 2)
• same utilization (% full) per device
• same number of PG per OSD := same number of request per device
• same number of primary PG per OSD = read operations spread evenly
  • primary PG - original/first PG - others are replicas. Primary PG is used for read.
• use relatively more PG than for big cluster - better balance, but handling PGs consumes resources (RAM)
  • i.e. for 7 OSD x 2TB PG autoscaler recommends 256 PG. After changing to 384 IOops drastivally increases and latency drops.

Setting to 512 PG wasn't possible because limit of 250PG/OSD.

ceph mgr module enable balancer
ceph balancer on
ceph balancer mode upmap

If possible use balancer

Override default CRUSH assignment.

Better to use in warn mode, to do not put unexpected load when PG number will change.

ceph mgr module enable pg_autoscaler
#ceph osd pool set <pool> pg_autoscale_mode <mode>
ceph osd pool set rbd pg_autoscale_mode warn

It is possible to set desired/target size of pool. This prevents autoscaler to move data every time new data are stored.

ceph -s
ceph osd df # shows standard deviation

no tools to show primary PG balancing. Tool on https://github.com/JoshSalomon/Cephalocon-2019/blob/master/pool_pgs_osd.sh

# ceph tell 'osd.*' bluestore allocator score block
osd.0: {
    "fragmentation_rating": 0.27187848765399758
}
osd.1: {
    "fragmentation_rating": 0.31147177012467503
}
osd.2: {
    "fragmentation_rating": 0.30870023661486262
}
osd.3: {
    "fragmentation_rating": 0.25266931194419928
}
osd.4: {
    "fragmentation_rating": 0.29409796398594706
}
osd.5: {
    "fragmentation_rating": 0.33731626650673441
}
osd.6: {
    "fragmentation_rating": 0.23903976339003158
}

Do not keep osd_memory_target below 2G:

ceph config set osd osd_memory_target 4294967296
ceph config get osd osd_memory_target
4294967296

If journal is on SSD, change low_threshold to sth bigger - NOTE - check if is valid for BLuestore, probably this is legacy paramater for Filestore:

# internal parameter calculated from other parameters:
ceph config get osd journal_throttle_low_threshhold
0.600000
 
# 5GB:
ceph config get osd osd_journal_size
5120

• https://pve.proxmox.com/wiki/Ceph_mClock_Tuning

• https://docs.ceph.com/en/latest/rados/configuration/mclock-config-ref/#osd-capacity-determination-automated
• https://docs.ceph.com/en/latest/rados/configuration/mclock-config-ref/#set-or-override-max-iops-capacity-of-an-osd

Upon startup ceph mClock scheduler performs benchmarking of storage and configure IOPS according to results:

# ceph tell 'osd.*' config show | grep osd_mclock_max_capacity_iops_hdd
    "osd_mclock_max_capacity_iops_hdd": "269.194638",
    "osd_mclock_max_capacity_iops_hdd": "310.961086",
    "osd_mclock_max_capacity_iops_hdd": "299.505949",
    "osd_mclock_max_capacity_iops_hdd": "345.471699",
    "osd_mclock_max_capacity_iops_hdd": "356.290246",
    "osd_mclock_max_capacity_iops_hdd": "229.234009",
    "osd_mclock_max_capacity_iops_hdd": "266.478860",
 
# ceph tell 'osd.*' config show | grep osd_mclock_max_sequential
    "osd_mclock_max_sequential_bandwidth_hdd": "157286400",
    "osd_mclock_max_sequential_bandwidth_ssd": "1258291200",

Manual benchmark:

ceph tell 'osd.*' bench 12288000 4096 4194304 100

Override settings:

ceph config dump | grep osd_mclock_max_capacity_iops
 
for i in $(seq 0 7); do ceph config rm osd.$i osd_mclock_max_capacity_iops_hdd; done
ceph config set global osd_mclock_max_capacity_iops_hdd 111
 
ceph config dump | grep osd_mclock_max_capacity_iops

ceph tell 'osd.*' config show | grep osd_mclock_profile

ceph tell 'osd.*' config set osd_mclock_profile [high_client_ops|high_recovery_ops|balanced]
 
ceph tell 'osd.*' config show | grep osd_mclock_profile

ceph tell 'osd.*' config set osd_mclock_profile custom